Image fusion method and device, computer equipment and readable storage medium

Abstract

The invention relates to an image fusion method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a visible light image and an infrared image of a target component in a power transmission system in a working process; inputting the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is obtained by training based on comparison loss, similarity loss and content loss. The image fusion network in the embodiment of the invention is obtained by training based on the comparison loss, the similarity loss and the content loss, so that model training can be carried out by more comprehensively utilizing shallow layer features and deep layer features in the visible light image and the infrared image according to a plurality of loss functions in a model training process; therefore, a more accurate image fusion network can be obtained through training, the more accurate image fusion network can be adopted to carry out image fusion on the visible light image and the infrared image, and a more accurate fusion image can be obtained.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to an image fusion method, apparatus, computer device, and readable storage medium. Background Art

[0002] With the development of image processing technology, image fusion technology has emerged. For example, in a power transmission system, image fusion technology can be used to fuse complementary image information from different visual sensors to obtain a rich-information fused image. Among them, the visual sensors include visible light sensors and infrared sensors, and the fused image can be an image generated by fusing the visible light image and the infrared image of the target component in the power transmission system.

[0003] However, currently, for the image fusion method of visible light images and infrared images, there is a problem of low fusion accuracy. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide an image fusion method, apparatus, computer device, and readable storage medium, which can improve the fusion accuracy of the image fusion method for visible light images and infrared images.

[0005] In a first aspect, this application provides an image fusion method, including:

[0006] Obtain the visible light image and the infrared image of the target component in the power transmission system during operation;

[0007] Input the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on contrast loss, similarity loss, and content loss.

[0008] In one embodiment, the image fusion network includes a mask-guided sub-network and a contrast learning sub-network, the similarity loss includes a first similarity loss and a second similarity loss, and the step of inputting the visible light image and the infrared image into the preset image fusion network for fusion to obtain a fused image includes:

[0009] Input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fused image; the mask-guided sub-network is trained based on the first similarity loss and the content loss;

[0010] Input the intermediate fused image, the visible light image, and the infrared image into the contrast learning sub-network for contrast learning to obtain the fused image; the contrast learning sub-network is trained based on the second similarity loss and the contrast loss.

[0011] In one embodiment, the mask-guided sub-network includes a feature extraction layer and a fusion layer. The step of inputting the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fusion image includes:

[0012] Inputting the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain output feature maps;

[0013] Performing background information processing on the infrared image to obtain a mask image of the infrared image;

[0014] Inputting the output feature maps and the mask image into the fusion layer for image fusion to obtain the intermediate fusion image.

[0015] In one embodiment, the step of inputting the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain output feature maps includes:

[0016] Performing downsampling processing on the visible light image and the infrared image to obtain feature maps of different scales;

[0017] Stitching the feature maps of different scales to obtain a plurality of stitched feature maps;

[0018] Performing upsampling processing on the plurality of stitched feature maps to obtain the output feature maps.

[0019] In one embodiment, the training method of the mask-guided sub-network includes:

[0020] Obtaining a sample data set; the sample data set includes historical fusion images, historical visible light images, and historical infrared images;

[0021] Determining the content loss according to the historical fusion image, the historical visible light image, and the historical infrared image;

[0022] Determining the first similarity loss according to the historical fusion image, the historical visible light image, and the mask image of the historical infrared image;

[0023] Performing model training on the initial mask-guided sub-network according to the first similarity loss and the content loss to obtain the mask-guided sub-network.

[0024] In one embodiment, the training method of the contrastive learning sub-network includes:

[0025] Obtaining a sample data set; the sample data set includes historical fusion images, historical visible light images, and historical infrared images;

[0026] Extract a query slice from the historical fused image;

[0027] Extract a positive slice and a negative slice from the historical visible image and the historical infrared image according to the position of the query slice in the historical fused image;

[0028] Train an initial contrastive learning sub-network according to the sample data set, the query slice, the positive slice and the negative slice to obtain the contrastive learning sub-network.

[0029] In one embodiment, the training of the initial contrastive learning sub-network according to the sample data set, the query slice, the positive slice and the negative slice to obtain the contrastive learning sub-network includes:

[0030] Encode the query slice, the positive slice and the negative slice respectively to obtain a first feature vector, a second feature vector and a third feature vector;

[0031] Determine the contrastive loss according to the similarity between the first feature vector, the second feature vector and the third feature vector;

[0032] Determine the second similarity loss according to the historical fused image, the historical visible image and the historical infrared image;

[0033] Train the initial contrastive learning sub-network according to the contrastive loss and the second similarity loss to obtain the contrastive learning sub-network.

[0034] In a second aspect, the present application also provides an image fusion device, including:

[0035] An acquisition module for acquiring visible light images and infrared images of a target component in a power transmission system during operation;

[0036] A fusion module for inputting the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on a contrastive loss, a similarity loss and a content loss.

[0037] In a third aspect, the present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method in any one of the embodiments in the first aspect above.

[0038] Fourthly, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments in the first aspect are implemented.

[0039] Fifthly, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments in the first aspect are implemented.

[0040] In the above image fusion method, device, computer device and readable storage medium, a visible light image and an infrared image of a target component in a power transmission system during operation are obtained; the visible light image and the infrared image are input into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on a contrast loss, a similarity loss and a content loss. Since the image fusion network in the embodiments of the present application is trained based on a contrast loss, a similarity loss and a content loss, during the process of model training, multiple loss functions can be used to more comprehensively utilize the shallow features and deep features in the visible light image and the infrared image for model training. Therefore, a more accurate image fusion network can be trained. Thus, a more accurate image fusion network can be used to perform image fusion on the visible light image and the infrared image, and thus a more accurate fused image can be obtained. Description of the Drawings

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can be obtained based on these drawings.

[0042] Figure 1 It is an application environment diagram of the image fusion method in one embodiment;

[0043] Figure 2 It is a flowchart of the image fusion method in one embodiment;

[0044] Figure 3 It is a schematic diagram of an infrared image in one embodiment;

[0045] Figure 4 It is a schematic diagram of a visible light image in one embodiment;

[0046] Figure 5 It is a schematic diagram of a fused image in one embodiment;

[0047] Figure 6Schematic diagram of the image fusion step in an embodiment;

[0048] Figure 7 Schematic diagram of the intermediate fusion image generation step in an embodiment;

[0049] Figure 8 Schematic diagram of the mask-guided sub-network in an embodiment;

[0050] Figure 9 Schematic diagram of the DMG network in an embodiment;

[0051] Figure 10 Schematic diagram of the process of training the mask-guided sub-network in an embodiment;

[0052] Figure 11 Schematic diagram of the process of training the contrastive learning sub-network in an embodiment;

[0053] Figure 12 Schematic diagram of the contrastive learning sub-network in an embodiment;

[0054] Figure 13 Schematic diagram of the image fusion method in another embodiment;

[0055] Figure 14 Structural block diagram of the image fusion device in an embodiment;

[0056] Figure 15 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above accompanying drawings are intended to cover non-exclusive inclusion.

[0059] In the description of the embodiments of this application, technical terms such as "first" and "second" are only used to distinguish different objects and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity, specific order or primary-secondary relationship of the indicated technical features. In the description of the embodiments of this application, "a plurality of" means more than two unless otherwise specifically defined.

[0060] As used herein, the mention of "embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appearing at various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0061] With the development of image processing technology, image fusion technology has emerged and has been widely applied in fields such as target recognition, detection, image enhancement, surveillance, and remote sensing. For example, in a power transmission system, image fusion technology can be used to fuse complementary image information from different visual sensors to obtain an information-rich fused image. Among them, the visual sensors include visible light sensors and infrared sensors, and the fused image may refer to an image generated by fusing the visible light image and the infrared image of the target component in the power transmission system.

[0062] However, in the current image fusion methods for visible light images and infrared images, only shallow features of the images are used for model training during model training, and the deep features of the images are not fully utilized. Therefore, the current image fusion methods for visible light images and infrared images have the problem of low fusion accuracy.

[0063] After introducing the background technology of the image fusion method provided by the embodiments of the present application as above, below, the implementation environment involved in the image fusion method provided by the embodiments of the present application will be briefly described. The image fusion method provided by the embodiments of the present application can be applied to, for example, Figure 1 the terminal shown as follows. Figure 1 The application environment includes a power transmission system 11 and a monitoring device 12, and an image acquisition device is installed on the monitoring device 12. The monitoring device 12 also includes a terminal, and the image acquisition device and the terminal are communicatively connected. The image acquisition device may include, but is not limited to, at least two visual sensors, etc. The image acquisition device is used to acquire the visible light image and the infrared image of the target component in the power transmission system, and the terminal is used to perform image fusion on the visible light image and the infrared image to obtain a fused image.

[0064] Those skilled in the art can understand that Figure 1 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the monitoring device to which the solution of the present application is applied. The specific monitoring device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0065] In one embodiment, as shown in Figure 2As shown, an image fusion method is provided. Taking the terminal in Figure 1 as an example for illustration, the method includes the following steps:

[0066] S201, acquire visible light images and infrared images of a target component in a power transmission system during operation.

[0067] Among them, the target component can be any key component in the power transmission system, such as specific target components like line insulators, pins, etc. The visible light images and infrared images are used to characterize information such as the working state, position, and attitude of the target component in the power transmission system during operation. Based on the principle of optical imaging, visible light images have rich texture details and high spatial resolution. However, visible light images are affected by dark environments, fog, and other types of environmental interference; infrared images are based on the thermal radiation of objects and can highlight significant targets in interference environments. For example, the region with the largest amount of information in an infrared image is the region target containing the power grid target component. However, the signal-to-noise ratio of infrared images is low and lacks texture details. Therefore, the fused image has the advantages of both types of source images, having rich details and significant targets. Exemplarily, as Figure 3 shown, Figure 3 is a schematic diagram of an infrared image in an embodiment, as Figure 4 shown, Figure 4 is a schematic diagram of a visible light image in an embodiment.

[0068] In the embodiments of the present application, an image acquisition device can acquire visible light images and infrared images of a target component in a power transmission system during operation, and send the acquired visible light images and infrared images to the terminal. Thus, the terminal can receive the visible light images and infrared images acquired by the image acquisition device. Optionally, the terminal can acquire visible light images and infrared images in real time, or the terminal can also acquire visible light images and infrared images at regular intervals. Of course, the embodiments of the present application do not limit the specific implementation manners and sequences of acquiring visible light images and infrared images.

[0069] S202, input the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on contrast loss, similarity loss, and content loss.

[0070] In an embodiment of the present application, optionally, the terminal may directly input the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; or, the terminal may first perform image processing on the visible light image and the infrared image, such as image filtering, image enhancement, etc., and then input the visible light image and the infrared image after image processing into the preset image fusion network for fusion to obtain a fused image. Of course, the specific implementation manner of inputting into the preset image fusion network for fusion in the embodiment of the present application is not limited. Thus, the terminal may monitor faults or anomalies during the operation of the target component in the power transmission system based on the fused image, and perform corresponding fault handling.

[0071] Among them, the fused image is used to monitor faults or anomalies during the operation of the target component in the power transmission system. Exemplarily, as Figure 5 shown, Figure 5 is a schematic diagram of the fused image in an embodiment. It should be noted that the image fusion network is trained based on contrast loss, similarity loss, and content loss. Optionally, the image fusion network may be trained jointly based on contrast loss, similarity loss, and content loss; or, a mask guidance sub-network may be trained based on contrast loss and similarity loss, and a contrast learning sub-network may be trained based on content loss and similarity loss, and then the image fusion network is generated according to the mask guidance sub-network and the contrast learning sub-network. Of course, the specific implementation manner of training to obtain the image fusion network in the embodiment of the present application is not limited.

[0072] In the above image fusion method, a visible light image and an infrared image of a target component in a power transmission system during operation are obtained; the visible light image and the infrared image are input into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on contrast loss, similarity loss, and content loss. Since the image fusion network in the embodiment of the present application is trained based on contrast loss, similarity loss, and content loss, during the model training process, the shallow features and deep features in the visible light image and the infrared image can be more comprehensively utilized according to multiple loss functions, so that a more accurate image fusion network can be trained. Thus, a more accurate image fusion network can be used to perform image fusion on the visible light image and the infrared image, and thus a more accurate fused image can be obtained.

[0073] In an embodiment, the above image fusion network includes a mask guidance sub-network and a contrast learning sub-network. The above similarity loss includes a first similarity loss and a second similarity loss. Based on this, an implementation manner of image fusion is provided, that is, "input the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image" in S202 above, as Figure 6 shown, including:

[0074] S301. Input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fusion image. The mask-guided sub-network is trained based on the first similarity loss and the content loss.

[0075] Among them, the above image fusion network includes a mask-guided sub-network and a contrast learning sub-network. The mask-guided sub-network is trained based on the first similarity loss and the content loss. The above similarity loss includes the first similarity loss and the second similarity loss.

[0076] In the embodiment of the present application, optionally, the terminal can directly input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fusion image; or, the terminal can also first perform image processing on the visible light image and the infrared image, such as image filtering, image enhancement, etc., and then input the processed visible light image and infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fusion image. Of course, the embodiment of the present application does not limit the specific implementation manner of inputting into the mask-guided sub-network for fusion.

[0077] S302. Input the intermediate fusion image, the visible light image, and the infrared image into the contrast learning sub-network for contrast learning to obtain a fusion image. The contrast learning sub-network is trained based on the second similarity loss and the contrast loss.

[0078] In the embodiment of the present application, optionally, the terminal can directly input the intermediate fusion image, the visible light image, and the infrared image into the contrast learning sub-network for contrast learning to obtain a fusion image; or, the terminal can first determine query image slices, positive graphic slices, and negative image slices according to the intermediate fusion image, the visible light image, and the infrared image, and then perform contrast learning according to the query image slices, the positive graphic slices, the negative image slices, and the contrast learning sub-network to obtain a fusion image. Of course, the embodiment of the present application does not limit the specific implementation manner of contrast learning. Among them, the contrast learning sub-network is trained based on the second similarity loss and the contrast loss.

[0079] In this embodiment, the mask-guided sub-network can be trained based on the first similarity loss and the content loss, and the contrast learning sub-network can be trained based on the second similarity loss and the contrast loss. That is, according to multiple loss functions, the shallow features and deep features in the visible light image and the infrared image can be more comprehensively utilized for model training. Therefore, a more accurate image fusion network can be trained. Thus, a more accurate image fusion network can be used to perform image fusion on the visible light image and the infrared image, and a more accurate fusion image can be obtained.

[0080] In one embodiment, the above-mentioned mask-guided sub-network includes a feature extraction layer and a fusion layer. Based on this, an implementation method for inputting into the mask-guided sub-network for image fusion is provided, that is, "input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fusion image" in the above S301, as Figure 7 shown, including:

[0081] S401, input the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain an output feature map.

[0082] In the embodiment of the present application, as Figure 8 shown, Figure 8 is a schematic diagram of the mask-guided sub-network in one embodiment. The terminal can input the visible light image (Visible Image) and the infrared image (Infrared Image) into the feature extraction layer for feature extraction to obtain an output feature map. Among them, the feature extraction layer can be a DMG network (dual-channel multi-scale generate network, dual-channel multi-scale generation network).

[0083] In one of the embodiments, S401 includes:

[0084] Perform downsampling processing on the visible light image and the infrared image to obtain feature maps of different scales;

[0085] Stitch the feature maps of different scales to obtain multiple stitched feature maps;

[0086] Perform upsampling processing on the multiple stitched feature maps to obtain an output feature map.

[0087] In the embodiment of the present application, as Figure 9 shown, Figure 9 is a schematic diagram of the DMG network in one embodiment. The terminal can use the downsampling layer to perform downsampling processing on the visible light image and the infrared image, that is, feature maps of different scales can be extracted from the visible light image and the infrared image. It should be emphasized that although the network designs for the two branches of the visible light image and the infrared image in the downsampling layer are the same, the parameters on each branch are different, so as to allow each branch to retain the information from a single-modal image. Exemplarily, the operation on each feature in the downsampling layer can be formulated as the following formula (1):

[0088] (1)

[0089] Among them, is the input feature map, that is, the visible light image (vi) and the infrared image (ir) input into the downsampling layer. They are feature maps of different scales output. denotes Figure 9 multiple convolutional layers (Convolution Block) depicted in, where Conv, k3n8s1, Conv, k3n64s1, etc. refer to the specific parameters of the convolutional layer. Figure 9 All convolutional layers in are only examples, and the embodiments of this application do not limit the convolutional layer. represents leaky rectified linear activation. is a downsampling operation (DownSample), implemented by a max pooling operation.

[0090] Thus, the terminal can map feature maps of different scales and splice them on the channel dimension to obtain multiple spliced feature maps. After that, the terminal can use an upsampling layer to perform upsampling processing on the multiple spliced feature maps to obtain an output feature map. Among them, the upsampling layer is essentially a pre-fusion network. The upsampling layer integrates a large amount of information about the coarse-to-fine features extracted from multi-modal images. The upsampling layer is used to extract refined multi-modal information from the coarse-to-fine features of two modal images (i.e., visible light image and infrared image). Learning the coarse-to-fine features can improve the spatial representation ability of the fusion for the source images and is beneficial to avoiding the loss of details in the fused image. In addition, skip connections are designed in the upsampling layer. Through the skip connections, more semantic information can be retained from the coarse-to-fine features of multi-modal. Exemplarily, the upsampling layer can operate on each spliced feature map in the following manner (2):

[0091] (2)

[0092] where represents the feature map corresponding to the concatenate connection, represents the feature map corresponding to the upsampling connection, denotes a per-channel concatenation operation, and represents the nearest neighbor interpolation method. In formula (2), refers to the output feature map ( )

[0093] In this way, the feature extraction layer can extract feature information from two source images (i.e., visible light image and infrared image) respectively, so that the output feature map after preliminary fusion can retain more detailed information.

[0094] S402. Process the background information of the infrared image to obtain the mask image of the infrared image.

[0095] In the embodiments of the present application, since the main source of thermal radiation is the heat source, and the corresponding region of the infrared image contains the most heat source information, and an object composed of a heat storage material also emits a small amount of infrared light due to the stored thermal energy. For example, although the transmission line region contributes little infrared information in the infrared image, the transmission line region provides rich texture information in the corresponding visible light image. Therefore, there are differences in the thermal radiation intensities of the visible light image and the infrared image. Based on this, combined with Figure 8 as shown, the terminal can perform suppression processing on the infrared background information of the infrared image to obtain a mask image (Target Mask) of the infrared image. The process of obtaining the mask image of the infrared image follows a truncation function, as shown in the following formula (3):

[0096] (3)

[0097] where x, y represent the positions of each pixel, and represents the pixel (x, y) on the mask image. represents the pixel value of the infrared image, and t is the threshold of the truncation operation.

[0098] It should be noted that the embodiments of the present application do not limit the sequence of S401 and S402.

[0099] S403, input the output feature map and the mask image into a fusion layer for image fusion to obtain an intermediate fusion image.

[0100] In the embodiments of the present application, combined with Figure 8 as shown, the terminal can input the output feature map and the mask image into a fusion layer (Mask - Guided Network) for image fusion to obtain an intermediate fusion image (Fused Image). Exemplarily, the feature map of the thermal target mask (i.e., the mask image) and the multi - modal image feature map (i.e., the output feature map) can be cascaded first, so that stress can be applied to the items emitting high - intensity thermal radiation during the fusion process, and then a stacked convolutional layer is used to reconstruct the intermediate fusion image. Among them, behind each convolutional layer is a batch normalization layer and a Leakly ReLU activation function, and the stride of each convolutional layer is 1 to ensure that the fusion image has the same resolution as the source image. Figure 8 Convk1n128s1, Conv k3n64s1, etc. in Figure 8 refer to the specific parameters of the convolutional layer. All the convolutional layers in

[0101] In this embodiment, two source images can be input into the DMG network, and the feature information in the two modal images can be completely extracted in the DMG network. In addition, a feature map of the thermal target mask can be created by suppressing the background information in the infrared image. Thus, in the mask-guided subnetwork, the obtained thermal target mask can be used to guide the feature information extracted from the DMG network for image fusion to implement the feature refinement process. The cross-modal interference can be reduced by the guidance of the thermal target mask, and it can be ensured that the intermediate fused image is rich in multi-modal information.

[0102] In one embodiment, an implementation method for training the mask-guided subnetwork is provided, that is, in the above image fusion method, as Figure 10 shown, the training method of the mask-guided subnetwork includes:

[0103] S501, obtaining a sample data set; the sample data set includes historical fused images, historical visible light images, and historical infrared images.

[0104] In the embodiment of the present application, the terminal can obtain the sample data set from the database or the image acquisition device in advance. Among them, the sample data set includes historical fused images, historical visible light images, and historical infrared images. The historical fused image refers to the fused image within the historical time period, the historical visible light image refers to the visible light image within the historical time period, and the historical infrared image refers to the infrared image within the historical time period.

[0105] S502, determining the content loss according to the historical fused image, the historical visible light image, and the historical infrared image.

[0106] In the embodiment of the present application, the terminal can determine the content loss according to the historical fused image, the historical visible light image, and the historical infrared image. Among them, the content loss is a consistency constraint that allows the fused image to display visible light and infrared characteristics. The content loss includes intensity loss and gradient loss . Since the goal of the embodiment of the present application is to extract all information in the source image as completely as possible through the network, there is no need to distinguish the primary and secondary relationships between the gradient loss and the intensity loss on the source image here. Exemplarily, the content loss can be expressed by the following formulas (4)-(8):

[0107] (4)

[0108] (5)

[0109] (6)

[0110] (7)

[0111] (8)

[0112] Among them, represents the historical fused image, represents the historical infrared image, represents the historical visible light image, H represents the height of the image, and W represents the width of the image. represents the Frobenius norm operator on the attention matrix, represents the gradient operation.

[0113] S503. Determine the first similarity loss according to the mask images of the historical fused image, the historical visible light image, and the historical infrared image.

[0114] In the embodiments of the present application, the terminal can determine the first similarity loss according to the mask images of the historical fused image, the historical visible light image, and the historical infrared image. Among them, the first similarity loss can be the structural similarity (SSIM) loss. Since there are sporadic thermal radiation regions in the background of the infrared image, the fused image is often interfered by these regions and is prone to serious image distortion. Therefore, in the embodiments of the present application, the thermal target mask (i.e., the mask image) is used to replace the original infrared image to calculate the first similarity loss, which can suppress the information interference in the multi-modal image fusion process. Therefore, using the SSIM loss can ensure that the fused image meets human vision and can maintain sufficient thermal target information in the fused image. Exemplarily, the first similarity loss is defined as shown in the following formula (9):

[0115] (9)

[0116] Among them, I ms represents the mask image of the historical infrared image, represents the historical fused image, represents the historical infrared image.

[0117] In the test stage, the acquisition of the fused image follows the following formula (10):

[0118] (10)

[0119] Among them, is the fusion layer (Mask-Guided Network), is the parameter in the fusion layer (Mask-Guided Network), is the DMG network, is the parameter in the DMG network.

[0120] S504. Train the initial mask guidance sub-network based on the first similarity loss and the content loss to obtain the mask guidance sub-network.

[0121] In the embodiments of the present application, the terminal can determine the target loss according to the first similarity loss and the content loss, and train the initial mask guidance sub-network according to the target loss to obtain the mask guidance sub-network. Exemplarily, the determination method of the target loss is shown in the following formula (11):

[0122] (11)

[0123] Where is a hyperparameter used to balance the first similarity loss and the content loss during the model training process.

[0124] In this embodiment, the initial mask guidance sub-network can be jointly trained according to the first similarity loss and the content loss, and a more accurate mask guidance sub-network can be obtained.

[0125] In one embodiment, an implementation method for training the contrastive learning sub-network is provided, that is, in the above image fusion method, as Figure 11 shown, the training method of the contrastive learning sub-network includes:

[0126] S601. Obtain a sample data set; the sample data set includes historical fused images, historical visible light images, and historical infrared images.

[0127] In the embodiments of the present application, the terminal can pre-obtain the sample data set from a database or an image acquisition device. Among them, the sample data set includes historical fused images, historical visible light images, and historical infrared images. The historical fused image refers to the fused image within a historical time period, the historical visible light image refers to the visible light image within a historical time period, and the historical infrared image refers to the infrared image within a historical time period.

[0128] S602. Extract query slices from the historical fused images.

[0129] S603. Extract positive slices and negative slices from the historical visible light images and historical infrared images according to the positions of the query slices in the historical fused images.

[0130] In the embodiments of the present application, Noise-Contrastive Estimation (NCE) is a new estimation principle for parametric statistical models. Its core idea is to determine certain characteristics of the original data by learning the differences between the original data distribution samples and the selected noise distribution, which can effectively simplify the model estimation problem into a binary problem and greatly reduce the computational complexity. Based on the idea of NCE, the concept of mutual information is introduced, and the embodiments of the present application can propose a new form of contrast loss function, called InfoNCE. Specifically, an unsupervised block contrast learning framework can be constructed. Since the image fusion task pays more attention to the significant targets of infrared images and the background texture information of visible light images, the contrast learning task based on image patches can be constructed from the local features of the images. Exemplarily, as Figure 12 shown Figure 12 FIG. is a schematic diagram of a contrast learning sub-network in an embodiment. The terminal can extract a query slice, a positive slice corresponding to the historical visible light image, and a positive slice corresponding to the historical infrared image at the same position in the historical fusion image, the historical visible light image, and the historical infrared image respectively, and select N negative slices from other positions of the infrared image and the visible light image according to the positions of the query slice and the positive slices.

[0131] S604. Train the initial contrast learning sub-network according to the sample data set, the query slice, the positive slice, and the negative slices to obtain the contrast learning sub-network.

[0132] In the embodiments of the present application, the terminal can extract a query slice from the historical fusion image and extract a set of encoded samples from the historical visible light image and the historical infrared image according to the position of the query slice in the historical fusion image , and this set of encoded samples includes a positive slice and multiple negative slices. Thus, the terminal can map the query slice, the positive slice, and the N negative slices to K-dimensional vectors, where , and , represents the nth negative slice. In this way, the initial contrast learning sub-network can be trained according to the sample data set, the query slice, the positive slice, and the negative slices. When the vector q corresponding to the query slice is similar to the positive example but not similar to all other negative examples , the value of the InfoNCE loss will be very small, and the contrast learning sub-network at this time can be determined as the trained contrast learning sub-network. Exemplarily, the InfoNCE loss is defined as shown in the following formula (12):

[0133] (12)

[0134] In one of the embodiments, S604 includes:

[0135] Encode the query slice, positive slice, and negative slice respectively to obtain the first feature vector, second feature vector, and third feature vector.

[0136] Determine the contrastive loss according to the similarity between the first feature vector, second feature vector, and third feature vector.

[0137] Determine the second similarity loss according to the historical fused image, historical visible light image, and historical infrared image.

[0138] Train the initial contrastive learning sub-network according to the contrastive loss and the second similarity loss to obtain the contrastive learning sub-network.

[0139] In the embodiment of the present application, as shown in Figure 12 , the terminal can use an infrared encoder and a visible light encoder, and add two layers of MLP (Multilayer Perceptron) network to encode the query slice, positive slice, and negative slice respectively to obtain the first feature vector, second feature vector, and third feature vector. Among them, the infrared encoder, visible light encoder, and MLP network are used to encode any spatial position in the patch source image and the fused image to obtain the feature vector. For example, the positive slices in the historical fused image, historical visible light image, and historical infrared image can be encoded as the feature vectors query, and . Thus, the terminal can determine the contrastive loss according to the similarity between the first feature vector, second feature vector, and third feature vector. Exemplarily, the similarity between the feature vector query corresponding to the historical fused image and the feature vector corresponding to the historical infrared image can be calculated respectively, and / or the similarity between the feature vector query corresponding to the historical fused image and the feature vector corresponding to the historical visible light image can be calculated, and the feature vector most similar to the feature vector query is selected to calculate the contrastive loss. Among them, the contrastive loss is used to directly utilize the encoded deep representation to prompt the encoder to retain sufficient complementary information.

[0140] Exemplarily, as shown in Figure 12 , the two encoders and that the terminal can use can be used to extract the effective feature stack. At the same time, the terminal can map the feature stack through a small neural network projection head H (that is, two layers of MLP), that is, it can be understood that the terminal can use the encoder to encode the historical infrared image, and use the encoder Encode the historical visible light images and use the above two encoders to encode the historical fused images, so as to obtain four feature sequences as shown in the following formula (13):

[0141] (13)

[0142] Thus, the similarity can be calculated based on the above four feature sequences to obtain the contrast loss. Exemplarily, the calculation formula of the patchNCE loss at any specific spatial position is as shown in the following formula (14):

[0143] (14)

[0144] Where , S is the number of spatial positions sampled from the last image feature layer. For any specific spatial position in the image feature level, the patch feature is called , is the general term of the set , and the remaining features in the same feature level are called , where C is the number of channels.

[0145] It should be noted that both the positive and negative samples used to calculate the InfoNCE loss are sampled from the source image and the fused image encoded by the same encoder. For the feature extraction network, it can be expected that the infrared encoder can retain more significant target features, and the visible light encoder can retain more detailed texture features, which are the most obvious complementary features between infrared and visible light images. The contrast loss pays more attention to the deep features extracted by the encoder. As the training process progresses, the patchNCE loss can effectively adjust the encoder and the projection head to retain the most similar parts of the source image and the fused image.

[0146] In addition, the terminal can also determine the second similarity loss according to the historical fused image, the historical visible light image, and the historical infrared image. Among them, the second similarity loss can be the SSIM loss. The second similarity loss mainly focuses on the structural features of the image itself. The SSIM loss combines image brightness, contrast, and structure to measure the image quality. Exemplarily, for any two images A and B, the expression of the SSIM loss is described as the following formula (15):

[0147] (15)

[0148] Where μ represents the mean of each pixel, δ represents the variance of each pixel, and . According to the above parameter settings, the SSIM loss can be set as the following formulas (16)-(18):

[0149] (16)

[0150] (17)

[0151] (18)

[0152] Among them, W represents a sliding window from the upper left to the lower right, with a step size of 1, represents the value of pixel i, m and n represent the size of the sliding window, and N represents the number of sliding windows in a single image. In the available work, the size of the window is . The SSIM loss is measured by normalizing the dot product between the query and other examples.

[0153] After that, the terminal can perform model training on the initial contrast learning sub-network according to the contrast loss and the second similarity loss to obtain the contrast learning sub-network. Exemplarily, the way to determine the total loss function according to the contrast loss and the second similarity loss can be defined as the following formula (19):

[0154] (19)

[0155] Among them, and are hyperparameters that control the loss balance between the two loss functions.

[0156] In this embodiment, slices can be randomly sampled and queried from historical fused images, and positive slices at the same position can be selected from historical infrared images and historical visible light images. Additionally, N random negative slices can be selected from other positions of historical infrared images and historical visible light images. Then, the infrared encoder and the visible light encoder are reused, and a two-layer MLP network is added to encode the positive slices in the historical fused images and historical source images into feature vector queries and perform feature fusion. Next, the similarity between the query slices, positive slices, and / or negative slices is calculated to compute the contrast loss, thereby maintaining the consistency of the deep features between the input and output. Furthermore, the second similarity loss based on the average brightness of local window pixels can not only retain the significant targets in the infrared image but also retain the bright areas and some significant textures in the visible light image. Additionally, the second similarity loss can utilize the shallow features of the image structure (i.e., structural similarity) to ensure the structural consistency between the input and output. In this way, the initial contrast learning sub-network can be jointly trained according to the contrast loss and the second similarity loss, and the loss function can guide the feature extraction network to adaptively acquire the significant targets in the infrared image and the background textures in the visible light image, thus enabling a more accurate contrast learning sub-network. In this manner, the fused image can selectively retain the most similar features in the local regions of different source images, that is, it can promote the fused image to selectively retain the most similar features from different source images.

[0157] In an alternative embodiment, as Figure 13 shown, an image fusion method is provided, which is applied to a terminal and includes:

[0158] S21, obtaining visible light images and infrared images of a target component in a power transmission system during operation;

[0159] S22, performing downsampling processing on the visible light images and infrared images to obtain feature maps of different scales;

[0160] S23, splicing the feature maps of different scales to obtain multiple spliced feature maps;

[0161] S24, performing upsampling processing on the multiple spliced feature maps to obtain an output feature map;

[0162] S25, performing background information processing on the infrared image to obtain a mask image of the infrared image;

[0163] S26, inputting the output feature map and the mask image into a fusion layer for image fusion to obtain an intermediate fused image;

[0164] S27, inputting the intermediate fused image, the visible light image, and the infrared image into a contrast learning sub-network for contrast learning to obtain a fused image; the contrast learning sub-network is trained based on the second similarity loss and the contrast loss.

[0165] Among them, the specific implementation manners of S21 - S27, the specific training method of the mask guidance sub - network, and the specific training method of the contrast learning sub - network have been introduced in the above embodiments and will not be elaborated here. It should be noted that the sequence of S22 - S24 and S25 is not limited.

[0166] In an exemplary embodiment, the method of the present application can be implemented by programming in Python language under a deep learning framework. Specifically, first, obtain a sample data set. The power grid fusion data sets used in the embodiments of the present application are historical infrared light images and historical visible light images respectively. The total number of images in each modality is 542, among which the number of training set data is 285, the number of validation set data is 66, and the number of test set data is 191. The image size is 256×256. During the training process, a certain data augmentation process can be performed on the training set first, so as to expand and obtain a more abundant sample data set. Second, complete the network framework design of the above - mentioned image fusion method. Third, send the training data into the initial image fusion network for training, and save the optimized network weights. This algorithm is an end - to - end unsupervised image fusion algorithm. There is no need for manual intervention in the middle process. The training and validation data sets are directly input for learning. The algorithm will automatically optimize the network through loss constraints. After the loss gradually decreases and stabilizes, the training ends and the optimal weights are saved. Among them, the training process of the mask guidance sub - network first undergoes 30 epochs of training. On this basis, without fixing the parameters, the training process of the contrast learning sub - network is directly started, and they are jointly trained for 270 epochs.

[0167] Fourth, use the trained network weights to perform image fusion tests on historical visible light images and historical infrared images in the power grid. That is to say, the test set or other images to be fused are input into the network, and the optimal weights reserved during the training process are applied for experiments to obtain the final registration result and objective indicators. For example, only the fusion network in the first stage is used in the test phase. The objective indicators applied in the embodiments of this application are EN, MI, VIF, and SF. Among them, EN is entropy, which is mainly an objective evaluation indicator for evaluating the information content of an image. If the entropy increases after fusion, it indicates an increase in fusion information and an improvement in fusion performance; MI is mutual information, which is an indicator for measuring the correlation between two random variables from the perspective of information theory and is used to evaluate the information sharing degree between two images, generally used for multi-modal image fusion; VIF is Visual Information Fidelity, which is an indicator for quantifying the information shared between the fused image and the source image based on natural scene statistics and the human visual system. The higher the VIF, the more the fusion result conforms to human visual perception; SF is spatial frequency, which is used to measure the richness of spatial details in an image. SF reveals the details and texture information of the fused image by measuring the gradient distribution of the fused image. A higher SF means richer edge and texture details. The comparison algorithms used in the embodiments of this application are DenseFuse, FusionGAN, GANMcC, and U2Fusion. The following table shows the comparison of objective indicators between the algorithm of this application and the four comparison algorithms. It can be seen that under each indicator, the algorithm of the present invention has the best effect, and the image quality and fusion performance both exceed the existing multi-modal image fusion algorithms.

[0168] Table 1

[0169]

[0170] In the above image fusion method, obtain visible light images and infrared images of target components during the operation of the power transmission system; input the visible light images and infrared images into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on contrast loss, similarity loss, and content loss. Since the image fusion network in the embodiments of this application is trained based on contrast loss, similarity loss, and content loss, during the model training process, multiple loss functions can be used to more comprehensively utilize the shallow features and deep features in the visible light images and infrared images for model training. Therefore, a more accurate image fusion network can be trained. Thus, a more accurate image fusion network can be used to perform image fusion on the visible light images and infrared images, and a more accurate fused image can be obtained.

[0171] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be completed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0172] Based on the same inventive concept, an embodiment of the present application also provides an image fusion device for implementing the above-mentioned image fusion method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following image fusion devices can refer to the limitations on the image fusion method in the above text, and will not be repeated here.

[0173] In an exemplary embodiment, as Figure 14 shown, an image fusion device is provided, including: an acquisition module 31 and a fusion module 32, where:

[0174] The acquisition module 31 is used to acquire visible light images and infrared images of a target component in a power transmission system during operation;

[0175] The fusion module 32 is used to input the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on a contrast loss, a similarity loss, and a content loss.

[0176] In one of the embodiments, the image fusion network includes a mask-guided sub-network and a contrast learning sub-network, the similarity loss includes a first similarity loss and a second similarity loss, and the fusion module 32 includes:

[0177] A fusion unit is used to input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fused image; the mask-guided sub-network is trained based on the first similarity loss and the content loss;

[0178] A contrast learning unit is used to input the intermediate fused image, the visible light image, and the infrared image into the contrast learning sub-network for contrast learning to obtain a fused image; the contrast learning sub-network is trained based on the second similarity loss and the contrast loss.

[0179] In one embodiment, the mask-guided sub-network includes a feature extraction layer and a fusion layer. The fusion unit includes:

[0180] A feature extraction sub-unit, configured to input the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain an output feature map;

[0181] A background information processing sub-unit, configured to perform background information processing on the infrared image to obtain a mask image of the infrared image;

[0182] A fusion sub-unit, configured to input the output feature map and the mask image into the fusion layer for image fusion to obtain an intermediate fusion image.

[0183] In one embodiment, the feature extraction sub-unit is specifically configured to:

[0184] Perform downsampling processing on the visible light image and the infrared image to obtain feature maps of different scales;

[0185] Stitch the feature maps of different scales to obtain a plurality of stitched feature maps;

[0186] Perform upsampling processing on the plurality of stitched feature maps to obtain an output feature map.

[0187] In one embodiment, the above image fusion device further includes:

[0188] A first sample data set acquisition module, configured to acquire a sample data set; the sample data set includes historical fusion images, historical visible light images, and historical infrared images;

[0189] A content loss determination module, configured to determine a content loss according to the historical fusion image, the historical visible light image, and the historical infrared image;

[0190] A first similarity loss determination module, configured to determine a first similarity loss according to the historical fusion image, the historical visible light image, and the mask image of the historical infrared image;

[0191] A first training module, configured to perform model training on the initial mask-guided sub-network according to the first similarity loss and the content loss to obtain the mask-guided sub-network.

[0192] In one embodiment, the above image fusion device further includes:

[0193] A first sample data set acquisition module, configured to acquire a sample data set; the sample data set includes historical fusion images, historical visible light images, and historical infrared images;

[0194] A first extraction module, configured to extract query slices from the historical fusion image;

[0195] A second extraction module, configured to extract positive slices and negative slices from the historical visible light image and the historical infrared image according to the position of the query slice in the historical fused image;

[0196] A second training module, configured to train an initial contrast learning sub-network according to the sample data set, the query slice, the positive slice, and the negative slice to obtain a contrast learning sub-network.

[0197] In one embodiment, the second training module includes:

[0198] An encoding processing unit, configured to perform encoding processing on the query slice, the positive slice, and the negative slice respectively to obtain a first feature vector, a second feature vector, and a third feature vector;

[0199] A contrast loss determination unit, configured to determine a contrast loss according to the similarity between the first feature vector, the second feature vector, and the third feature vector;

[0200] A second similarity loss determination unit, configured to determine a second similarity loss according to the historical fused image, the historical visible light image, and the historical infrared image;

[0201] A second training unit, configured to train an initial contrast learning sub-network according to the contrast loss and the second similarity loss to obtain a contrast learning sub-network.

[0202] Each module in the above image fusion device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0203] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 15As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. The computer program, when executed by the processor, implements an image fusion method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0204] Those skilled in the art can understand that Figure 15 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0205] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0206] Obtain visible light images and infrared images of the target component during the operation of the power transmission system;

[0207] Input the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on contrast loss, similarity loss, and content loss.

[0208] In one embodiment, the image fusion network includes a mask-guided sub-network and a contrastive learning sub-network. The similarity loss includes a first similarity loss and a second similarity loss. The visible light image and the infrared image are input into a preset image fusion network for fusion to obtain a fused image. When the processor executes the computer program, the following steps are further implemented:

[0209] Input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fused image; the mask-guided sub-network is trained based on the first similarity loss and the content loss;

[0210] Input the intermediate fused image, the visible light image, and the infrared image into the contrastive learning sub-network for contrastive learning to obtain a fused image; the contrastive learning sub-network is trained based on the second similarity loss and the contrast loss.

[0211] In one embodiment, the mask-guided sub-network includes a feature extraction layer and a fusion layer. When the processor executes the computer program to input the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fused image, the following steps are further implemented:

[0212] Input the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain an output feature map;

[0213] Perform background information processing on the infrared image to obtain a mask image of the infrared image;

[0214] Input the output feature map and the mask image into the fusion layer for image fusion to obtain an intermediate fused image.

[0215] In one embodiment, when the processor executes the computer program to input the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain an output feature map, the following steps are further implemented:

[0216] Perform downsampling processing on the visible light image and the infrared image to obtain feature maps of different scales;

[0217] Stitch the feature maps of different scales to obtain multiple stitched feature maps;

[0218] Perform upsampling processing on the multiple stitched feature maps to obtain an output feature map.

[0219] In one embodiment, for the training method of the mask-guided sub-network, when the processor executes the computer program, the following steps are further implemented:

[0220] Obtain a sample data set; the sample data set includes historical fused images, historical visible light images, and historical infrared images;

[0221] Determine content loss based on the historical fused image, historical visible light image, and historical infrared image;

[0222] Determine the first similarity loss based on the mask images of the historical fused image, historical visible light image, and historical infrared image;

[0223] Train the initial mask guidance sub-network based on the first similarity loss and content loss to obtain the mask guidance sub-network.

[0224] In one embodiment, for the training method of the contrastive learning sub-network, when the processor executes the computer program, the following steps are further implemented:

[0225] Obtain a sample data set; the sample data set includes the historical fused image, historical visible light image, and historical infrared image;

[0226] Extract query slices from the historical fused image;

[0227] Extract positive slices and negative slices from the historical visible light image and historical infrared image according to the positions of the query slices in the historical fused image;

[0228] Train the initial contrastive learning sub-network based on the sample data set, query slices, positive slices, and negative slices to obtain the contrastive learning sub-network.

[0229] In one embodiment, for training the initial contrastive learning sub-network based on the sample data set, query slices, positive slices, and negative slices to obtain the contrastive learning sub-network, when the processor executes the computer program, the following steps are further implemented:

[0230] Perform encoding processing on the query slices, positive slices, and negative slices respectively to obtain a first feature vector, a second feature vector, and a third feature vector;

[0231] Determine the contrastive loss according to the similarity between the first feature vector, the second feature vector, and the third feature vector;

[0232] Determine the second similarity loss based on the historical fused image, historical visible light image, and historical infrared image;

[0233] Train the initial contrastive learning sub-network based on the contrastive loss and the second similarity loss to obtain the contrastive learning sub-network.

[0234] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0235] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.

[0236] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., and are not limited thereto.

[0237] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope recorded in the present application.

[0238] The above embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An image fusion method, characterized in that, The method includes: Obtaining visible light images and infrared images of a target component during the operation of a power transmission system; Inputting the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on a contrast loss, a similarity loss, and a content loss.

2. The method according to claim 1, wherein The image fusion network includes a mask-guided sub-network and a contrast learning sub-network, the similarity loss includes a first similarity loss and a second similarity loss, and the inputting the visible light image and the infrared image into the preset image fusion network for fusion to obtain a fused image includes: Inputting the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fused image; the mask-guided sub-network is trained based on the first similarity loss and the content loss; Inputting the intermediate fused image, the visible light image, and the infrared image into the contrast learning sub-network for contrast learning to obtain the fused image; the contrast learning sub-network is trained based on the second similarity loss and the contrast loss.

3. The method according to claim 2, characterized in that The mask-guided sub-network includes a feature extraction layer and a fusion layer, and the inputting the visible light image and the infrared image into the mask-guided sub-network for image fusion to obtain an intermediate fused image includes: Inputting the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain an output feature map; Performing background information processing on the infrared image to obtain a mask image of the infrared image; Inputting the output feature map and the mask image into the fusion layer for image fusion to obtain the intermediate fused image.

4. The method according to claim 3, wherein The inputting the visible light image and the infrared image into the feature extraction layer for feature extraction to obtain an output feature map includes: Performing downsampling processing on the visible light image and the infrared image to obtain feature maps of different scales; Stitching the feature maps of different scales to obtain a plurality of stitched feature maps; Performing upsampling processing on the plurality of stitched feature maps to obtain the output feature map.

5. The method according to any one of claims 2-4, characterized in that The training method of the mask-guided sub-network includes: Obtaining a sample data set; the sample data set includes historical fused images, historical visible light images, and historical infrared images; Determining the content loss according to the historical fused images, the historical visible light images, and the historical infrared images; Determining the first similarity loss according to the historical fused images, the historical visible light images, and the mask images of the historical infrared images; Performing model training on an initial mask-guided sub-network according to the first similarity loss and the content loss to obtain the mask-guided sub-network.

6. The method according to any one of claims 2-4, characterized in that, The training method of the contrast learning sub-network includes: Obtaining a sample data set; the sample data set includes historical fused images, historical visible light images, and historical infrared images; Extracting query slices from the historical fused images; Extracting positive slices and negative slices from the historical visible light images and the historical infrared images according to the positions of the query slices in the historical fused images; Training the initial contrastive learning sub-network according to the sample data set, the query slice, the positive slice, and the negative slice to obtain the contrastive learning sub-network.

7. The method according to claim 6, wherein The training of the initial contrastive learning sub-network according to the sample data set, the query slice, the positive slice, and the negative slice to obtain the contrastive learning sub-network includes: Performing encoding processing on the query slice, the positive slice, and the negative slice respectively to obtain a first feature vector, a second feature vector, and a third feature vector; Determining the contrastive loss according to the similarity between the first feature vector, the second feature vector, and the third feature vector; Determining the second similarity loss according to the historical fused image, the historical visible light image, and the historical infrared image; Training the initial contrastive learning sub-network according to the contrastive loss and the second similarity loss to obtain the contrastive learning sub-network.

8. An image fusion device, characterized in that, The device includes: An acquisition module for acquiring visible light images and infrared images of a target component in a power transmission system during operation; A fusion module for inputting the visible light image and the infrared image into a preset image fusion network for fusion to obtain a fused image; the image fusion network is trained based on contrastive loss, similarity loss, and content loss.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images