Image fusion method, device, equipment and storage medium

Abstract

The application discloses an image fusion method and device, equipment and a storage medium, wherein the method extracts image features by using an encoder after obtaining an infrared image and a visible light image, then fuses corresponding image features to obtain fusion features, finally decodes the fusion features by using a decoder to obtain a reconstructed image, wherein the encoder comprises a first convolution block, a second convolution block and a threshold generation network, the first convolution block convolves an image to obtain a first feature mapping, the second convolution block convolves the first feature mapping to obtain a second feature mapping, the threshold generation network generates a soft threshold according to the second feature mapping, and the image feature is calculated according to the first feature mapping, the second feature mapping and the soft threshold. The application introduces a threshold generation network in the encoder, completes feature enhancement while extracting features of the infrared image and the visible light image, and effectively reduces noise of the original image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image fusion method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous development of science and technology, the acquired images gradually tend to be high information, high resolution and multi-source. A single information processing method cannot comprehensively integrate the characteristics of multi-source information to meet the rich analysis and processing requirements, thus giving birth to the image fusion processing technology. The image fusion technology is a process of integrating image information acquired by the same or different sensors under the same target scene to obtain a fusion image containing the imaging characteristics of different sensors and fully reflecting the scene information. The infrared imaging technology can obtain clear thermal radiation targets without being affected by light, climate and the like, but the background of the image is blurred and the texture information is missing. The visible light imaging technology obtains an image with rich details by reflecting visible light, but the imaging conditions are harsh and are easily affected by natural climate. The infrared and visible light image fusion technology can obtain a single image containing clear thermal imaging targets and rich background information by merging and processing infrared and visible light images, and has been widely applied in various fields such as military reconnaissance, target recognition and tracking, security monitoring, agricultural production and remote sensing measurement.

[0003] In recent years, with the development of deep learning technology, infrared and visible light image fusion algorithms can be divided into two categories: traditional image processing theory-based methods and deep learning-based methods. Representative traditional methods include multi-scale transformation, sparse representation, etc. Multi-scale transformation decomposes source images into different scales through pyramid, wavelet, contourlet, etc., and then obtains the result image through inverse transformation. Sparse representation enhances meaningful features in the process of sparse representation of source images through a learned over-complete dictionary. These methods have achieved good fusion performance, but problems such as the need for manual feature creation and high computational complexity still exist. Deep learning-based methods include convolutional neural network-based methods, generative adversarial network-based methods and autoencoder network-based methods according to different network frameworks. However, the original infrared and visible light image features extracted by the convolutional network are not selectively selected and enhanced, and for the neural network model, the influence of the original features' slight noise and defects will gradually increase after being processed by multiple layers of network, resulting in distortion and blur of the final output image of the network. SUMMARY

[0004] Therefore, the present application provides an image fusion method, device, equipment and storage medium to solve the problem of poor image fusion effect.

[0005] To solve the above technical problems, one technical solution adopted by the present application is to provide an image fusion method, comprising: acquiring a first image and a second image, the first image and the second image belonging to different image types; inputting the first image and the second image into an encoder of a pre-trained image fusion model for feature extraction to obtain first image features and second image features, the encoder comprising a first convolutional block, a second convolutional block and a threshold generation network, the first convolutional block convolving the image to obtain a first feature map, the second convolutional block convolving the first feature map to obtain a second feature map, the threshold generation network generating a soft threshold according to the second feature map, and the image features being calculated according to the first feature map, the second feature map and the soft threshold; using a fusion layer of the image fusion model to perform weighted fusion on the first image features and the second image features to obtain fused image features; and inputting the fused image features into a decoder of the image fusion model for decoding to obtain a final reconstructed image.

[0006] As a further improvement of the present application, the process of the convolutional block convolving the image to obtain the feature map is represented as:

[0007]

[0008] wherein, represents the feature map generated by the kth convolutional block convolving the image, a (m) represents the convolution matrix of the convolutional block, b (m) represents the bias parameter of the convolutional block, and m represents the number of channels, represents convolution, and δ represents the activation function.

[0009] As a further improvement of the present application, the threshold generation network comprises an absolute value processing and global average pooling layer, a first fully connected layer, a batch normalization layer, a ReLU activation function, a second fully connected layer and a sigmoid activation function, and the generation process of the soft threshold is represented as:

[0010]

[0011] wherein, α represents the soft threshold, F represents fully connected calculation, sigmoid represents the sigmoid activation function, δ represents the ReLU activation function, and W represents the convolution matrix, represents a third feature map obtained by globally average pooling the absolute value of the second feature map, and m represents the number of channels.

[0012] As a further improvement of the present application, the image features are calculated according to the first feature map, the second feature map and the soft threshold, comprising:

[0013]

[0014] wherein, fRSB represents an image feature, and a represents a soft threshold value, represents a third feature map obtained by performing global average pooling on absolute values of the second feature map, represents a first feature map, and m represents a number of channels.

[0015] As a further improvement of the present application, the method further comprises pre-training the encoder and the decoder, specifically comprising: obtaining a first sample image and a second sample image, the first sample image and the second sample image belonging to different image types; inputting the first sample image or the second sample image into the encoder to extract features, to obtain a first sample image feature or a second sample image feature; inputting the first sample image feature or the second sample image feature into the decoder to decode, to obtain a first sample reconstructed image or a second sample reconstructed image; training the encoder and the decoder based on the first sample reconstructed image or the second sample reconstructed image and a preset loss function, the preset loss function comprising a first loss function for optimizing pixel information of the reconstructed image, a second loss function for optimizing structural similarity information of the reconstructed image, a third loss function for optimizing gradient information of the reconstructed image, and a fourth loss function for optimizing color information of the reconstructed image.

[0016] As a further improvement of the present application, the preset loss function is represented as:

[0017] Loss=aL pixel +bL ssim +cL grad +dL color ;

[0018] Loss represents the preset loss function, L pixel represents the first loss function, L ssim represents the second loss function, L grad represents the third loss function, L color represents the fourth loss function, and a, b, c, d represent preset hyperparameters.

[0019] As a further improvement of the present application, the first loss function is represented as:

[0020] L pixel =MSE(I ir ,I re1 )+MSE(I vr ,I re2 );

[0021]

[0022]

[0023] wherein, L pixelrepresents a first loss function, MSE represents a mean square error function, I ir represents a first sample image, I re1 represents a first sample reconstructed image, I vr represents a second sample image, I re2 represents a second sample reconstructed image, N represents a number of pixels.

[0024] A second loss function is represented as:

[0025]

[0026]

[0027] wherein, L ssim represents a second loss function, SSIM(X, F) represents a similarity between a sample image and a sample reconstructed image, μ x represents a mean of the sample image, μ f represents a mean of the sample reconstructed image, σ x represents a standard deviation of the sample image, σ f represents a standard deviation of the sample reconstructed image, σ xf represents a covariance of the sample image and the sample reconstructed image.

[0028] A third loss function is represented as:

[0029] L grad =MSE(Gradient(I re ), Gradient(I in ));

[0030]

[0031] wherein, L grad represents a third loss function, Gradient represents an image gradient map calculated by a Laplacian operator, I re represents a sample reconstructed image, I in represents a sample image, (x, y) represents a coordinate of a point of the image, and f(x, y) represents a pixel value of the point (x, y) of the image.

[0032] A fourth loss function value is represented as:

[0033]

[0034] wherein, L color represents a fourth loss function, and Histogram represents a color histogram of the image.

[0035] To solve the above technical problems, the present application adopts another technical solution: providing an image fusion device, comprising: an acquisition module for acquiring a first image and a second image, the first image and the second image belonging to different image types; a feature extraction module for inputting the first image and the second image into a pre-trained image fusion model encoder for feature extraction to obtain first image features and second image features, the encoder comprising a first convolution block, a second convolution block and a threshold generation network, the first convolution block convolving the image to obtain a first feature map, the second convolution block convolving the first feature map to obtain a second feature map, the threshold generation network generating a soft threshold value according to the second feature map, and the image features being calculated according to the first feature map, the second feature map and the soft threshold value; a fusion module for weighting and fusing the first image features and the second image features using the fusion layer of the image fusion model to obtain fused image features; and a reconstruction module for inputting the fused image features into the decoder of the image fusion model for decoding to obtain a final reconstructed image.

[0036] To solve the above technical problems, the present application adopts another technical solution: providing a computer device, comprising a processor, a memory coupled to the processor, and program instructions stored in the memory, which when executed by the processor, cause the processor to perform the steps of the image fusion method of any one of the above.

[0037] To solve the above technical problems, the present application adopts another technical solution: providing a storage medium storing program instructions capable of implementing the image fusion method of any one of the above.

[0038] The image fusion method of the present application has the following beneficial effects: the image fusion method of the present application extracts first image features and second image features using an encoder for different image types of the first image and the second image, then fuses the first image features and the second image features, and finally decodes the fused image features using a decoder to obtain a final reconstructed image, wherein the encoder comprises a first convolution block, a second convolution block and a threshold generation network, the first convolution block convolves the image to obtain a first feature map, the second convolution block convolves the first feature map to obtain a second feature map, the threshold generation network generates a soft threshold value according to the second feature map, and the image features are calculated according to the first feature map, the second feature map and the soft threshold value, which introduces a threshold generation network inside the encoder to complete feature enhancement when extracting features of infrared images and visible light images, while effectively reducing the noise of the original image, so that the final reconstructed image has higher imaging quality. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1is a flowchart of an image fusion method of an embodiment of the present application;

[0040] Figure 2 is a structural diagram of an image fusion model of an embodiment of the present application;

[0041] Figure 3 is a structural diagram of an encoder of an image fusion model of an embodiment of the present application;

[0042] Figure 4 is a flowchart of an image fusion model training process of an embodiment of the present application;

[0043] Figure 5 is a functional module diagram of an image fusion device of an embodiment of the present application;

[0044] Figure 6 is a structural diagram of a computer device of an embodiment of the present application;

[0045] Figure 7 is a structural diagram of a storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0047] The terms "first", "second", "third" in the present application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second", "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between the components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0048] Reference to an "embodiment" in this document means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As those skilled in the art will appreciate, embodiments described herein can be combined with other embodiments in various ways.

[0049] Figure 1 is a flowchart of an image fusion method according to an embodiment of the application. It should be noted that the method of the application is not limited to the order of the flowchart shown. Figure 1 As shown in the flowchart, the image fusion method comprises the steps of: Figure 1

[0050] Step S101: obtaining a first image and a second image, the first image and the second image belonging to different image types.

[0051] It should be noted that the image types in this embodiment include infrared image types and visible light image types. Specifically, the first image and the second image belong to different image types and are obtained by a visible light image sensor and an infrared light image sensor, respectively.

[0052] Step S102: inputting the first image and the second image into an encoder of a pre-trained image fusion model for feature extraction to obtain first image features and second image features, the encoder comprising a first convolutional block, a second convolutional block and a threshold generation network, the first convolutional block convolving the image to obtain a first feature map, the second convolutional block convolving the first feature map to obtain a second feature map, the threshold generation network generating a soft threshold according to the second feature map, and the image features being calculated according to the first feature map, the second feature map and the soft threshold.

[0053] It should be noted that, as shown in Figure 2 , Figure 2 shows a structural diagram of the image fusion model according to this embodiment, which comprises an encoder, a fusion layer and a decoder, wherein the encoder comprises at least one residual shrinkage block (RSB), preferably three residual shrinkage blocks connected in sequence, and the decoder comprises at least one convolutional block (BC), preferably three convolutional blocks connected in sequence.

[0054] Specifically, after obtaining the first image and the second image, the first image or the second image is input into the encoder for encoding to obtain first image features corresponding to the first image or second image features corresponding to the second image. Further, please refer to Figure 3 , Figure 3 ​The structural schematic diagram of the encoder of the embodiment is shown, each residual shrinkage block includes two first convolution blocks (BC1), a second convolution block (BC2) and a threshold generation network, the first convolution block and the second convolution block each include a convolution layer (Conv), a batch normalization layer (BatchNorm) and an activation function (ReLU / sigmoid). When the encoder encodes, first, the first image or the second image is input to the first convolution block for convolution operation to obtain a first feature map, then the first feature map is input to the second convolution block for convolution operation to obtain a second feature map, then the second feature map is input to the threshold generation network to generate a soft threshold, then the second feature map is processed by using the soft threshold, and then the first feature map and the second feature map processed by the soft threshold are superimposed through an identity link to obtain a first image feature or a second image feature.

[0055] wherein the process that the convolution block convolves the image to obtain the feature map is represented as:

[0056]

[0057] wherein, represents the feature map generated by the kth convolution block convolving the image, a (m) represents the convolution matrix of the convolution block, b (m) represents the bias parameter of the convolution block, and m represents the number of channels, represents convolution, and δ represents an activation function.

[0058] wherein, please further refer to Figure 3 the threshold generation network includes an absolute value processing and a global average pooling layer (Absolute GAP), a first full connection layer (FC1), a batch normalization layer (BN), a ReLU activation function (ReLU), a second full connection layer (FC2) and a sigmoid activation function (sigmoid), and the generation process of the soft threshold is represented as:

[0059]

[0060] wherein, α represents the soft threshold, F represents full connection calculation, sigmoid represents a sigmoid activation function, δ represents a ReLU activation function, and W represents a convolution matrix, represents a third feature map obtained by globally averaging the absolute value of the second feature map, and m represents the number of channels. The soft threshold can be normalized to between 0 and 1 through the sigmoid function finally.

[0061] Further, the image feature is calculated according to the first feature map, the second feature map and the soft threshold, including:

[0062]

[0063] Among them, f RSB Represents image features, where α represents the soft threshold. The third feature map is obtained by global average pooling of the absolute values ​​of the second feature map. Let m represent the first feature map, and m represent the number of channels.

[0064] Step S103: Use the fusion layer of the image fusion model to perform weighted fusion of the first image features and the second image features to obtain fused image features.

[0065] Specifically, after obtaining the first image features of the first image and the second image features of the second image, the first image features and the second image features are weighted and fused to obtain fused image features. In this embodiment, an average fusion strategy is preferably used to weight and fuse the first image features and the second image features, where the weights of the first image features and the second image features each account for 50%. This average fusion strategy can reduce computational overhead and lower the complexity of the image fusion model.

[0066] Step S104: Input the fused image features into the decoder of the image fusion model for decoding to obtain the final reconstructed image.

[0067] Specifically, after obtaining the fused image features, the fused image features are input to the decoder. The decoder performs a decoding operation on the fused image features to complete image reconstruction and obtain the final reconstructed image. In this embodiment, the decoder includes at least one convolutional block, and the structure of the convolutional block is the same as that of the first and second convolutional blocks. Preferably, the decoder also includes three convolutional blocks.

[0068] Furthermore, such as Figure 4 As shown, this image fusion method also includes a pre-trained encoder and decoder, specifically including:

[0069] Step S201: Obtain the first sample image and the second sample image, which belong to different image types.

[0070] Step S202: Input the first sample image or the second sample image into the encoder for feature extraction to obtain the features of the first sample image or the features of the second sample image.

[0071] Step S203: Input the features of the first sample image or the features of the second sample image into the decoder for decoding to obtain the first sample reconstructed image or the second sample reconstructed image.

[0072] Step S204: training the encoder and the decoder based on the first sample reconstructed image or the second sample reconstructed image and a preset loss function, the preset loss function including a first loss function for optimizing pixel information of the reconstructed image, a second loss function for optimizing structure similarity information of the reconstructed image, a third loss function for optimizing gradient information of the reconstructed image, and a fourth loss function for optimizing color information of the reconstructed image.

[0073] It should be noted that in the embodiment, when the image fusion model is trained, the first sample image or the second sample image belonging to different image types is used to train the encoder and the decoder of the image fusion model separately, that is, in the training stage, the image fusion model only includes the encoder and the decoder, which reduces the complexity of the image fusion model in the training stage and improves the training efficiency of the image fusion model.

[0074] Further, the preset loss function is represented as:

[0075] Loss = aL pixel +bL ssim +cL grad +dL color ;

[0076] Loss represents the preset loss function, L pixel represents the first loss function, L ssim represents the second loss function, L grad represents the third loss function, and L color represents the fourth loss function, and a, b, c, and d represent preset hyperparameters.

[0077] Further, the first loss function is represented as:

[0078] L pixel = MSE(I ir ,I re1 ) + MSE(I vr ,I re2 );

[0079]

[0080]

[0081] wherein L pixel represents the first loss function, MSE represents a mean square error function used to calculate the difference between the sample image and the sample reconstructed image, I ir represents the first sample image, I re1 represents the first sample reconstructed image, I vr represents the second sample image, I re2 represents the second sample reconstructed image, and N represents the number of pixel points.

[0082] The second loss function is represented as:

[0083]

[0084]

[0085] wherein L ssim represents the second loss function, SSIM(X, F) represents the similarity between the sample image and the sample reconstructed image, μ x represents the mean of the sample image, μ f represents the mean of the sample reconstructed image, σ x represents the standard deviation of the sample image, σ f represents the standard deviation of the sample reconstructed image, σ xf represents the covariance of the sample image and the sample reconstructed image.

[0086] The third loss function is represented as:

[0087] L grad = MSE(Gradient(I re ), Gradient(I in ));

[0088]

[0089] wherein L grad represents the third loss function, Gradient represents the image gradient map calculated by the Laplacian operator, I re represents the sample reconstructed image, I in represents the sample image, (x, y) represents the coordinates of the point of the image, and f(x, y) represents the pixel value of the point (x, y) of the image.

[0090] The fourth loss function value is represented as:

[0091]

[0092] wherein L color represents the fourth loss function, and Histogram represents the color histogram.

[0093] In the embodiment, a brand-new loss function is adopted in the training process of the model, which is composed of four parts of pixel information, structural similarity information, gradient information and color information, and comprehensively considers the influence of multiple angles on image fusion, so that the image fusion model obtained by training is suitable for different application scenarios.

[0094] The image fusion method of the embodiment of the application is characterized in that the first image and the second image of different image types are extracted to obtain the first image feature and the second image feature by using an encoder respectively, the first image feature and the second image feature are fused, the fused image feature is decoded by using a decoder to obtain the final reconstructed image, the encoder comprises a first convolution block, a second convolution block and a threshold generation network, the first convolution block is used to convolve the image to obtain a first feature mapping, the second convolution block is used to convolve the first feature mapping to obtain a second feature mapping, the threshold generation network is used to generate a soft threshold according to the second feature mapping, and the image feature is calculated according to the first feature mapping, the second feature mapping and the soft threshold. The threshold generation network is introduced into the encoder to enhance the features during the feature extraction of the infrared image and the visible light image, the noise of the original image is effectively reduced, and the imaging quality of the reconstructed image obtained finally is higher.

[0095] Figure 5 is a functional module schematic diagram of the image fusion device of the embodiment of the application. As shown in the figure, Figure 5 the image fusion device 20 comprises an acquisition module 21, a feature extraction module 22, a fusion module 23 and a reconstruction module 24.

[0096] The acquisition module 21 is used to acquire the first image and the second image, and the first image and the second image belong to different image types.

[0097] The feature extraction module 22 is used to input the first image and the second image into the encoder of the pre-trained image fusion model for feature extraction to obtain the first image feature and the second image feature. The encoder comprises a first convolution block, a second convolution block and a threshold generation network, the first convolution block is used to convolve the image to obtain a first feature mapping, the second convolution block is used to convolve the first feature mapping to obtain a second feature mapping, the threshold generation network is used to generate a soft threshold according to the second feature mapping, and the image feature is calculated according to the first feature mapping, the second feature mapping and the soft threshold.

[0098] The fusion module 23 is used to perform weighted fusion on the first image feature and the second image feature by using the fusion layer of the image fusion model to obtain the fused image feature.

[0099] The reconstruction module 24 is used to input the fused image feature into the decoder of the image fusion model for decoding to obtain the final reconstructed image.

[0100] Optionally, the process of convolving the image by the convolution block to obtain the feature mapping is represented as:

[0101]

[0102] wherein, denotes a feature map generated by convolving the k-th convolutional block on the image, a (m) denotes a convolution matrix of the convolutional block, b (m) denotes a bias parameter of the convolutional block, m denotes the number of channels, denotes a convolution, and δ denotes an activation function.

[0103] Optionally, the threshold generation network comprises an absolute value processing and global average pooling layer, a first fully connected layer, a batch normalization layer, a ReLU activation function, a second fully connected layer, and a sigmoid activation function, and a generation process of the soft threshold is represented as:

[0104]

[0105] wherein, α denotes a soft threshold, F denotes a fully connected calculation, sigmoid denotes a sigmoid activation function, δ denotes a ReLU activation function, and W denotes a convolution matrix, denotes a third feature map obtained by performing global average pooling on the absolute value of the second feature map, and m denotes the number of channels.

[0106] Optionally, the image feature is calculated according to the first feature map, the second feature map, and the soft threshold, and comprises:

[0107]

[0108] wherein, f RSB denotes an image feature, and α denotes a soft threshold, denotes a third feature map obtained by performing global average pooling on the absolute value of the second feature map, denotes a first feature map, and m denotes the number of channels.

[0109] Optionally, the image fusion device 20 further comprises a training module configured to pre-train the encoder and the decoder, and specifically comprises: obtaining a first sample image and a second sample image, the first sample image and the second sample image belonging to different image types; inputting the first sample image or the second sample image into the encoder to extract features, to obtain a first sample image feature or a second sample image feature; inputting the first sample image feature or the second sample image feature into the decoder to decode, to obtain a first sample reconstructed image or a second sample reconstructed image; and training the encoder and the decoder based on the first sample reconstructed image or the second sample reconstructed image and a preset loss function, the preset loss function comprising a first loss function for optimizing pixel information of the reconstructed image, a second loss function for optimizing structural similarity information of the reconstructed image, a third loss function for optimizing gradient information of the reconstructed image, and a fourth loss function for optimizing color information of the reconstructed image.

[0110] Optionally, the preset loss function is represented as:

[0111] Loss = aL pixel +bL ssim +cL grad +dL color ;

[0112] Loss represents a preset loss function, L pixel represents a first loss function, L ssim represents a second loss function, L grad represents a third loss function, L color represents a fourth loss function, and a, b, c, d represent preset hyperparameters.

[0113] Optionally, the first loss function is represented as:

[0114] L pixel = MSE(I ir , I re1 ) + MSE(I vr , I re2 );

[0115]

[0116]

[0117] wherein L pixel represents the first loss function, NSE represents that the difference between the sample image and the sample reconstructed image is calculated using the mean square error function, I ir represents the first sample image, I re1 represents the first sample reconstructed image, I vr represents the second sample image, I re2 represents the second sample reconstructed image, and N represents the number of pixel points.

[0118] The second loss function is represented as:

[0119]

[0120]

[0121] wherein L ssim represents the second loss function, SSIM(X, F) represents the similarity between the sample image and the sample reconstructed image, μ x represents the mean of the sample image, μ f represents the mean of the sample reconstructed image, σ x represents the standard deviation of the sample image, σ f represents the standard deviation of the sample reconstructed image, σ xf represents the covariance of the sample image and the sample reconstructed image.

[0122] The third loss function is represented as:

[0123] L grad = MSE(Gradient(I re , Gradient(I in ));

[0124]

[0125] wherein L grad represents the third loss function, Gradient represents an image gradient map calculated by a Laplacian operator, I re represents a sample reconstructed image, I in represents a sample image, (x, y) represents coordinates of a point of the image, and f(x, y) represents a pixel value of the point (x, y) of the image.

[0126] The fourth loss function value is represented as:

[0127]

[0128] wherein L color represents the fourth loss function, and Histogram represents a color histogram.

[0129] For other details of the implementation of the technical solutions of the modules in the image fusion device described above, refer to the description in the image fusion method in the above embodiments, which will not be repeated here.

[0130] It should be noted that each of the embodiments in the present specification adopts a progressive manner for description, and each embodiment focuses on the different places from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the part of the description of the method embodiment.

[0131] Please refer to Figure 6 , Figure 6 which is a structural schematic diagram of a computer device of the embodiment of the present application. As shown in the figure, the computer device 30 includes a processor 31 and a memory 32 coupled with the processor 31, the memory 32 stores program instructions, and the program instructions are executed by the processor 31 to make the processor 31 execute the image fusion method steps described in any of the above embodiments. Figure 6

[0132] ​The processor 31 can also be called a CPU (Central Processing Unit). The processor 31 can be an integrated circuit chip having a processing capability of signals. The processor 31 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0133] Referring to Figure 7 , Figure 7 is a structural schematic diagram of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 41 capable of realizing the image fusion method described above, wherein the program instructions 41 can be stored in the storage medium in the form of a software product, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and various media capable of storing program codes, or a computer, a server, a mobile phone, a tablet computer, etc.

[0134] In several embodiments provided in the present application, it should be understood that the disclosed computer device, apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0135] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. The above is only an implementation manner of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent flow transformation, or direct or indirect application in other related technical fields by using the content of the present application specification and drawings, are all included in the patent protection scope of the present application.

Claims

1. An image fusion method, characterized by, The method comprises: obtaining a first image and a second image, the first image and the second image belonging to different image types; inputting the first image and the second image into an encoder of a pre-trained image fusion model for feature extraction to obtain first image features and second image features, the encoder comprising a first convolutional block, a second convolutional block and a threshold generation network, the first convolutional block convolving an image to obtain a first feature map, the second convolutional block convolving the first feature map to obtain a second feature map, the threshold generation network generating a soft threshold value according to the second feature map, the first image features being calculated according to the first feature map corresponding to the first image, the second feature map and the soft threshold value, and the second image features being calculated according to the first feature map corresponding to the second image, the second feature map and the soft threshold value; performing weighted fusion on the first image features and the second image features by using a fusion layer of the image fusion model to obtain fused image features; inputting the fused image features into a decoder of the image fusion model for decoding to obtain a final reconstructed image; wherein: the threshold generation network comprises an absolute value processing and global average pooling layer, a first full connection layer, a batch normalization layer, a ReLU activation function, a second full connection layer and a sigmoid activation function, and the generation process of the soft threshold value is represented as: Wherein, alpha represents the soft threshold, F represents full connection calculation, sigmoid represents sigmoid activation function, delta represents ReLU activation function, and W represents convolution matrix, A third feature map obtained by performing global average pooling on absolute values of the second feature map, and m represents the number of channels.

2. The image fusion method of claim 1, wherein, the process of convolving an image by a convolutional block to obtain a feature map is represented as: wherein, represents a feature map generated by the kth convolutional block convolving the image, a (m) represents a convolution matrix of the convolutional block, b (m) represents a bias parameter of the convolutional block, m represents the number of channels, represents a convolution, and δ represents an activation function.

3. The image fusion method of claim 1, wherein, The method further comprises pre-training the encoder and the decoder, specifically comprising: obtaining a first sample image and a second sample image, the first sample image and the second sample image belonging to different image types; inputting the first sample image or the second sample image into the encoder for feature extraction to obtain first sample image features or second sample image features; inputting the first sample image features or the second sample image features into the decoder for decoding to obtain a first sample reconstructed image or a second sample reconstructed image; training the encoder and the decoder based on the first sample reconstructed image or the second sample reconstructed image and a preset loss function, the preset loss function comprising a first loss function for optimizing pixel information of the reconstructed image, a second loss function for optimizing structural similarity information of the reconstructed image, a third loss function for optimizing gradient information of the reconstructed image and a fourth loss function for optimizing color information of the reconstructed image.

4. The image fusion method of claim 3, wherein, The preset loss function is represented as: Loss = aL pixel +bL ssim +cL grad +dL color ; Loss denotes the preset loss function, L pixel denotes the first loss function, L ssim denotes the second loss function, L grad denotes the third loss function, L color denotes the fourth loss function, a, b, c, d denote preset hyperparameters.

5. The image fusion method of claim 4, wherein, The first loss function is represented as: L pixel = MSE(I ir , I re1 ) + MSE(I vr , I re2 ); wherein, L pixel represents the first loss function, MSE represents calculating the difference between the sample image and the sample reconstructed image using the mean square error function, I ir represents the first sample image, I re1 represents the first sample reconstructed image, I vr represents the second sample image, I re2 represents the second sample reconstructed image, N represents the number of pixel points; The second loss function is represented as: wherein L ssim denotes the second loss function, SSIM(X, F) denotes the similarity between the sample image and the sample reconstructed image, μ x denotes the mean of the sample image, μ f denotes the mean of the sample reconstructed image, σ x denotes the standard deviation of the sample image, σ f denotes the standard deviation of the sample reconstructed image, σ xf denotes the covariance of the sample image and the sample reconstructed image; The third loss function is represented as: L grad = MSE(Gradient(I re ), Gradient(I in )); wherein L grad represents the third loss function, Gradient represents an image gradient map calculated by a Laplacian operator, I re represents a sample reconstructed image, I in represents a sample image, (x, y) represents coordinates of a point of the image, and f(x, y) represents a pixel value of the point (x, y) of the image. The fourth loss function value is represented as: where L color represents the fourth loss function, Histogram represents a color histogram.

6. An image fusion apparatus using the image fusion method of claim 1, characterized by, The device comprises: an obtaining module configured to obtain a first image and a second image, the first image and the second image belonging to different image types; The feature extraction module is configured to input the first image and the second image into an encoder of a pre-trained image fusion model respectively for feature extraction to obtain a first image feature and a second image feature. The encoder comprises a first convolutional block, a second convolutional block and a threshold generation network. The first convolutional block is configured to perform convolution on an image to obtain a first feature map. The second convolutional block is configured to perform convolution on the first feature map to obtain a second feature map. The threshold generation network is configured to generate a soft threshold according to the second feature map. The first image feature is calculated according to the first feature map corresponding to the first image, the second feature map and the soft threshold. The second image feature is calculated according to the first feature map corresponding to the second image, the second feature map and the soft threshold. The fusion module is configured to perform weighted fusion on the first image feature and the second image feature by using a fusion layer of the image fusion model to obtain a fused image feature. The reconstruction module is configured to input the fused image feature into a decoder of the image fusion model for decoding to obtain a final reconstructed image.

7. A computer device, characterized by The computer device comprises a processor and a memory coupled to the processor. The memory stores program instructions which, when executed by the processor, cause the processor to perform the steps of the image fusion method according to any one of claims 1-5.

8. A storage medium, characterized by The computer device stores program instructions which implement the image fusion method according to any one of claims 1-5.

Images