Image fusion method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides an image fusion method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring an infrared image and a visible light image, inputting the infrared image into an infrared feature extraction model, circularly performing convolution operation, global and local feature extraction and down-sampling for multiple times to obtain a plurality of infrared image features, and inputting the visible light image into a visible light feature extraction model. And convolution operation, global and local feature extraction and down-sampling are circularly performed for multiple times to obtain multiple visible light image features, and finally the multiple infrared image features and the multiple visible light image features are input into a feature fusion model for fusion to obtain a fused image. By obtaining the infrared image features and the visible light image features of different sizes, it can be ensured that all important-scale information in the image is captured, and it is ensured that the finally obtained fused image is clearer.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image fusion method, electronic equipment, storage medium and program product. Background Art

[0002] Autonomous driving technology aims to achieve autonomous perception, decision-making and control of vehicles through on-board sensors and artificial intelligence systems, thereby improving traffic efficiency and safety. However, autonomous driving technology faces many challenges in the perception of complex environments. Traditional environmental perception methods mainly rely on visible light cameras and radar sensors. However, these methods have obvious limitations in severe weather conditions such as night, haze, rain and snow. When the light is insufficient or the visibility is low, the image quality of the visible light camera is greatly reduced, affecting the perception accuracy; although the radar is not affected by light, it has deficiencies in detail recognition and object classification.

[0003] Infrared and visible light fusion technology is a technology that combines the information of infrared images and visible light images to produce a new image that contains the advantages of both. The existing technology mainly extracts the features of infrared images and visible light images through neural network technology, and then fuses the features of infrared images and visible light images to form a fused image.

[0004] However, the existing technology only extracts infrared features and visible light features at the same fixed scale or resolution, and then fuses the extracted infrared features and visible light features. As a result, when the image scale changes greatly, it is impossible to fully extract information of all important scales in the image, which in turn leads to the loss of some important features in the original image. Summary of the invention

[0005] The embodiments of the present application provide an image fusion method, an electronic device, a storage medium, and a program product for forming a visible light image with rich texture and clear target.

[0006] In a first aspect, the present application provides an image fusion method, comprising:

[0007] Acquire infrared images and visible light images;

[0008] The infrared image is input into an infrared feature extraction model, and convolution operation, global and local feature extraction and downsampling are performed repeatedly to obtain multiple infrared image features, wherein the infrared feature extraction model comprises: a first convolution module and multiple cascaded infrared feature extraction units, each of the infrared feature extraction units comprises a first feature perception module and a first downsampling module cascaded in sequence, and each of the infrared feature extraction units outputs one infrared image feature;

[0009] The visible light image is input into a visible light feature extraction model, and convolution operation, global and local feature extraction and downsampling are performed repeatedly to obtain multiple visible light image features, wherein the visible light feature extraction model comprises: a second convolution module and multiple cascaded visible light feature extraction units, each of the visible light feature extraction units comprises a second feature perception module and a second downsampling module cascaded in sequence, and each of the visible light feature extraction units outputs one visible light image feature;

[0010] The plurality of infrared image features and the plurality of visible light image features are input into a feature fusion model for fusion to obtain a fused image.

[0011] In a possible implementation, the first feature perception module and the second feature perception module both include a conversion block and a convolution block;

[0012] The operations in the conversion block include:

[0013] Inputting the input features of the conversion block into the self-attention layer, performing regularization processing and self-attention operation in sequence, and obtaining self-attention output features;

[0014] Inputting the self-attention output feature into a multi-layer perceptron, performing regularization processing and multi-layer perceptron operation in sequence, and obtaining a conversion block output feature;

[0015] The operations in the convolution block include:

[0016] Inputting the input features of the convolution block into the first convolution sub-block, performing a first convolution operation, and obtaining a first convolution feature;

[0017] Inputting the first convolution feature into a first activation function, performing an activation operation, and obtaining a first activation feature;

[0018] Inputting the first activation feature into the second convolution sub-block, performing a second convolution operation, and obtaining a second convolution feature;

[0019] Inputting the second convolution feature into the third convolution sub-block, performing a third convolution operation, and obtaining a third convolution feature;

[0020] Inputting the third convolution feature into a second activation function, performing an activation operation, and obtaining a second activation feature;

[0021] The second activation feature is input into the fourth convolution sub-block, and a fourth convolution operation is performed to obtain a convolution block output feature.

[0022] In a possible implementation, the plurality of infrared image features and the plurality of visible light image features are input into a feature fusion model for fusion to obtain a fused image, including:

[0023] splicing the i-th infrared image feature and the i-th visible light image feature to obtain the i-th spliced ​​image feature, where i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the number of the infrared feature extraction units or the number of the visible light feature extraction units;

[0024] Performing an upsampling process on the second stitched image feature to obtain a first image feature;

[0025] Performing upsampling processing twice on the third spliced ​​image feature to obtain a second image feature;

[0026] The first stitched image feature, the first image feature and the second image feature are fused to obtain the fused image.

[0027] In a possible implementation, fusing the first stitched image feature, the first image feature, and the second image feature to obtain the fused image includes:

[0028] Inputting the first spliced ​​image feature, the first image feature and the second image feature into the fifth convolution sub-block, performing a first convolution operation, and obtaining a fourth convolution feature;

[0029] Inputting the fourth convolution feature into a third activation function, performing an activation operation, and obtaining a third activation feature;

[0030] The third activated feature is input into the sixth convolution sub-block, and a second convolution operation is performed to obtain the fused image.

[0031] In a possible implementation, the sampling rates of the first down-sampling module and the second down-sampling module are both 1 / 2, the number of the infrared feature extraction units and the number of the visible light feature extraction units are both 3, the first convolution module and the second convolution module each include three convolution kernels, and the number of channels of the first convolution module and the second convolution module are both 32.

[0032] In a possible implementation, the infrared feature extraction model, the visible light feature extraction model, and the feature fusion model are obtained by joint training, and the loss function used in the training process is used to weight the amplitude loss and the texture loss, including: the amplitude loss is the difference between the pixel value and the pixel gradient in the output image of the feature fusion model during the training process, and the pixel gradient includes the pixel horizontal gradient and / or the pixel vertical gradient;

[0033] The texture loss is the norm of the difference between the absolute value of the change in pixel value in the output image and the maximum value of the absolute value of the first-order gradient of the horizontal gradient of the fused image and the absolute value of the first-order gradient of the vertical gradient of the fused image, divided by the height and width of the fused image.

[0034] In a second aspect, the present application provides an image fusion device, comprising:

[0035] An acquisition module, used for acquiring infrared images and visible light images;

[0036] A processing module is used to input the infrared image into an infrared feature extraction model, and perform convolution operations, global and local feature extraction and downsampling in a cycle for multiple times to obtain multiple infrared image features, wherein the infrared feature extraction model comprises: multiple cascaded infrared feature extraction units, each of which comprises a first convolution module, a first feature perception module and a first downsampling module cascaded in sequence, and each of which outputs one infrared image feature;

[0037] The processing module is further used to input the visible light image into a visible light feature extraction model, and perform convolution operations, global and local feature extraction and downsampling in a cycle for multiple times to obtain multiple visible light image features, wherein the visible light feature extraction model comprises: multiple cascaded visible light feature extraction units, each of which comprises a second convolution module, a second feature perception module and a second downsampling module that are cascaded in sequence, and each of which outputs one visible light image feature;

[0038] The processing module is further used to input the plurality of infrared image features and the plurality of visible light image features into a feature fusion model for fusion to obtain a fused image.

[0039] In an image fusion device provided in this application:

[0040] The operations in the conversion block include:

[0041] Inputting the input features of the conversion block into the self-attention layer, performing regularization processing and self-attention operation in sequence, and obtaining self-attention output features;

[0042] Inputting the self-attention output feature into a multi-layer perceptron, performing regularization processing and multi-layer perceptron operation in sequence, and obtaining a conversion block output feature;

[0043] The operations in the convolution block include:

[0044] Inputting the input features of the convolution block into the first convolution sub-block, performing a first convolution operation, and obtaining a first convolution feature;

[0045] Inputting the first convolution feature into a first activation function, performing an activation operation, and obtaining a first activation feature;

[0046] Inputting the first activation feature into the second convolution sub-block, performing a second convolution operation, and obtaining a second convolution feature;

[0047] Inputting the second convolution feature into the third convolution sub-block, performing a third convolution operation, and obtaining a third convolution feature;

[0048] Inputting the third convolution feature into a second activation function, performing an activation operation, and obtaining a second activation feature;

[0049] The second activation feature is input into the fourth convolution sub-block, and a fourth convolution operation is performed to obtain a convolution block output feature.

[0050] In an image fusion device provided in this application:

[0051] splicing the i-th infrared image feature and the i-th visible light image feature to obtain the i-th spliced ​​image feature, where i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the number of the infrared feature extraction units or the number of the visible light feature extraction units;

[0052] Performing an upsampling process on the second stitched image feature to obtain a first image feature;

[0053] Performing upsampling processing twice on the third spliced ​​image feature to obtain a second image feature;

[0054] The first stitched image feature, the first image feature and the second image feature are fused to obtain the fused image.

[0055] In an image fusion device provided in this application:

[0056] Inputting the first spliced ​​image feature, the first image feature and the second image feature into the fifth convolution sub-block, performing a first convolution operation, and obtaining a fourth convolution feature;

[0057] Inputting the fourth convolution feature into a third activation function, performing an activation operation, and obtaining a third activation feature;

[0058] The third activated feature is input into the sixth convolution sub-block, and a second convolution operation is performed to obtain the fused image.

[0059] In an image fusion device provided in this application:

[0060] The sampling rates of the first down-sampling module and the second down-sampling module are both 1 / 2, the number of the infrared feature extraction units and the number of the visible light feature extraction units are both 3, the first convolution module and the second convolution module each include three convolution kernels, and the number of channels of the first convolution module and the second convolution module are both 32.

[0061] In an image fusion device provided in this application:

[0062] The infrared feature extraction model, the visible light feature extraction model and the feature fusion model are obtained by joint training, and the loss function used in the training process is used to weight the amplitude loss and the texture loss, including: the amplitude loss is the difference between the pixel value and the pixel gradient in the output image of the feature fusion model during the training process, and the pixel gradient includes the pixel horizontal gradient and / or the pixel vertical gradient;

[0063] The texture loss is the norm of the difference between the absolute value of the change in pixel value in the output image and the maximum value of the absolute value of the first-order gradient of the horizontal gradient of the fused image and the absolute value of the first-order gradient of the vertical gradient of the fused image, divided by the height and width of the fused image.

[0064] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0065] The memory stores computer-executable instructions;

[0066] The processor executes the computer-executable instructions stored in the memory to implement an image fusion method as described in any one of the first aspects.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement an image fusion method as described in any one of the first aspects.

[0068] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements an image fusion method as described in any one of the first aspects.

[0069] The present application provides an image fusion method, electronic device, storage medium and program product, which obtains an infrared image and a visible light image, then inputs the infrared image into an infrared feature extraction model, performs convolution operation, global and local feature extraction and downsampling in a loop for multiple times, obtains multiple infrared image features, and inputs the visible light image into a visible light feature extraction model, performs convolution operation, global and local feature extraction and downsampling in a loop for multiple times, obtains multiple visible light image features, and finally inputs the multiple infrared image features and the multiple visible light image features into a feature fusion model for fusion to obtain a fused image. Compared with the prior art that only extracts infrared features and visible light features at the same fixed scale or resolution, which results in that when the image size changes greatly, the information of all important scales in the image cannot be fully extracted, resulting in the loss of some important features in the original image, the infrared feature extraction model and the visible light feature extraction model of the present application extract infrared image features of multiple sizes and visible light image features of multiple sizes, and obtains infrared image features and visible light image features of different sizes, so as to ensure that the information of all important scales in the image is captured, avoid the key features that may be missed at a single scale, and ensure that the final fused image is clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0071] Figure 1 It is a schematic diagram of infrared feature extraction model, visible light feature extraction model and feature fusion model;

[0072] Figure 2 A schematic diagram of a flow chart of a first embodiment of an image fusion method provided in this embodiment;

[0073] Figure 3 Schematic diagram of a first feature perception module and a second feature perception module;

[0074] Figure 4 A schematic diagram of a flow chart of a second embodiment of an image fusion method provided in this embodiment;

[0075] Figure 5 A schematic diagram of a flow chart of a third embodiment of an image fusion method provided in this embodiment;

[0076] Figure 6 Schematic diagram of feature fusion model;

[0077] Figure 7 A flowchart of a fourth embodiment of an image fusion method provided in this embodiment;

[0078] Figure 8An image fusion device provided in this embodiment;

[0079] Fig. 9 A schematic diagram of the structure of an electronic device embodiment 1 provided in an embodiment of the present application.

[0080] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0083] In the prior art, since infrared features and visible light features are only extracted at the same fixed scale or resolution, when the image scale changes greatly, the neural network cannot fully extract the infrared features in the infrared image and the visible light features in the visible light image, resulting in the fused image losing some important features of the original image.

[0084] Based on this, the inventive concept of the present application is to provide an image fusion method by extracting the infrared features of the infrared image and the visible light features of the visible light image at multi-scale, and then fusing the extracted multi-scale infrared features and visible light features at multi-scale, and finally forming a visible light image with rich texture and clear target.

[0085] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0086] Figure 1 Figure 1 is a schematic diagram of the infrared feature extraction model, visible light feature extraction model and feature fusion model. Figure 1 As shown in the figure, the visible light image is extracted through the three infrared feature extraction models and three downsampling modules to obtain three infrared image features. The infrared image is extracted through the three visible light feature extraction models and three downsampling modules to obtain three visible light image features. The three infrared image features and the three visible light image features are spliced ​​separately and then sent to the feature fusion model to obtain a fused image.

[0087] Figure 2 A flowchart of a first embodiment of an image fusion method provided in this embodiment is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0088] S201: Acquire an infrared image and a visible light image.

[0089] In this embodiment, the infrared image is obtained by capturing the infrared radiation emitted by the object through the radar. The infrared image has good target detection and recognition capabilities at night or in bad weather conditions. The visible light image is obtained by the visible light camera. The visible light image has rich color and texture information. By fusing the two, a new image can be generated that contains both the target detection capability of the infrared image and the color and texture information of the visible light image. How to describe them as a pair

[0090] S202: Input the infrared image into the infrared feature extraction model, perform convolution operation, global and local feature extraction and downsampling in a cycle for multiple times to obtain multiple infrared image features. The infrared feature extraction model includes: a first convolution module and multiple cascaded infrared feature extraction units, each infrared feature extraction unit includes a first feature perception module and a first downsampling module cascaded in sequence, and each infrared feature extraction unit outputs an infrared image feature.

[0091] In this embodiment, the infrared image is input into the infrared feature extraction model to extract the infrared image features. The infrared image extraction model first performs feature mapping on the input infrared image by the first convolution module, and then inputs the output features into a plurality of cascaded infrared feature extraction units. Optionally, there are three infrared feature extraction units. Each infrared feature extraction unit includes a first feature perception module and a first downsampling module cascaded in sequence, wherein each infrared feature extraction unit outputs an infrared image feature. For example, when there are three infrared feature extraction units, three infrared image features are output.

[0092] S203: Input the visible light image into the visible light feature extraction model, perform convolution operation, global and local feature extraction and downsampling in a cycle for multiple times, and obtain multiple visible light image features. The visible light feature extraction model includes: a second convolution module and multiple cascaded visible light feature extraction units, each visible light feature extraction unit includes a second feature perception module and a second downsampling module cascaded in sequence, and each visible light feature extraction unit outputs a visible light image feature.

[0093] In this embodiment, the visible light image is input into the infrared feature extraction model to extract the visible light image features. The visible light image extraction model first performs feature mapping on the input visible light image by the second convolution module, and then inputs the output features into a plurality of cascaded visible light feature extraction units. Optionally, there are three visible light feature extraction units. Each visible light feature extraction unit includes a second feature perception module and a second downsampling module cascaded in sequence, wherein each visible light feature extraction unit outputs a visible light image feature. For example, when there are three visible light feature extraction units, three visible light image features are output.

[0094] The sampling rates of the first down-sampling module and the second down-sampling module are both 1 / 2, there are 3 infrared feature extraction units and 3 visible light feature extraction units, the first convolution module and the second convolution module both include three convolution kernels, and the number of channels of the first convolution module and the second convolution module are both 32.

[0095] The first feature perception module and the second feature perception module mentioned in the above step S102 and step S103 both include a conversion block and a convolution block.

[0096] Figure 3 The first feature perception module and the second feature perception module are shown as schematic diagrams. The first feature perception module and the second feature perception module have the same structure. The first feature perception module and the second feature perception module mentioned in the above step S202 and step S203 both include a conversion block and a convolution block. Figure 3 As shown, the conversion block is used to extract global features and the convolution block is used to extract local features.

[0097] The operations in the conversion block include:

[0098] The input features of the conversion block are input into the self-attention layer, and regularization and self-attention operations are performed in sequence to obtain the self-attention output features.

[0099] In this embodiment, the input features of the conversion block are input into the self-attention layer, wherein the input features of the conversion block are features input into the conversion block, including features after passing through the first convolution module or the second convolution module, features after passing through the first feature perception module and the first downsampling, and features after passing through the second feature perception module and the second downsampling. The self-attention layer first performs regularization processing on the features input to the conversion block, converts the input features into three parts, namely: query (Q, Query), key (K, Key) and value (V, Value), and then performs self-attention operation to obtain self-attention output features, and then adds the self-attention output features to the input features of the conversion block to obtain the final self-attention output features. The regularization operation and self-attention operation can be expressed as:

[0100] X1=SA(LN(X))+X

[0101]

[0102] Among them, SA represents the self-attention operation, LN represents the regularization operation, X represents the input feature of the conversion block, X1 represents the self-attention output feature, softmax represents the activation function, and K T represents the transpose of K, and d is the dimension of K.

[0103] The self-attention output features are input into the multi-layer perceptron, and regularization and multi-layer perception operations are performed in sequence to obtain the conversion block output features.

[0104] In this embodiment, the obtained self-attention output feature is input into a multi-layer perceptron, and regularization processing and multi-layer perception operation are performed in sequence, wherein the regularization and multi-layer perception operation formulas are:

[0105] X trans =MLP(LN(X1))+X1

[0106] Where MLP stands for multi-layer perceptron, X trans represents the output features of the conversion block, LN represents the regularization operation, and X1 represents the self-attention output features.

[0107] The operations in the convolution block include:

[0108] The input features of the convolution block are input into the first convolution sub-block, and the first convolution operation is performed to obtain the first convolution features.

[0109] The first convolution feature is input into the first activation function, an activation operation is performed, and the first activation feature is obtained.

[0110] The first activation feature is input into the second convolution sub-block, and a second convolution operation is performed to obtain the second convolution feature.

[0111] In this embodiment, the input features of the convolution block are features input into the convolution block, including features after passing through the first convolution module or the second convolution module, features after passing through the first feature perception module and the first downsampling, and features after passing through the second feature perception module and the second downsampling. The input features of the convolution block are input into the first convolution sub-block, and the first convolution operation is performed to obtain the first convolution feature. The first convolution feature is input into the first activation function, and the activation operation is performed to obtain the first activation feature. Optionally, the activation function is a ReLU activation function, and the first activation feature is input into the second convolution sub-block, and the second convolution operation is performed to obtain the second convolution feature. The above-mentioned first convolution sub-block and second convolution sub-block are both convolutions with a kernel of 3.

[0112] The second convolution feature is input into the third convolution sub-block, and a third convolution operation is performed to obtain the third convolution feature.

[0113] The third convolution feature is input into the second activation function, an activation operation is performed, and a second activation feature is obtained.

[0114] The second activation feature is input into the fourth convolution sub-block, and the fourth convolution operation is performed to obtain the convolution block output feature.

[0115] In this embodiment, the second convolution feature obtained above is input into the third convolution sub-block, and the third convolution operation is performed to obtain the third convolution feature, and then the third convolution feature is input into the second activation function, and the activation operation is performed to obtain the second activation feature, and the second activation feature is input into the fourth convolution sub-block, and the fourth convolution operation is performed to obtain the convolution block output feature. The third convolution sub-block and the fourth convolution sub-block are both convolutions with a kernel of 3, and the second activation function is the ReLU activation function.

[0116] S204: Inputting the plurality of infrared image features and the plurality of visible light image features into a feature fusion model for fusion to obtain a fused image.

[0117] In this embodiment, the obtained multiple infrared image features and multiple visible light image features are input into a feature fusion model for feature fusion, and finally a fused image is obtained.

[0118] In this embodiment, firstly, an infrared image and a visible light image are acquired, and then the infrared image is input into an infrared feature extraction model, and convolution operation, global and local feature extraction and downsampling are performed repeatedly in a cycle to obtain multiple infrared image features, and the visible light image is input into a visible light feature extraction model, and convolution operation, global and local feature extraction and downsampling are performed repeatedly in a cycle to obtain multiple visible light image features, and finally the multiple infrared image features and the multiple visible light image features are input into a feature fusion model for fusion to obtain a fused image. Compared with the prior art that only extracts infrared features and visible light features at the same fixed scale or resolution, resulting in the inability to fully extract information of all important scales in the image when the image size changes greatly, resulting in the loss of some important features in the original image, the infrared feature extraction model and the visible light feature extraction model of the present application extract infrared image features of multiple sizes and visible light image features of multiple sizes, and by acquiring infrared image features and visible light image features of different sizes, it can ensure that information of all important scales in the image is captured, avoid key features that may be missed at a single scale, and ensure that the final fused image is clearer.

[0119] Figure 4 A schematic diagram of a flow chart of a second embodiment of an image fusion method provided in this embodiment, such as Figure 4 As shown, in Figure 2 Based on the illustrated embodiment, the specific implementation of step S204 is as follows:

[0120] S401: splicing the i-th infrared image feature and the i-th visible light image feature to obtain the i-th spliced ​​image feature, where i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the number of infrared feature extraction units or the number of visible light feature extraction units.

[0121] In this embodiment, the obtained i-th infrared image feature and the i-th visible light image feature are spliced ​​to obtain the i-th spliced ​​image, where i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the number of infrared feature extraction units or the number of visible light feature extraction units. Because the number of infrared feature extraction units and visible light feature extraction units is the same, the number of spliced ​​images obtained is the same as the number of infrared feature extraction units or visible light extraction units. Optionally, when the number of infrared image extraction units and visible light image extraction units is 3, the number of infrared image features and visible light image features is 3.

[0122] S402: Perform an upsampling process on the second spliced ​​image feature to obtain a first image feature.

[0123] In this embodiment, the second stitched image feature is formed by stitching together the second infrared image feature and the second visible light image feature, the second infrared image feature is extracted by the second infrared feature extraction unit, and the second visible light image feature is extracted by the second visible light image feature. The second stitched image feature is upsampled once to obtain the first image feature.

[0124] S403: Perform upsampling processing twice on the third spliced ​​image feature to obtain a second image feature.

[0125] In this embodiment, the third stitched image feature is formed by stitching together the third infrared image feature and the third visible light image feature. The third infrared image feature is extracted by the third infrared feature extraction unit, and the third visible light image feature is extracted by the third visible light image feature. The third stitched image feature is upsampled twice to obtain the second image feature.

[0126] S404: Fusing the first spliced ​​image feature, the first image feature, and the second image feature to obtain a fused image.

[0127] In this embodiment, the first stitched image feature is formed by stitching together the first infrared image feature and the first visible light image feature, the first infrared image feature is extracted by the first infrared feature extraction unit, and the first visible light image feature is extracted by the first visible light image feature. The first stitched image feature, the first image feature and the second image feature are fused to finally obtain a fused image.

[0128] In this embodiment, the corresponding infrared image features and visible light image features are spliced, and the obtained second spliced ​​image features are upsampled once to obtain the first image features. Through the upsampling operation, the size of the first image features is the same as the size of the first spliced ​​image features. The third spliced ​​image features are upsampled twice to obtain the second image features. Through the upsampling operation twice, the size of the second image features is the same as the size of the first spliced ​​image features. Finally, the first spliced ​​image features, the first image features and the second image features of the same size are fused to obtain a fused image. By extracting feature maps of different sizes, it is ensured that all important scale information in the image is captured. Finally, the feature maps of different sizes captured are upsampled to form feature maps of the same size. Finally, the feature maps of the same size are fused to obtain a fused image, so that the fused image has information of all important scales, avoiding key features that may be missed at a single scale, and ensuring that the final fused image is clearer.

[0129] Figure 5 A flowchart of a third embodiment of an image fusion method provided in this embodiment is shown in FIG. Figure 5 As shown, in Figure 4 Based on the illustrated embodiment, the specific implementation of step S404 is as follows:

[0130] S501: Input the first concatenated image feature, the first image feature, and the second image feature into the fifth convolution sub-block, perform the first convolution operation, and obtain a fourth convolution feature.

[0131] In this embodiment, the first concatenated image feature, the first image feature and the second image feature are first concatenated and then input into the fifth convolution sub-block to perform the first convolution operation to obtain the fourth convolution feature.

[0132] S502: Input the fourth convolution feature into the third activation function, perform activation operation, and obtain the third activation feature.

[0133] In this embodiment, the obtained fourth convolution feature is input into the third activation function, and an activation operation is performed to obtain the third activation feature, wherein the activation function is a ReLU activation function.

[0134] S503: Input the third activated feature into the sixth convolution sub-block, perform a second convolution operation, and obtain a fused image.

[0135] In this embodiment, the obtained third activation feature is input into the sixth convolution block, and a second convolution operation is performed to obtain a final fused image.

[0136] In this embodiment, the concatenated feature map is input into the fifth convolution sub-block, the first convolution operation is performed, the fourth convolution feature is obtained, and then the ReLU activation function is used to activate the third activation feature, and finally the third activation feature is input into the sixth convolution sub-block for convolution operation to obtain the final fused image. By fusing feature maps with information of different scales, the obtained fused image has information at different scales, ensuring that the final fused image is clearer.

[0137] Figure 6 It is a schematic diagram of the feature fusion model. Figure 6 As shown, the first spliced ​​image feature is not upsampled, the second spliced ​​image feature is upsampled once to obtain the first image feature, the third spliced ​​image feature is upsampled twice to obtain the second image feature, and then the first spliced ​​image feature, the first image feature and the second image feature are spliced, and then input into the convolution sub-block, and then into the ReLU activation function, and then into the convolution word block, and finally a fused image is obtained.

[0138] Figure 7 A flowchart of a fourth embodiment of an image fusion method provided in this embodiment is shown in FIG. Figure 7 As shown, the infrared feature extraction model, the visible light feature extraction model and the feature fusion model are obtained by joint training. The loss function used in the training process is used to weight the amplitude loss and the texture loss. The method includes the following steps:

[0139] S701: The amplitude loss is the norm of the difference between the pixel value and the pixel gradient in the output image of the feature fusion model during the training process, and is divided by the height and width of the fused image. The pixel gradient includes the pixel horizontal gradient and / or the pixel vertical gradient.

[0140] In this embodiment, the amplitude loss function is:

[0141]

[0142] Among them, L int represents the amplitude loss, H represents the height of the output image, W represents the width of the output image, and I f Represents the pixel value of the output image, I ir Represents the pixel level gradient of the output image, I iv represents the pixel vertical gradient of the output image, and ||·||1 represents the L1 norm.

[0143] The above formula indicates that the pixel horizontal gradient and pixel vertical gradient of each pixel of the output image are first obtained, and the difference between the pixel value and the pixel gradient is the pixel value of each pixel in the output image minus the corresponding pixel horizontal gradient or the pixel vertical gradient, whichever is larger. That is, when the pixel horizontal gradient and the pixel vertical gradient corresponding to a certain pixel value are larger than the pixel vertical gradient, the pixel value minus the pixel horizontal gradient corresponding to the pixel value. When the differences between all pixel values ​​and pixel gradients are obtained, the L1 norm is calculated and divided by the height and width of the fused image.

[0144] S702: Texture loss is the norm of the difference between the absolute value of the change in pixel value in the output image and the maximum value of the absolute value of the first-order gradient of the horizontal gradient of the fused image and the absolute value of the first-order gradient of the vertical gradient of the fused image, divided by the height and width of the fused image.

[0145] In this embodiment, the texture loss function is:

[0146]

[0147] Among them, L texture represents texture loss, H represents the height of the output image, W represents the width of the output image, and I f Represents the pixel value of the output image, I ir Represents the pixel level gradient of the output image, I ivrepresents the pixel vertical gradient of the output image, ||·||1 represents the L1 norm, Represents the Sobel operator.

[0148] The above formula indicates that the pixel horizontal gradient and pixel vertical gradient of each pixel of the output image are first obtained, and the Sobel operator is used to operate the pixel horizontal gradient and pixel vertical gradient of each pixel to obtain the first-order gradient of the horizontal gradient and the first-order gradient of the vertical gradient, and the absolute value of the first-order gradient of the horizontal gradient and the absolute value of the first-order gradient of the vertical gradient are obtained, and then the norm of the difference between each pixel value and the maximum value of the absolute value of the first-order gradient of the horizontal gradient and the absolute value of the first-order gradient of the vertical gradient is obtained, and divided by the height and width of the fused image.

[0149] The loss function used in the training process is to weight the amplitude loss and texture loss. Optionally, the loss function used in the training process is:

[0150] L=L int +αL texture

[0151] Among them, L is the loss function, L int Indicates the amplitude loss, L texture represents texture loss and α represents a hyperparameter.

[0152] In this embodiment, an infrared feature extraction model, a visible light feature extraction model and a feature fusion model are obtained through training, and finally the trained infrared feature extraction model, visible light feature extraction model and feature fusion model are used to extract features and fuse features of infrared images and visible light images. In particular, by constructing an infrared image and visible light image dataset, the dataset contains multiple pairs of images, each pair of images contains an infrared image and a corresponding visible light image. The images are cropped to ensure the alignment of the resolution and image content of the infrared image and the visible light image. The size of each image block is 128×128, and the infrared image and visible light image datasets are randomly divided into 80% of the datasets as training sets and 20% of the datasets as test sets. The infrared feature extraction model, the visible light feature extraction model and the feature fusion model are trained using the data of the training set. The network parameters are initialized using the random initialization method. The Adam optimizer is used, the initial learning rate is set to 0.001, and the batch size is set to 8. The infrared feature extraction model, the visible light feature extraction model and the feature fusion model are trained. When the results of the loss function on the training set and the test set tend to be stable, it means that the current infrared feature extraction model, the visible light feature extraction model and the feature fusion model are the trained models infrared feature extraction model, the visible light feature extraction model and the feature fusion model.

[0153] Figure 8An image fusion device provided in this embodiment. An image fusion device provided in this application includes: an acquisition module 81 and a processing module 82; wherein,

[0154] The acquisition module 81 is used to acquire an infrared image and a visible light image; the processing module 82 is used to input the infrared image into the infrared feature extraction model, perform convolution operations, global and local feature extraction and downsampling in a loop for multiple times, and obtain multiple infrared image features. The infrared feature extraction model includes: multiple cascaded infrared feature extraction units, each infrared feature extraction unit includes a first convolution module, a first feature perception module and a first downsampling module cascaded in sequence, and each infrared feature extraction unit outputs an infrared image feature; the processing module 82 is also used to input the visible light image into the visible light feature extraction model, perform convolution operations, global and local feature extraction and downsampling in a loop for multiple times, and obtain multiple visible light image features. The visible light feature extraction model includes: multiple cascaded visible light feature extraction units, each visible light feature extraction unit includes a second convolution module, a second feature perception module and a second downsampling module cascaded in sequence, and each visible light feature extraction unit outputs a visible light image feature; the processing module 82 is also used to input the multiple infrared image features and the multiple visible light image features into the feature fusion model for fusion, and obtain a fused image.

[0155] Fig. 9 A schematic diagram of the structure of an electronic device embodiment 1 provided in the present application. Fig. 9 The electronic device provided in this embodiment includes: at least one processor 91 and a memory 92. The processor 91 and the memory 92 are connected via a bus 93.

[0156] The specific implementation process of the processor 91 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0157] In the above Fig. 9 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0158] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0159] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0160] The present application also provides a computer-readable storage medium on which computer-executable instructions are stored. When a processor executes the computer-executable instructions, an image fusion method as described above is implemented.

[0161] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0162] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0163] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0167] The present application also provides a computer program product, including a computer program, which is used to implement the above image fusion method when executed by a processor.

[0168] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0169] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0170] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0171] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0172] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0173] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0174] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0176] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image fusion method, characterized in that: include: Acquire infrared images and visible light images; The infrared image is input into an infrared feature extraction model, and convolution operation, global and local feature extraction and downsampling are performed repeatedly to obtain multiple infrared image features, wherein the infrared feature extraction model comprises: a first convolution module and multiple cascaded infrared feature extraction units, each of the infrared feature extraction units comprises a first feature perception module and a first downsampling module cascaded in sequence, and each of the infrared feature extraction units outputs one infrared image feature; The visible light image is input into a visible light feature extraction model, and convolution operation, global and local feature extraction and downsampling are performed repeatedly to obtain multiple visible light image features, wherein the visible light feature extraction model comprises: a second convolution module and multiple cascaded visible light feature extraction units, each of the visible light feature extraction units comprises a second feature perception module and a second downsampling module cascaded in sequence, and each of the visible light feature extraction units outputs one visible light image feature; The plurality of infrared image features and the plurality of visible light image features are input into a feature fusion model for fusion to obtain a fused image.

2. The method according to claim 1, characterized in that The first feature perception module and the second feature perception module both include a conversion block and a convolution block; The operations in the conversion block include: Inputting the input features of the conversion block into the self-attention layer, performing regularization processing and self-attention operation in sequence, and obtaining self-attention output features; Inputting the self-attention output feature into a multi-layer perceptron, performing regularization processing and multi-layer perceptron operation in sequence, and obtaining a conversion block output feature; The operations in the convolution block include: Inputting the input features of the convolution block into the first convolution sub-block, performing a first convolution operation, and obtaining a first convolution feature; Inputting the first convolution feature into a first activation function, performing an activation operation, and obtaining a first activation feature; Inputting the first activation feature into the second convolution sub-block, performing a second convolution operation, and obtaining a second convolution feature; Inputting the second convolution feature into the third convolution sub-block, performing a third convolution operation, and obtaining a third convolution feature; Inputting the third convolution feature into a second activation function, performing an activation operation, and obtaining a second activation feature; The second activation feature is input into the fourth convolution sub-block, and a fourth convolution operation is performed to obtain a convolution block output feature.

3. The method according to claim 2, characterized in that The step of inputting the plurality of infrared image features and the plurality of visible light image features into a feature fusion model for fusing to obtain a fused image comprises: splicing the i-th infrared image feature and the i-th visible light image feature to obtain the i-th spliced ​​image feature, where i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the number of the infrared feature extraction units or the number of the visible light feature extraction units; Performing an upsampling process on the second stitched image feature to obtain a first image feature; Performing upsampling processing twice on the third spliced ​​image feature to obtain a second image feature; The first stitched image feature, the first image feature and the second image feature are fused to obtain the fused image.

4. The method according to claim 3, characterized in that: The step of fusing the first stitched image feature, the first image feature, and the second image feature to obtain the fused image includes: Inputting the first spliced ​​image feature, the first image feature and the second image feature into the fifth convolution sub-block, performing a first convolution operation, and obtaining a fourth convolution feature; Inputting the fourth convolution feature into a third activation function, performing an activation operation, and obtaining a third activation feature; The third activated feature is input into the sixth convolution sub-block, and a second convolution operation is performed to obtain the fused image.

5. The method according to any one of claims 1 to 4, characterized in that The sampling rates of the first down-sampling module and the second down-sampling module are both 1 / 2, the number of the infrared feature extraction units and the number of the visible light feature extraction units are both 3, the first convolution module and the second convolution module each include three convolution kernels, and the number of channels of the first convolution module and the second convolution module is both 32.

6. The method according to any one of claims 1 to 4, characterized in that The infrared feature extraction model, the visible light feature extraction model and the feature fusion model are obtained by joint training, and the loss function used in the training process is used to weight the amplitude loss and the texture loss, including: the amplitude loss is the difference between the pixel value and the pixel gradient in the output image of the feature fusion model during the training process, and the pixel gradient includes the pixel horizontal gradient and / or the pixel vertical gradient; The texture loss is the norm of the difference between the absolute value of the change in pixel value in the output image and the maximum value of the absolute value of the first-order gradient of the horizontal gradient of the fused image and the absolute value of the first-order gradient of the vertical gradient of the fused image, divided by the height and width of the fused image.

7. An image fusion device, characterized in that: include: An acquisition module, used for acquiring infrared images and visible light images; A processing module is used to input the infrared image into an infrared feature extraction model, and perform convolution operations, global and local feature extraction and downsampling in a cycle for multiple times to obtain multiple infrared image features, wherein the infrared feature extraction model comprises: multiple cascaded infrared feature extraction units, each of which comprises a first convolution module, a first feature perception module and a first downsampling module cascaded in sequence, and each of which outputs one infrared image feature; The processing module is further used to input the visible light image into a visible light feature extraction model, and perform convolution operations, global and local feature extraction and downsampling in a cycle for multiple times to obtain multiple visible light image features, wherein the visible light feature extraction model comprises: multiple cascaded visible light feature extraction units, each of which comprises a second convolution module, a second feature perception module and a second downsampling module that are cascaded in sequence, and each of which outputs one visible light image feature; The processing module is further used to input the plurality of infrared image features and the plurality of visible light image features into a feature fusion model for fusion to obtain a fused image.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images