A method and apparatus for fusing a visible light image and an infrared image
By using a self-attention convolution module and a multi-level multimodal information fusion method, visible light images and infrared images are fused, which solves the problem of insufficient recognition of individual images in complex environments, and the generated fused image improves the recognition effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-17
- Publication Date
- 2026-03-27
AI Technical Summary
Visible light and infrared images alone cannot meet the needs of visual tasks in complex environments. In particular, visible light images cannot identify hidden targets in smoke environments, while infrared images cannot display cold backgrounds.
A method using a self-attention convolution module for feature extraction and multi-level multimodal information fusion is proposed to fuse visible light images and infrared images. The method involves multiple feature extractions and multi-level multimodal information fusions through the self-attention convolution module, and finally generates a fused image using ReLU and Tanh functions.
The generated fused image not only contains information about the cold background, but also shows information about the target with thermal radiation, thus improving the image recognition effect.
Smart Images
Figure CN116433549B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of visible light image and infrared image fusion, and in particular to a visible light image and infrared image fusion method and a fusion network thereof. BACKGROUND
[0002] With the development of computer vision technology, computer algorithms such as image segmentation and target detection emerge in endlessly. A single visible light image cannot meet the needs of complex visual tasks. In many harsh environments, for example, on a battlefield full of smoke, a visible light image can only reflect the surrounding environmental information, but the soldier target hidden in the smoke cannot be presented in the visible light image. Although infrared images can obtain information of objects with heat radiation, they cannot display the cold background.
[0003] Therefore, whether the visible light image and the infrared image can be fused to integrate the advantages of the two kinds of images and improve the recognition effect of the image has become a problem to be solved. SUMMARY
[0004] In view of this, the present disclosure provides a visible light image and infrared image fusion method and a fusion network thereof to improve the recognition effect of the image.
[0005] In one aspect, the present disclosure provides a visible light image and infrared image fusion method, which comprises the following steps:
[0006] The input visible light image and infrared image are respectively subjected to convolution processing, and then feature extraction is performed by a self-attention convolution module to obtain a first visible light image and a first infrared image;
[0007] After the first visible light image and the first infrared image are respectively subjected to self-attention convolution processing, a second visible light image and a second infrared image are obtained;
[0008] After the first visible light image and the second visible light image are subjected to multi-level multi-modal information fusion, a first visible light fusion image is obtained;
[0009] After the first infrared image and the second infrared image are subjected to multi-level multi-modal information fusion, a first infrared fusion image is obtained;
[0010] After the first visible light fusion image and the first infrared fusion image are respectively subjected to feature extraction by a self-attention convolution module, a third visible light image and a third infrared image are obtained;
[0011] The third visible light image and the third infrared image are subjected to secondary feature extraction by a self-attention convolution module to obtain a visible light fourth image and an infrared fourth image;
[0012] The visible light third image and the visible light fourth image are subjected to multi-level multi-modal information fusion to obtain a visible light second fusion image;
[0013] The infrared third image and the infrared fourth image are subjected to multi-level multi-modal information fusion to obtain an infrared second fusion image;
[0014] The visible light second fusion image and the infrared second fusion image are subjected to convolution processing respectively, and are spliced by channels to obtain a spliced image;
[0015] The spliced image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a first processed image;
[0016] The first processed image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a second processed image;
[0017] The second processed image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a third processed image;
[0018] The third processed image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a fourth processed image;
[0019] The fourth processed image is subjected to convolution processing, and then is subjected to a Tanh function to obtain a fifth processed image;
[0020] The fifth processed image is subjected to 255 processing to finally obtain a fusion image.
[0021] Preferably, the self-attention convolution module performs feature extraction, including the following steps:
[0022] The input value is subjected to convolution processing, and then is subjected to global pooling processing to obtain an initial 1-dimensional vector;
[0023] The initial 1-dimensional vector is processed by a sigmod function to obtain a processed 1-dimensional vector, and the values in the processed 1-dimensional vector are constrained in [0-1];
[0024] The processed 1-dimensional vector is multiplied by the initial 1-dimensional vector to obtain extracted features.
[0025] Further preferably, the multi-level multi-modal information fusion comprises the following steps:
[0026] 1) fusing the shallow features of infrared and the shallow features of visible light to obtain fused shallow features;
[0027] 2) using the fused shallow features to make a weight vector, and the specific formula is as follows:
[0028] ;
[0029] In the formula, fused shallow features are represented, 1*1 convolution operation is represented, which is used to balance the scales of shallow features and deep features, global average pooling is represented, which is used to extract information of the fused shallow features, sigmod function is represented, which is used to constrain the values of the extracted features in [0-1], the obtained weight vector is represented;
[0030] 3) element-wise multiplication with deep features to obtain fused deep features, and the specific formula is as follows:
[0031]
[0032] In the formula, channel-wise multiplication operation is represented, a vector with the same size as w and each element being 1 is represented, deep infrared features are represented, deep visible light features are represented, fused infrared light features after multi-level multi-modal fusion are represented, fused visible light features after multi-level multi-modal fusion are represented.
[0033] On the other hand, the present application also provides a fusion network of visible light images and infrared images, which comprises: a first convolutional layer, a second convolutional layer, a first attention convolutional module, a second attention convolutional module, a third attention convolutional module, a fourth attention convolutional module, a first multi-level multi-modal information fusion module, a fifth attention convolutional module, a sixth attention convolutional module, a seventh attention convolutional module, an eighth attention convolutional module, a second multi-level multi-modal information fusion module, a third convolutional layer, a fourth convolutional layer, a splicing module, four first convolutional activation layers, a second convolutional activation layer and a folding unit.
[0034] The first convolutional layers and the second convolutional layers are arranged in parallel as two input ports of the fusion network;
[0035] The input end of the first attention convolutional module is connected with the output end of the first convolutional layer;
[0036] The input end of the second attention convolutional module is connected with the output end of the first attention convolutional module;
[0037] The input end of the third attention convolutional module is connected with the output end of the second convolutional layer;
[0038] The input end of the fourth attention convolutional module is connected with the output end of the third attention convolutional module;
[0039] The input end of the first multi-level multi-modal information fusion module is connected with the output end of the first attention convolutional module, the output end of the second attention convolutional module, the output end of the third attention convolutional module and the output end of the fourth attention convolutional module respectively, for fusing the output image of the first attention convolutional module and the output image of the second attention convolutional module to obtain a first image, and fusing the image output by the third attention convolutional module and the image output by the fourth attention convolutional module to obtain a second image;
[0040] The input end of the fifth attention convolutional module is connected with the output end of the first multi-level multi-modal information fusion module, for extracting features from the first image output by the first multi-level multi-modal information fusion module;
[0041] The input end of the sixth attention convolutional module is connected with the output end of the fifth attention convolutional module;
[0042] The input end of the seventh attention convolutional module is connected with the output end of the first multi-level multi-modal information fusion module, for extracting features from the second image output by the first multi-level multi-modal information fusion module;
[0043] The input end of the eighth attention convolutional module is connected with the output end of the seventh attention convolutional module;
[0044] The input end of the second multi-level multi-modal information fusion module is connected with the output end of the fifth attention convolution module, the output end of the sixth attention convolution module, the output end of the seventh attention convolution module and the output end of the eighth attention convolution module respectively, for fusing the output image of the fifth attention convolution module and the output image of the sixth attention convolution module to obtain a third image, and fusing the image output by the seventh attention convolution module and the image output by the eighth attention convolution module to obtain a fourth image;
[0045] The third convolution layer is connected with the output end of the second multi-level multi-modal information fusion module, for performing convolution processing on the third image output by the second multi-level multi-modal information fusion module;
[0046] The fourth convolution layer is connected with the output end of the second multi-level multi-modal information fusion module, for performing convolution processing on the fourth image output by the second multi-level multi-modal information fusion module;
[0047] The input end of the splicing module is connected with the output end of the third convolution layer and the output end of the fourth convolution layer, the image output by the third convolution layer is spliced with the image output by the fourth convolution layer by channel to obtain a spliced image;
[0048] The spliced image is sequentially processed by four first convolution activation layers, and then processed by a second convolution activation layer, and finally processed by a folding and unfolding unit to obtain a finally fused image.
[0049] Preferably, the first attention convolution module, the second attention convolution module, the third attention convolution module, the fourth attention convolution module, the fifth attention convolution module, the sixth attention convolution module, the seventh attention convolution module and the eighth attention convolution module have the same function, specifically:
[0050] After the input value is processed by convolution, global pooling processing is performed to obtain an initial 1-dimensional vector;
[0051] The initial 1-dimensional vector is processed by a sigmod function to obtain a processed 1-dimensional vector, and the values in the processed 1-dimensional vector are constrained in [0-1];
[0052] The processed 1-dimensional vector is multiplied by the initial 1-dimensional vector to obtain an extracted feature.
[0053] Further preferably, the first multi-level multimodal information fusion module and the second multi-level multimodal information fusion module have the same function, specifically:
[0054] 1) The shallow features of infrared and the shallow features of visible light are fused to obtain the fused shallow features;
[0055] 2) Use the fused shallow features to create a weight vector, the specific formula is as follows:
[0056] ;
[0057] In the formula, This represents the shallow features after fusion. This represents a 1×1 convolution operation used to balance the scale of shallow and deep features. This indicates global average pooling, used to extract information from the fused shallow features. This refers to the sigmoid function, which constrains the extracted feature values to the range [0-1]. This represents the resulting weight vector;
[0058] 3) Element-wise multiplication with the deep features yields the fused deep features, as shown in the following formula:
[0059]
[0060] In the formula, This indicates a channel-by-channel multiplication operation. This represents a vector of the same size as w, where each element is 1. These are deep infrared features. For visible light deep features, The infrared light characteristics are obtained after multi-level and multi-modal fusion. It represents the visible light characteristics after multi-level and multi-modal fusion.
[0061] Further preferably, the first convolutional activation layer specifically involves: performing image processing... After convolution, it is then processed by the ReLU function.
[0062] Further preferred, the second convolutional activation layer specifically involves: performing image processing... After convolution, it is then processed by the Tanh function.
[0063] Further preferred, the loss function in the fusion network is:
[0064] ;
[0065] in, ;
[0066] ;
[0067] ;
[0068] wherein, represents gradient of image, represents absolute value, represents a norm, represents taking maximum value, represents infrared image, represents visible light image, H represents height of image, and W represents width of image, represents visible light image, represents visible light image sequence obtained by segmentation, represents infrared light image, represents fusion image, represents illumination vector of visible light image, represents illumination value corresponding to each visible light image block, represents illumination vector of infrared image, represents pixel loss of each fusion image block and visible light image block, represents pixel loss of each fusion image block and infrared image block, represents illumination degree of input visible light image block, represents illumination degree of input infrared image block, 、 is a natural number.
[0069] Further preferably, the illumination degree of image block wherein, is probability of day, is probability of night.
[0070] The fusion method of visible light image and infrared image provided by the present application integrates the advantages of two kinds of images by fusing visible light image and infrared image, and the generated fusion image not only includes cold background information, but also displays target information with thermal radiation, so as to improve the recognition effect of image.
[0071] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure of the present application. BRIEF DESCRIPTION OF DRAWINGS
[0072] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.
[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.
[0074] Figure 1 A composition structure schematic diagram of a visible light image and an infrared image fusion network provided by the present application is shown in the following figure.
[0075] Figure 2 A schematic diagram for feature extraction by a self-attention convolution module is shown in the following figure.
[0076] Figure 3 A schematic diagram for multi-level multi-modal information fusion is shown in the following figure. DETAILED DESCRIPTION
[0077] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. Unless otherwise indicated, the same numbers on different drawings represent the same or similar elements. The following exemplary embodiments described are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of methods consistent with some aspects of the present application as detailed in the appended claims.
[0078] In order to improve the recognition effect of the image and solve the problem of insufficient single image information, the present application provides a visible light image and infrared image fusion method, which comprises the following steps:
[0079] The input visible light image and infrared image are respectively subjected to convolution processing, and then are respectively subjected to feature extraction by a self-attention convolution module to obtain a first visible light image and a first infrared image.
[0080] After the first visible light image and the first infrared image are respectively subjected to self-attention convolution processing, a second visible light image and a second infrared image are obtained.
[0081] After the first visible light image and the second visible light image are subjected to multi-level multi-modal information fusion, a first visible light fusion image is obtained.
[0082] After the first infrared image and the second infrared image are subjected to multi-level multi-modal information fusion, a first infrared fusion image is obtained.
[0083] After the first visible light fusion image and the first infrared fusion image are respectively subjected to feature extraction by a self-attention convolution module, a third visible light image and a third infrared image are obtained.
[0084] The third visible light image and the third infrared image are subjected to secondary feature extraction by the self-attention convolution module to obtain a visible light fourth image and an infrared fourth image;
[0085] The visible light third image and the visible light fourth image are subjected to multi-level multi-modal information fusion to obtain a visible light second fusion image;
[0086] The infrared third image and the infrared fourth image are subjected to multi-level multi-modal information fusion to obtain an infrared second fusion image;
[0087] The visible light second fusion image and the infrared second fusion image are subjected to convolution processing respectively, and then are spliced by channels to obtain a spliced image;
[0088] The spliced image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a first processed image;
[0089] The first processed image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a second processed image;
[0090] The second processed image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a third processed image;
[0091] The third processed image is subjected to convolution processing, and then is subjected to a ReLU function to obtain a fourth processed image;
[0092] The fourth processed image is subjected to convolution processing, and then is subjected to a Tanh function to obtain a fifth processed image;
[0093] The fifth processed image is subjected to 255 processing to finally obtain a fusion image.
[0094] Since the traditional convolution pays equal attention to each channel, which is not reasonable, the self-attention convolution module is used for feature extraction in the above embodiment, as shown in Figure 2 , and the specific process of the feature extraction is as follows:
[0095] First, the input value is subjected to convolution operation, and then the result after the convolution is subjected to global pooling to obtain a 1-dimensional vector. Then, the 1-dimensional vector is processed by using a sigmod function to constrain the values in the 1-dimensional vector in [0-1]. Finally, the vector is multiplied by the value after the convolution to obtain an output result.
[0096] The specific steps are as follows:
[0097] Step 1.1. Convolution is performed on the input vector
[0098]
[0099] where x represents the result of convolution of the input value, conv represents convolution with a kernel of 3, and input represents the input value.
[0100] Step 1.2. Global pooling is performed on the result of convolution to obtain a one-dimensional vector
[0101]
[0102] represents global pooling, which can stretch the result of convolution into a one-dimensional vector, is a sigmod function, and is used to constrain the value in [0-1] to obtain the final one-dimensional vector.
[0103] Step 1.3. The one-dimensional vector is multiplied with the value after convolution
[0104]
[0105] represents multiplication of w and x by channel, and output represents the output result after self-attention convolution processing.
[0106] The above multi-level multi-modal information fusion, specifically, see Figure 3 , includes the following steps:
[0107] Shallow feature fusion is defined as follows:
[0108]
[0109] represents channel connection between channels, represents the infrared feature of the shallow layer, represents the visible light feature of the shallow layer, represents the fused shallow feature.
[0110] Step 2.2. A weight vector is made using the shallow feature, defined as follows:
[0111]
[0112] represents a 1x1 convolution operation used to balance the scales of the shallow feature and the deep feature, represents global average pooling used to extract information of the fused shallow feature. represents a sigmod function, which restricts the value of the extracted feature to [0-1], and the output represents the obtained weight vector.
[0113] Step 2.3: Element-wise multiplication is performed on the deep features to obtain the fused deep features, which are defined as follows:
[0114]
[0115] wherein represents a channel-wise multiplication operation, represents a channel-wise addition, represents a vector of the same size as w, whose each element is 1, is the deep feature, is the deep supplementary information extracted according to the shallow feature. represents the infrared / visible light feature after the MLFF module and the multi-level and multi-modal fusion.
[0116] The embodiment provides a visible light image and infrared image fusion network, referring to Figure 1 The fusion network mainly comprises a first convolutional layer 1, a second convolutional layer 2, a first attention convolutional module 3, a second attention convolutional module 4, a third attention convolutional module 5, a fourth attention convolutional module 6, a first multi-level and multi-modal information fusion module 7, a fifth attention convolutional module 8, a sixth attention convolutional module 9, a seventh attention convolutional module 10, an eighth attention convolutional module 11, a second multi-level and multi-modal information fusion module 12, a third convolutional layer 13, a fourth convolutional layer 14, a splicing module 15, four first convolutional activation layers 16, a second convolutional activation layer 17 and a folding and unfolding unit 18.
[0117] The first convolutional layer 1 and the second convolutional layer 2 are arranged in parallel and serve as two input ports of the fusion network, the input end of the first attention convolutional module 3 is connected with the output end of the first convolutional layer 1, the input end of the second attention convolutional module 4 is connected with the output end of the first attention convolutional module 3, the input end of the third attention convolutional module 5 is connected with the output end of the second The output end of the convolutional layer 2 is connected to the input end of the fourth attention convolutional module 6, the output end of the third attention convolutional module 5, the input end of the first multi-level multi-modal information fusion module 7 is connected to the output end of the first attention convolutional module 3, the output end of the second attention convolutional module 4, the output end of the third attention convolutional module 5 and the output end of the fourth attention convolutional module 6, for fusing the output images of the first attention convolutional module 3 and the second attention convolutional module 4 to obtain a first image, and fusing the images output by the third attention convolutional module 5 and the fourth attention convolutional module 6 to obtain a second image, the input end of the fifth attention convolutional module 8 is connected to the output end of the first multi-level multi-modal information fusion module 7, for extracting features from the first image output by the first multi-level multi-modal information fusion module 7, the input end of the sixth attention convolutional module 9 is connected to the output end of the fifth attention convolutional module 8, the input end of the seventh attention convolutional module 10 is connected to the output end of the first multi-level multi-modal information fusion module 7, for extracting features from the second image output by the first multi-level multi-modal information fusion module 7, the input end of the eighth attention convolutional module 11 is connected to the output end of the seventh attention convolutional module 10, the input end of the second multi-level multi-modal information fusion module 12 is connected to the output end of the fifth attention convolutional module 8, the output end of the sixth attention convolutional module 9, the output end of the seventh attention convolutional module 10 and the output end of the eighth attention convolutional module 11, for fusing the output images of the fifth attention convolutional module 8 and the sixth attention convolutional module 9 to obtain a third image, and fusing the images output by the seventh attention convolutional module 10 and the eighth attention convolutional module 11 to obtain a fourth image, the third The input end of the convolutional layer 13 is connected to the output end of the second multi-level multi-modal information fusion module 12, for performing convolutional processing on the third image output by the second multi-level multi-modal information fusion module 12 The input end of the convolutional layer 13 is connected to the output end of the second multi-level multi-modal information fusion module 12, for performing convolutional processing on the third image output by the second multi-level multi-modal information fusion module 12 The input end of the convolutional layer 14 is connected to the output end of the second multi-level multi-modal information fusion module 12, for performing convolutional processing on the fourth image output by the second multi-level multi-modal information fusion module 12 The input end of the convolutional layer 14 is connected to the output end of the second multi-level multi-modal information fusion module 12, for performing convolutional processing on the fourth image output by the second multi-level multi-modal information fusion module 12 The output end of the convolutional layer 13 and the output end of the fourth The output end of the convolutional layer 13 and the output end of the fourth The output end of the convolutional layer 13 and the output end of the fourth The output end of the convolutional layer 13 and the output end of the fourth
[0118] The first attention convolution module, the second attention convolution module, the third attention convolution module, the fourth attention convolution module, the fifth attention convolution module, the sixth attention convolution module, the seventh attention convolution module, and the eighth attention convolution module are the same in function, specifically as follows:
[0119] The input value is subjected to convolution processing, and then global pooling processing is performed to obtain an initial 1-dimensional vector;
[0120] The initial 1-dimensional vector is processed by using a sigmod function to obtain a processed 1-dimensional vector, and the values in the processed 1-dimensional vector are constrained in [0-1];
[0121] The processed 1-dimensional vector is multiplied by the initial 1-dimensional vector to obtain an extracted feature.
[0122] The first multi-level multi-modal information fusion module and the second multi-level multi-modal information fusion module are the same in function, specifically as follows:
[0123] 1) The shallow features of infrared and the shallow features of visible light are fused to obtain fused shallow features;
[0124] 2) The fused shallow features are used to make a weight vector, and the specific formula is as follows:
[0125]
[0126] In the formula, represents the fused shallow features, represents a 1x1 convolution operation, which is used to balance the scales of the shallow features and the deep features, represents a global average pooling, which is used to extract information of the fused shallow features, represents a sigmod function, which constrains the values of the extracted features in [0-1], represents the obtained weight vector;
[0127] 3) The deep features are multiplied element by element to obtain fused deep features, and the specific formula is as follows:
[0128]
[0129] In the formula, represents a channel-by-channel multiplication operation, represents a vector of the same size as w, and each element is 1, is the deep feature of infrared, is the deep feature of visible light, is the multi-level and multi-modal fused infrared light feature, The visible light features after multi-level and multi-modal fusion.
[0130] The first convolutional activation layer is specifically configured to perform convolution on the image. After convolution, the image is processed by a ReLU function.
[0131] The second convolutional activation layer is specifically configured to perform convolution on the image. After convolution, the image is processed by a Tanh function.
[0132] The loss function in the fusion network is specifically designed as follows:
[0133] 1. Design an illumination perception module
[0134] 1.1 Design a binary classification network
[0135] The binary classification network input is an image, and the output is the illumination level.
[0136] Step 1.1.1: Perform convolution with a kernel of 4, and then pass it through a ReLU function.
[0137] Step 1.1.2: Repeat step 1.1.1 three times, for a total of four times.
[0138] Step 1.1.3: Perform global pooling on the output of the previous step.
[0139] Step 1.1.4: Pass the output of the previous step through a fully connected layer.
[0140] Step 1.1.5: Repeat step 1.1.4 to obtain the output.
[0141] 1.2 Design an illumination perception module
[0142] Step 1.2.1: Let the 1.1 binary classification network be , and the network output is the probability that the input image is daytime and the probability that it is nighttime.
[0143]
[0144] The probability that it is daytime is , and the probability that it is nighttime is input.
[0145] Step 1.2.2: Normalize the output of 1.2.1 to obtain the illumination level.
[0146] , w is the illumination level of the input image block.
[0147] 2. Design a loss function
[0148] Step 2.1 Design details loss function:
[0149]
[0150] where, denotes the gradient of the image, in this paper, the Sobel operator is used to calculate the gradient, denotes the absolute value, denotes a norm, denotes the maximum value. denotes the infrared image, denotes the visible light image, H, W denotes the height and width of the image. is the basic loss function.
[0151] Step 2.2 Design the basic loss function:
[0152]
[0153] is the basic loss function.
[0154] Step 2.3 Design the area lighting loss:
[0155] Step 2.3.1 Define the image block rule:
[0156] Different areas in each picture have different lighting levels. It is obviously unreasonable to calculate the lighting loss for the whole picture. Therefore, the picture is divided into n small blocks, and the lighting level of each small block is calculated. The block definition is as follows:
[0157]
[0158] denotes each small image block obtained by segmentation, n denotes the number of small image blocks obtained by segmentation, denotes the input image.
[0159] Step 2.3.2 Define the area lighting vector:
[0160] For each small block of each picture, the lighting intensity of the small block needs to be obtained before calculating the loss. The lighting intensity w can be obtained by step 1.2.2. The lighting intensity sequence is defined as follows:
[0161]
[0162] denotes the lighting level corresponding to each small image block, denotes the lighting intensity vector corresponding to the segmented image sequence.
[0163] Step 2.3.3 Define the area lighting loss:
[0164]
[0165] In the formula, represents the visible light image, represents the segmented visible light image sequence, and the following symbols have similar definitions. represents the infrared light image, represents the fused image. represents the illumination vector of the visible light image, represents the illumination value corresponding to each small block of the visible light image, and the following symbols have similar definitions. represents the illumination vector of the infrared image. represents the pixel loss of each fused image block and the visible light image block, represents the pixel loss of each fused image block and the infrared image block.
[0166] wherein the pixel loss is defined as follows:
[0167]
[0168] represents the pixel loss of image a and image b, H and W represent the height and width of the image, represents a norm.
[0169] Step 2.4 The three loss functions of 2.1, 2.2 and 2.3 are fused to be the total loss function of the fusion network.
[0170] ;
[0171] In the formula, is a natural number, and L is the total loss of the fusion network.
[0172] Step 2.5 A binary classification network loss function is designed: cross-entropy loss is adopted.
[0173] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.
[0174] It should be understood that the application is not limited to the embodiments described above, and various modifications and changes can be made without departing from the scope thereof. The scope of the application is only limited by the appended claims.
Claims
1. A method of fusing a visible light image and an infrared image, characterized by, The method comprises the following steps: The input visible light image and infrared image are respectively processed After the convolution processing, the first visible light image and the first infrared image are obtained by the self-attention convolution module respectively. The first visible light image and the first infrared image are respectively subjected to self-attention convolution processing to obtain a second visible light image and a second infrared image; The first visible light image and the second visible light image are subjected to multi-level multi-modal information fusion to obtain a first visible light fusion image; The first infrared image and the second infrared image are subjected to multi-level multi-modal information fusion to obtain a first infrared fusion image; The first visible light fusion image and the first infrared fusion image are respectively subjected to feature extraction by a self-attention convolution module to obtain a third visible light image and a third infrared image; The third visible light image and the third infrared image are respectively subjected to secondary feature extraction by a self-attention convolution module to obtain a fourth visible light image and a fourth infrared image; The third visible light image and the fourth visible light image are subjected to multi-level multi-modal information fusion to obtain a second visible light fusion image; The third infrared image and the fourth infrared image are subjected to multi-level multi-modal information fusion to obtain a second infrared fusion image; The visible light second fusion image and the infrared second fusion image are respectively processed After the convolution processing, the channels are spliced to obtain a spliced image. The spliced image is subjected to After convolution processing, a first processed image is obtained through a ReLU function. The first processing image is subjected to After the convolution processing, a ReLU function is used to obtain a second processing image. performing convolution processing on the second processing image to obtain a third processing image After the convolution processing, a ReLU function is used to obtain the third processing image. The third processing image is subjected to After the convolution processing, a fourth processing image is obtained through a ReLU function. The fourth processing image is subjected to After the convolution processing, the fifth processing image is obtained through a Tanh function. the fifth processed image is obtained by processing the fourth processed image 255processing, and finally a fused image is obtained.
2. The method of claim 1, wherein the visible light image and the infrared image are fused by using a weighted average of the visible light image and the infrared image. The self-attention convolution module performs feature extraction, comprising the following steps: The input value is processed After the convolution processing, global pooling processing is performed to obtain an initial 1-dimensional vector; The initial 1D vector is processed by using a sigmod function to obtain a processed 1D vector, and the values in the processed 1D vector are constrained in [0-1]; The processed 1D vector is multiplied by the initial 1D vector to obtain extracted features.
3. The method of claim 1, wherein the step of fusing the visible light image and the infrared image is performed by using a weighted average of the visible light image and the infrared image. The multi-level multi-modal information fusion comprises the following steps: 1) The shallow features of infrared light and the shallow features of visible light are fused to obtain fused shallow features; 2) A weight vector is made by using the fused shallow features, and the specific formula is as follows: ； In the formula, represents the fused shallow features, represents a 1x1 convolution operation to balance the scale of the shallow features and the deep features, represents a global average pooling to extract information of the fused shallow features, represents a sigmod function to constrain the value of the extracted features in [0-1], represents the obtained weight vector; 3) The deep features are multiplied element by element to obtain fused deep features, and the specific formula is as follows: ； In the formula, represents a channel-by-channel multiplication operation, represents a vector of the same size as w, each element of which is 1, is a deep feature of infrared, is a deep feature of visible light, is an infrared light feature after multi-level and multi-modal fusion, is a visible light feature after multi-level and multi-modal fusion.
4. An apparatus for fusing a visible light image and an infrared image, characterized by comprising: The method comprises the following steps: first convolutional layer, second convolutional layer, first attention convolutional module, second attention convolutional module, third attention convolutional module, fourth attention convolutional module, first multi-level multi-modal information fusion module, fifth attention convolutional module, sixth attention convolutional module, seventh attention convolutional module, eighth attention convolutional module, second multi-level multi-modal information fusion module, third convolutional layer, fourth convolutional layer, splicing module, four first convolutional activation layers, second convolutional activation layer, and folding and unfolding unit; said first convolutional layers and said second convolutional layers are arranged in parallel as two input ports of said fusion device; An input end of the first attention convolution module is connected with an output end of the first convolution layer. An output end of the first attention convolution module is connected with an output end of the first convolution layer. The input end of the second attention convolution module is connected with the output end of the first attention convolution module; An input end of the third attention convolution module is connected with an output end of the second convolution layer The input end of the fourth attention convolution module is connected with the output end of the third attention convolution module; The input end of the first multi-level multi-modal information fusion module is connected with the output end of the first attention convolution module, the output end of the second attention convolution module, the output end of the third attention convolution module and the output end of the fourth attention convolution module, respectively, for fusing the output images of the first attention convolution module and the second attention convolution module to obtain a first image, and fusing the images output by the third attention convolution module and the fourth attention convolution module to obtain a second image; The input end of the fifth attention convolution module is connected with the output end of the first multi-level multi-modal information fusion module, for extracting features from the first image output by the first multi-level multi-modal information fusion module; The input end of the sixth attention convolution module is connected with the output end of the fifth attention convolution module. An input end of the seventh attention convolution module is connected with an output end of the first multi-level multi-modal information fusion module, and is configured to perform feature extraction on a second image output by the first multi-level multi-modal information fusion module; An input end of the eighth attention convolution module is connected with an output end of the seventh attention convolution module; An input end of the second multi-level multi-modal information fusion module is connected with output ends of the fifth attention convolution module, the sixth attention convolution module, the seventh attention convolution module and the eighth attention convolution module respectively, and is configured to fuse an output image of the fifth attention convolution module and an output image of the sixth attention convolution module to obtain a third image, and fuse an output image of the seventh attention convolution module and an output image of the eighth attention convolution module to obtain a fourth image; The third The input end of the convolutional layer is connected with the output end of the second multi-level multi-modal information fusion module, and is configured to perform convolutional processing on the third image output by the second multi-level multi-modal information fusion module. Convolutional processing The fourth The input end of the convolutional layer is connected with the output end of the second multi-level multi-modal information fusion module, and is configured to perform convolutional processing on the fourth image output by the second multi-level multi-modal information fusion module. Convolutional processing An input end of the splicing module is connected with the third An output end of the convolutional layer and the fourth The output end of the convolutional layer is connected with the third The image output by the third convolutional layer is spliced with the image output by the fourth convolutional layer by channel to obtain a spliced image; The spliced image is sequentially processed by the four first convolutional activation layers, and then is processed by a second convolutional activation layer, and finally is processed by a folding unit to obtain a final fused image.
5. The apparatus for fusing a visible light image and an infrared image according to claim 4, wherein, The first attention convolution module, the second attention convolution module, the third attention convolution module, the fourth attention convolution module, the fifth attention convolution module, the sixth attention convolution module, the seventh attention convolution module and the eighth attention convolution module have the same function, which is specifically as follows: The input value is processed After the convolution processing, global pooling processing is performed to obtain an initial 1-dimensional vector; The initial 1D vector is processed by using a sigmod function to obtain a processed 1D vector, and values in the processed 1D vector are constrained in [0-1]; The processed 1D vector is multiplied by the initial 1D vector to obtain extracted features.
6. The apparatus for fusing a visible light image and an infrared image according to claim 4, wherein, The first multi-level multi-modal information fusion module and the second multi-level multi-modal information fusion module have the same function, which is specifically as follows: 1) shallow features of infrared and visible light are fused to obtain fused shallow features; 2) a weight vector is made by using the fused shallow features, and a specific formula is as follows: ； wherein, represents the fused shallow features, represents a 1x1 convolution operation to balance the scale of the shallow features and the deep features, represents a global average pooling to extract information of the fused shallow features, represents a sigmod function to constrain the value of the extracted features in [0-1], represents the obtained weight vector; 3) the weight vector is multiplied by deep features element by element to obtain fused deep features, and a specific formula is as follows: ； In the formula, represents a channel-wise multiplication operation, represents a vector of the same size as w, each element of which is 1, is a deep feature of infrared, is a deep feature of visible light, is an infrared light feature after multi-level and multi-modal fusion, is a visible light feature after multi-level and multi-modal fusion.
7. The apparatus for fusing a visible light image and an infrared image according to claim 4, wherein The first convolutional activation layer specifically is: image is processed After convolution processing, the ReLU function is processed.
8. The apparatus for fusing a visible light image and an infrared image according to claim 4, wherein, The second convolutional activation layer specifically is: the image is processed After the convolutional processing, the image is processed by a Tanh function.
9. The apparatus for fusing a visible light image and an infrared image according to claim 4, wherein, The loss function used by the fusion device is as follows: ； wherein ; ； ； wherein, denotes taking gradient of an image, denotes absolute value, denotes a norm, denotes taking maximum value, denotes an infrared image, denotes a visible light image, H denotes height of the image, and W denotes width of the image, denotes a visible light image, denotes a sequence of visible light images obtained by segmentation, denotes an infrared light image, denotes a fused image, denotes an illumination vector obtained from a visible light image, denotes an illumination value corresponding to each small block of a visible light image, denotes an illumination vector of an infrared image, denotes pixel loss of each fused image block and visible light image block, denotes pixel loss of each fused image block and infrared image block, denotes illumination degree of an input visible light image block, denotes illumination degree of an input infrared image block, , is a natural number.
10. The apparatus for fusing a visible light image and an infrared image according to claim 9, wherein, Degree of illumination of an image block wherein, probability that it is daytime, probability that it is nighttime.