Deep learning based infrared and visible image fusion and object detection method

Abstract

The application discloses an infrared and visible light image fusion and target detection method based on deep learning, and belongs to the technical field of computer vision and artificial intelligence. The method comprises the following steps: acquiring multiple image pairs and preprocessing the acquired image pairs, and constructing an image sample dataset by using the preprocessed image pairs; wherein each image pair is composed of an infrared image and a visible light image; constructing an image fusion and target detection model by using a deep learning algorithm; training the constructed image fusion and target detection model by using the image sample dataset; wherein the model input is an image pair, and the output is a fusion image and a target detection result; and performing infrared and visible light image fusion and target detection by using the trained image fusion and target detection model. By using the technical scheme provided by the application, two visual tasks of infrared and visible light image fusion and target detection can be completed at the same time, and the quality of the fusion result and the performance of target detection can be improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method for infrared and visible light image fusion and target detection based on deep learning. Background Technology

[0002] In recent years, utilizing both infrared and visible light images to perform computer vision tasks has become a research hotspot. Infrared images captured by infrared sensors can record the thermal radiation emitted by different objects, and are minimally affected by changes in lighting and occlusion. However, infrared images have limited texture information because the texture of an object generates almost no heat. In contrast, visible light images capture spectral information of the reflections from different targets, containing discriminative details. However, in certain situations, such as nighttime environments, objects obscured by smoke, and complex backgrounds, identifying targets in visible light images can be challenging. Therefore, in practical tasks, simultaneously utilizing infrared and visible light images can enhance the task by providing different types of information.

[0003] With the tremendous success of deep learning technology in the field of computer vision, research on image fusion using infrared and visible light images [Ma J, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion[J]. Information fusion, 2019, 48: 11-26] and object detection [Zhang Y, Yu H, He Y, et al. Illumination-Guided RGBT Object Detection With Inter-and Intra-Modality Fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-13] is increasing. Infrared and visible light image fusion aims to create a unified fused image that combines the effective target regions of the infrared image with the rich detail information of the visible light image. These fused images are also valuable for object detection. Object detection aims to identify the location and category of targets in a visual scene, a task crucial in practical applications such as robotics, autonomous driving, and video surveillance.

[0004] Existing deep learning-based infrared and visible light image fusion and target detection methods first employ target detection as an optimization during the training of the image fusion network, forcing the network to generate a fused image with easily detectable targets. Then, these methods input the fused image into the target detection network and train the network again to improve the target detection results. These methods lack end-to-end training capabilities, failing to output both the fused image and target detection results simultaneously within a unified network. Furthermore, performing the image fusion task first and then the target detection task are two sequential steps, increasing the complexity and time required for processing. Summary of the Invention

[0005] This invention provides a deep learning-based method for infrared and visible light image fusion and target detection, addressing the shortcomings of existing technologies that lack end-to-end training capabilities and cannot simultaneously output fused images and target detection results within a unified network. Furthermore, performing image fusion followed by target detection involves two sequential steps, increasing the complexity and time required for processing.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] On one hand, the present invention provides a method for infrared and visible light image fusion and target detection based on deep learning, the method comprising:

[0008] Multiple image pairs are acquired and preprocessed. The preprocessed image pairs are then used to construct an image sample dataset. Each image pair consists of an infrared image and a visible light image acquired by an infrared imaging sensor and a light imaging sensor, respectively, in the same scene and are registered.

[0009] Image fusion and object detection models are constructed using deep learning algorithms;

[0010] The constructed image fusion and target detection model is trained using the image sample dataset. The input of the image fusion and target detection model is a pair of images that have been registered in the same scene, consisting of an infrared image and a visible light image acquired by an infrared imaging sensor and a light imaging sensor, respectively. The output is the fused image of the input infrared image and the visible light image, as well as the target detection result.

[0011] Using a trained image fusion and target detection model, infrared and visible light image fusion and target detection are performed on image pairs consisting of infrared and visible light images to be fused and detected.

[0012] Furthermore, the preprocessing of the acquired image pairs includes the following steps:

[0013] Adjust the single-channel infrared image in the image pair to a 3-channel infrared image;

[0014] Scale the infrared and visible light images in the image pair to a preset size;

[0015] Data augmentation operations are performed on the infrared and visible light images in the image pair respectively.

[0016] Furthermore, the data processing flow of the image fusion and target detection model is as follows:

[0017] Simple feature extraction operations are performed on the infrared image and the visible light image in the input image pair respectively. The first infrared feature map with 16 channels is extracted from the input infrared image, and the first visible light feature map with 16 channels is extracted from the input visible light image.

[0018] A difference fusion operation is performed on the first infrared feature map and the first visible light feature map to obtain a first infrared difference feature map and a first visible light difference feature map;

[0019] Simple feature extraction operations are performed on the first infrared difference feature map and the first visible light difference feature map respectively. A second infrared feature map with 32 channels is extracted from the first infrared difference feature map, and a second visible light feature map with 32 channels is extracted from the first visible light difference feature map.

[0020] A difference fusion operation is performed on the second infrared feature map and the second visible light feature map to obtain a second infrared difference feature map and a second visible light difference feature map;

[0021] Deep feature extraction operations are performed on the second infrared difference feature map and the second visible light difference feature map respectively. A third infrared feature map with 64 channels is extracted from the second infrared difference feature map, and a third visible light feature map with 64 channels is extracted from the second visible light difference feature map.

[0022] A difference fusion operation is performed on the third infrared feature map and the third visible light feature map to obtain a third infrared difference feature map and a third visible light difference feature map;

[0023] Deep feature extraction operations are performed on the third infrared difference feature map and the third visible light difference feature map respectively. A fourth infrared feature map with 128 channels is extracted from the third infrared difference feature map, and a fourth visible light feature map with 128 channels is extracted from the third visible light difference feature map.

[0024] A difference fusion operation is performed on the fourth infrared feature map and the fourth visible light feature map to obtain a fourth infrared difference feature map and a fourth visible light difference feature map;

[0025] Upsampling operations are performed on the second and third infrared difference feature maps respectively, so that the upsampled second and third infrared difference feature maps have the same size as the first infrared difference feature map. Then, the first infrared difference feature map and the upsampled second and third infrared difference feature maps are stitched together along the channel dimension to obtain the stitched infrared feature map. Finally, a simple feature extraction operation is performed on the stitched infrared feature map to obtain the dimensionality-reduced infrared feature map.

[0026] Upsampling operations are performed on the second and third visible light difference feature maps respectively, so that the upsampled second and third visible light difference feature maps have the same size as the first visible light difference feature map. Then, the first visible light difference feature map and the upsampled second and third visible light difference feature maps are stitched together along the channel dimension to obtain a stitched visible light feature map. Finally, a simple feature extraction operation is performed on the stitched visible light feature map to obtain a dimension-reduced visible light feature map.

[0027] The dimensionality-reduced infrared feature map and visible light feature map are stitched together along the channel dimension to obtain the first fused feature map;

[0028] The first fusion feature map is upsampled to the size of the preprocessed image, and the sampled first fusion feature map is input into the image fusion head. A fusion image of infrared and visible light images is obtained using a 5-layer convolutional structure.

[0029] The fourth infrared difference feature map and the fourth visible light difference feature map are fused using an attention mechanism to obtain a second fused feature map. The second fused feature map is then input into the target detection head to obtain the target detection result.

[0030] Furthermore, the simple feature extraction operation refers to extracting image features using a simple feature extraction module; wherein, the simple feature extraction module is composed of a stack of convolutional layers, batch normalization layers, and SiLU activation functions.

[0031] Furthermore, the deep feature extraction operation refers to extracting image features using a deep feature extraction module; wherein, the deep feature extraction module is composed of multiple simple feature extraction modules and residual connections stacked together.

[0032] Furthermore, the process of the difference fusion operation is as follows:

[0033] Subtracting the visible light feature map from the infrared feature map yields the infrared difference feature;

[0034] Subtracting the infrared feature map from the visible light feature map yields the visible light difference feature;

[0035] Global average pooling is performed on the infrared difference features and the visible light difference features respectively to obtain the infrared difference vector and the visible light difference vector;

[0036] The infrared difference vector and the visible light difference vector are respectively input into the fully connected layer. The processed result of the infrared difference vector output by the fully connected layer is multiplied with the infrared feature map to obtain the infrared feature. The processed result of the visible light difference vector output by the fully connected layer is multiplied with the visible light feature map to obtain the visible light feature.

[0037] The infrared feature map is added with infrared features and then input into the residual structure to obtain the residual infrared features; the visible light feature map is added with visible light features and then input into the residual structure to obtain the residual visible light features.

[0038] Adding residual infrared features to the infrared feature map yields the infrared difference feature map; adding residual visible light features to the visible light feature map yields the visible light difference feature map.

[0039] Further, the step of fusing the fourth infrared difference feature map and the fourth visible light difference feature map using an attention mechanism to obtain a second fused feature map includes:

[0040] Four simple feature extraction operations are performed on the fourth infrared difference feature map and the fourth visible light difference feature map to obtain the fifth infrared feature map and the fifth visible light feature map.

[0041] Using an attention mechanism, the attention features of the fifth infrared feature map and the fifth visible light feature map are calculated respectively. Then, the attention features of the fifth infrared feature map are added to the fourth infrared difference feature map to obtain enhanced infrared features. The attention features of the fifth visible light feature map are added to the fourth visible light difference feature map to obtain enhanced visible light features.

[0042] The enhanced infrared and enhanced visible light features are stitched together along the channel dimension, and then a simple feature extraction operation is performed on the stitched features to obtain the second fused feature map.

[0043] Furthermore, the data processing flow of the target detection head is as follows:

[0044] Perform multiple convolution operations on the second fused feature map to obtain multiple feature maps of different scales;

[0045] By processing multiple feature maps of different scales obtained from the feature pyramid structure, three prediction feature maps of different sizes are obtained.

[0046] Prediction operations are performed on prediction feature maps of different sizes to obtain object detection results.

[0047] Furthermore, the loss function L of the image fusion and object detection model is expressed as:

[0048] L=αL d +βL f

[0049] L d =α1L box +α2L obj +α3L cls

[0050] L f =β1L var_SSIM +β2L grad

[0051] Among them, L d Indicates detection loss; L f L represents the fusion loss; α and β are the coefficients balancing the detection loss term and the fusion loss term; box L represents regression loss; obj Indicates confidence loss; L cls Represents the classification loss; α1, α2, and α3 are respectively L box L obj L cls Balance coefficient; L var_SSIM L represents the variance-based structural similarity loss; grad This represents the gradient loss; β1 and β2 are respectively L var_SSIM and L grad The balance coefficient.

[0052] Furthermore, the variance-based structural similarity loss is expressed as:

[0053]

[0054]

[0055] Among them, I vi Represents a visible light image; I ir Indicates an infrared image; I f The generated fused image is represented by N; N represents the number of image patches after segmenting the infrared and visible light images respectively; SSIM represents the structural similarity loss; σ 2 (.) represents the image variance;

[0056] The gradient loss is expressed as:

[0057]

[0058] Where S represents the image size; ||·||1 represents the L1 norm; represents the Sobel gradient operator; |·| represents the absolute value operation; max represents the maximum value selection operation based on the element.

[0059] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0060] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the above method.

[0061] The beneficial effects of the technical solution provided by this invention include at least the following:

[0062] This invention can generate fusion and detection results simultaneously under end-to-end training, simplifying the previous complex two-stage training and improving the practicality of the method. Furthermore, by combining difference-based fusion methods and attention-based fusion methods, the image fusion model can retain sufficient detail information and focus on the target region during training, thereby obtaining high-quality fused images and superior detection results. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the execution flow of the infrared and visible light image fusion and target detection method based on deep learning provided in an embodiment of the present invention;

[0065] Figure 2 This is a data processing flowchart of the image fusion and target detection model provided in the embodiments of the present invention;

[0066] Figure 3 This is a system block diagram of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0068] First, it should be noted that in the embodiments of the present invention, the words "exemplarily," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplarily" is intended to present the concept in a specific manner. Furthermore, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either one or the other.

[0069] Furthermore, in the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0070] Furthermore, in embodiments of the present invention, sometimes a subscript (such as W1) may be mistakenly written as a non-subscript form (such as W1). Without emphasizing the difference, the meaning they express is the same.

[0071] First Embodiment

[0072] This embodiment provides a method for infrared and visible light image fusion and target detection based on deep learning. This method can be implemented by an electronic device, which can be a terminal or a server.

[0073] The execution flow of this method is as follows: Figure 1 As shown, it includes the following steps:

[0074] S1, acquire multiple pairs of images, preprocess the acquired image pairs, and construct an image sample dataset using the preprocessed image pairs; wherein, each image pair consists of an infrared image and a visible light image acquired by an infrared imaging sensor and a light imaging sensor respectively, which have been registered in the same scene.

[0075] In this embodiment, the multiple pairs of infrared and visible light images and their annotations are derived from M. 3 The FD dataset and the FLIR dataset. This example uses M... 3 893 pairs of images were selected for training and 158 pairs for testing from the FD dataset; 1101 pairs of images were selected for training and 169 pairs for testing from the FLIR dataset.

[0076] The preprocessing of the acquired image pairs includes the following steps:

[0077] S11, determine the number of channels in the infrared image. If it is a single channel, copy the single channel to generate a 3-channel infrared image.

[0078] S12, use scaling operations to obtain infrared and visible light images of size 640*640;

[0079] In this embodiment, the scaling operation utilizes OpenCV's resize function.

[0080] S13, using Mosaic data augmentation operation, scale and stitch four infrared images and visible light images into one image respectively;

[0081] S14, Modify the content of the target detection annotation file according to the corresponding image processing operation;

[0082] S15 divides multiple pairs of infrared and visible light image pairs into training and test sets according to a preset ratio.

[0083] S2 uses deep learning algorithms to construct an image fusion and object detection model;

[0084] S3, The constructed image fusion and target detection model is trained using the image sample dataset; wherein, the input of the image fusion and target detection model is an image pair consisting of an infrared image and a visible light image acquired by an infrared imaging sensor and a light imaging sensor respectively, which have been registered in the same scene, and the output is the fused image of the input infrared image and the visible light image and the target detection result;

[0085] S4. Using the trained image fusion and target detection model, perform infrared and visible light image fusion and target detection on the image pair consisting of infrared and visible light images to be fused and detected.

[0086] Specifically, in this embodiment, after obtaining the dataset, the model is constructed and trained through the following steps:

[0087] Step 1: Use the feature extraction module to extract features from the preprocessed infrared and visible light images, generating feature maps for both modalities; the specific process is as follows:

[0088] Step 11: Use a simple feature extraction module to process the infrared and visible light images respectively to obtain infrared and visible light feature maps with 16 channels; wherein, in this embodiment, the simple feature extraction module refers to a module composed of a convolutional layer, a batch normalization layer and a SiLU activation function stacked together.

[0089] Step 12: Use a simple feature extraction module to process the infrared and visible light feature maps with 16 channels respectively, and obtain infrared and visible light feature maps with 32 channels.

[0090] Step 13: Use the deep feature extraction module to process the infrared and visible light feature maps with 32 channels respectively to obtain infrared and visible light feature maps with 64 channels; wherein, in this embodiment, the deep feature extraction module refers to a module formed by stacking multiple simple feature extraction modules and residual connections.

[0091] Step 14: Use the deep feature extraction module to process the infrared and visible light feature maps with 64 channels respectively, and obtain infrared and visible light feature maps with 128 channels.

[0092] Step 2: Fuse infrared and visible light feature maps; the specific process is as follows:

[0093] Step 21: Use difference fusion to combine infrared and visible light feature maps;

[0094] Step 22: Use an attention mechanism to fuse infrared and visible light feature maps.

[0095] Furthermore, the method of fusing infrared and visible light feature maps using difference includes:

[0096] The feature maps of 16 channels, 32 channels, 64 channels, and 128 channels were subjected to interpolation fusion to obtain the fused infrared and visible light features. The interpolation fusion operation was as follows:

[0097] Subtracting the visible light feature map from the infrared feature map yields the infrared difference feature;

[0098] Subtracting the infrared feature map from the visible light feature map yields the visible light difference feature;

[0099] Global average pooling is performed on the infrared difference features and the visible light difference features respectively to obtain the infrared difference vector and the visible light difference vector;

[0100] The infrared difference vector and the visible light difference vector are input into the fully connected layer respectively, and the results are multiplied with the original infrared feature map and visible light feature map respectively to obtain the infrared feature and visible light feature.

[0101] The original infrared feature map and visible light feature map are respectively added with visible light features and infrared features, and then input into the residual structure to obtain residual infrared features and residual visible light features.

[0102] The original infrared and visible light feature maps are respectively supplemented with residual infrared and visible light features to obtain the difference feature map.

[0103] Furthermore, the method of fusing infrared and visible light feature maps using an attention mechanism includes:

[0104] Four simple feature extraction operations were performed on the infrared and visible light difference feature maps of the 128 channels after difference fusion;

[0105] Using an attention mechanism, the attention features of the convolved infrared and visible light are calculated and added to the original infrared and visible light features to obtain enhanced infrared and visible light features;

[0106] In this embodiment, the attention mechanism uses a projection function to convert the features extracted from simple features into keys, values, and queries; the query and key are multiplied and input into the Softmax function to obtain the attention feature map; the attention feature is obtained by multiplying the attention feature by the value.

[0107] Infrared and visible light features are stitched together along the channel dimension, and a simple feature extraction operation is performed on the stitched features.

[0108] Step 3: Construct image fusion and object detection models; the specific process is as follows:

[0109] Step 31: Construct an image fusion model to generate a fused image of infrared and visible light;

[0110] Step 32: Construct a target detection model to generate target detection results using the fused feature maps of infrared and visible light;

[0111] Furthermore, in this embodiment, constructing the image fusion model includes:

[0112] The difference feature maps of 32 and 64 channels of infrared and visible light are upsampled to the size of 16-channel difference feature maps, respectively. The difference feature maps of 16 channels, upsampled 32 channels and 64 channels are then stitched together along the channel dimension.

[0113] Simple feature extraction operations are performed on the stitched infrared and visible light feature maps respectively to reduce the number of channels and size of the stitched feature maps;

[0114] The extracted infrared and visible light feature maps are stitched together along the channel dimension to obtain the fused image feature map;

[0115] The fused image feature map is upsampled to the size of the preprocessed image, and then input into a 5-layer convolutional structure to obtain a fused image that combines infrared and visible light images.

[0116] Furthermore, in this embodiment, constructing the target detection model includes:

[0117] Multiple convolution operations are performed on the feature maps fused using the attention mechanism to obtain multiple feature maps of different sizes;

[0118] By using the feature pyramid structure to process feature maps of multiple different scales, three prediction feature maps of different sizes are obtained.

[0119] Prediction operations are performed on prediction feature maps of different sizes to obtain object detection results.

[0120] Step 4: Jointly train the image fusion and object detection models using the detection loss and image fusion loss functions.

[0121] The loss function resulting from the combined detection loss and image fusion loss function is expressed as follows:

[0122] L=αL d +βL f

[0123] L d =α1L box +α2L obj +α3L cls

[0124] L f =β1L var_SSIM +β2L grad

[0125] Among them, L d Indicates detection loss; L f L represents the fusion loss; α and β are the coefficients balancing the detection loss term and the fusion loss term; box L represents regression loss; obj Indicates confidence loss; L cls The classification loss is represented by α1, α2, and α3; the regression loss is calculated using the CIOU loss function; the confidence loss and classification loss are calculated using the cross-entropy loss function; α1, α2, and α3 are respectively L box L obj L cls The balance coefficients, in this embodiment, are α1 = 0.05, α2 = 1, and α3 = 0.5; L var_SSIM L represents the variance-based structural similarity loss; grad This represents the gradient loss; β1 and β2 are respectively L var_SSIM and L grad The balance coefficients are β1 = 10 and β2 = 0.5 in this embodiment.

[0126] Furthermore, the variance-based structural similarity loss is expressed as:

[0127]

[0128]

[0129] Among them, I vi Represents a visible light image; I ir Indicates an infrared image; I fThe generated fused image is represented by N; N represents the number of image patches after segmenting the infrared and visible light images respectively, with each image patch being 11*11 in size; W represents the index of the segmented image patch; SSIM represents the structural similarity loss; σ 2 (.) represents the image variance. The size of images is compared by their image variance.

[0130] The gradient loss is expressed as:

[0131]

[0132] Where N represents the image size; ||·|| represents the L1 norm; represents the Sobel gradient operator; |·| represents the absolute value operation; max represents the maximum value selection operation based on the element.

[0133] Based on the above, the data processing flow of the image fusion and target detection model in this embodiment is as follows: Figure 2 As shown, the main processing steps include the following:

[0134] Simple feature extraction operations are performed on the infrared image and the visible light image in the input image pair respectively. The first infrared feature map with 16 channels is extracted from the input infrared image, and the first visible light feature map with 16 channels is extracted from the input visible light image.

[0135] A difference fusion operation is performed on the first infrared feature map and the first visible light feature map to obtain a first infrared difference feature map and a first visible light difference feature map;

[0136] Simple feature extraction operations are performed on the first infrared difference feature map and the first visible light difference feature map respectively. A second infrared feature map with 32 channels is extracted from the first infrared difference feature map, and a second visible light feature map with 32 channels is extracted from the first visible light difference feature map.

[0137] A difference fusion operation is performed on the second infrared feature map and the second visible light feature map to obtain a second infrared difference feature map and a second visible light difference feature map;

[0138] Deep feature extraction operations are performed on the second infrared difference feature map and the second visible light difference feature map respectively. A third infrared feature map with 64 channels is extracted from the second infrared difference feature map, and a third visible light feature map with 64 channels is extracted from the second visible light difference feature map.

[0139] A difference fusion operation is performed on the third infrared feature map and the third visible light feature map to obtain a third infrared difference feature map and a third visible light difference feature map;

[0140] Deep feature extraction operations are performed on the third infrared difference feature map and the third visible light difference feature map respectively. A fourth infrared feature map with 128 channels is extracted from the third infrared difference feature map, and a fourth visible light feature map with 128 channels is extracted from the third visible light difference feature map.

[0141] A difference fusion operation is performed on the fourth infrared feature map and the fourth visible light feature map to obtain a fourth infrared difference feature map and a fourth visible light difference feature map;

[0142] Upsampling operations are performed on the second and third infrared difference feature maps respectively, so that the upsampled second and third infrared difference feature maps have the same size as the first infrared difference feature map. Then, the first infrared difference feature map and the upsampled second and third infrared difference feature maps are stitched together along the channel dimension to obtain the stitched infrared feature map. Finally, a simple feature extraction operation is performed on the stitched infrared feature map to obtain the dimensionality-reduced infrared feature map.

[0143] Upsampling operations are performed on the second and third visible light difference feature maps respectively, so that the upsampled second and third visible light difference feature maps have the same size as the first visible light difference feature map. Then, the first visible light difference feature map, the upsampled visible light infrared difference feature map, and the third visible light difference feature map are stitched together along the channel dimension to obtain a stitched visible light feature map. Then, a simple feature extraction operation is performed on the stitched visible light feature map to obtain a dimension-reduced visible light feature map.

[0144] The dimensionality-reduced infrared feature map and visible light feature map are stitched together along the channel dimension to obtain the first fused feature map;

[0145] The first fusion feature map is upsampled to the size of the preprocessed image, and the sampled first fusion feature map is input into the image fusion head. A fusion image of infrared and visible light images is obtained using a 5-layer convolutional structure.

[0146] The fourth infrared difference feature map and the fourth visible light difference feature map are fused using an attention mechanism to obtain a second fused feature map. The second fused feature map is then input into the target detection head to obtain the target detection result.

[0147] In this embodiment, the hyperparameter settings for training the image fusion and object detection models are as follows: the execution probability of the Mosaic operation is set to 0.5, the initial learning rate is 0.01, and the objective function is optimized using the SGD (Stochastic Gradient Descent) algorithm. The batch size is 16, and the training lasts for 300 epochs. The training GPU is an NVIDIA Tesla V100 GPU.

[0148] In this embodiment, in M 3 Image fusion and object detection models were tested on the FD and FLIR datasets, using four commonly used evaluation methods in the field of object detection: Precision, Recall, AP, and AP. 50 and AP 50:95 As the evaluation index, five commonly used evaluation methods in the field of image fusion, namely EN, SF, STD, MG, and SCD, were used as fusion evaluation indexes (the higher the above detection index and fusion index, the higher the performance of the detection method and fusion method). Various classic methods commonly used in this field were tested, and the performance evaluation results of different methods are shown in Table 1 and Table 2.

[0149] In this diagram, bold text represents the best-performing result. Yolov5s(V) indicates that the Yolov5s algorithm was executed using a single-modality visible light image, Yolov5s(I) indicates that the Yolov5s algorithm was executed using a single-modality infrared image, and the others use both infrared and visible light images. Secondary training refers to first generating a fused image, then performing detection training to generate detection results; primary training refers to simultaneously generating both the fused image and the detection results.

[0150] Table 1 Different methods in M 3 Performance evaluation results on the FD test set

[0151]

[0152] Table 2. Performance evaluation results of different methods on the FLIR test set.

[0153]

[0154]

[0155] As can be seen from Tables 1 and 2, the method provided in this embodiment has achieved excellent performance results under multiple different indicators, indicating that the method provided by the present invention can be effectively applied in practice.

[0156] In summary, this embodiment provides a deep learning-based infrared and visible light image fusion and target detection method that can simultaneously generate fusion and detection results under end-to-end training. This simplifies the previously complex two-stage training, improves the practicality of the method, and, by combining difference-based and attention-based fusion methods, enables the image fusion model to retain sufficient detail information while focusing on the target region during training, thereby obtaining high-quality fused images and superior detection results.

[0157] Second Embodiment

[0158] This embodiment provides an electronic device, such as... Figure 3 As shown, the electronic device includes a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. Furthermore, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0159] Below, in conjunction with Figure 3 A detailed introduction to each component of this electronic device is provided below:

[0160] The processor is the control center of the electronic device. The electronic device may include multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The term "processor" can refer to a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), other general-purpose processors, application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0161] In a specific implementation, as one example, the processor may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 shown are, of course, merely illustrative examples.

[0162] The memory is used to store the software program that executes the solution of the present invention, and the processor controls its execution. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0163] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may be integrated with the processor or may exist independently, and may be accessed through the interface circuit of the electronic device (…). Figure 3 (Not shown in the image) is coupled to the processor; however, this embodiment of the invention does not impose specific limitations on this.

[0164] The transceiver may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. The transceiver can be integrated with the processor or exist independently, and is connected through the interface circuit of the electronic device (…). Figure 3 (Not shown in the image) is coupled to the processor, and this embodiment of the invention does not specifically limit this.

[0165] In addition, it should be noted that, Figure 3 The structure of the electronic device shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or combine certain components, or have different component arrangements. Furthermore, the technical effects achieved by this electronic device when performing the method of the first embodiment described above can be referenced to the technical effects described in the first embodiment; therefore, they will not be repeated here.

[0166] Fourth embodiment

[0167] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0168] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely or partially hardware embodiment, a completely or partially software embodiment, or an embodiment combining software and hardware aspects. Moreover, when implemented in software, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium accessible to a computer or a data storage device such as a server or data center containing one or more sets of usable media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0169] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. Furthermore, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Additionally, the character " / " in this text generally indicates an "or" relationship between the preceding and following objects, but it can also indicate an "AND / OR" relationship. Please refer to the context for specific interpretations. "At least one" refers to one or more items, while "more than" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0172] Furthermore, it is understood that in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0173] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0174] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of functional modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0175] If the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for infrared and visible light image fusion and target detection based on deep learning, characterized in that, The deep learning-based infrared and visible light image fusion and target detection method includes: Multiple image pairs are acquired and preprocessed. The preprocessed image pairs are then used to construct an image sample dataset. Each image pair consists of an infrared image and a visible light image acquired by an infrared imaging sensor and a light imaging sensor, respectively, in the same scene and are registered. Image fusion and object detection models are constructed using deep learning algorithms; The constructed image fusion and target detection model is trained using the image sample dataset. The input of the image fusion and target detection model is a pair of images that have been registered in the same scene, consisting of an infrared image and a visible light image acquired by an infrared imaging sensor and a light imaging sensor, respectively. The output is the fused image of the input infrared image and the visible light image, as well as the target detection result. Using a trained image fusion and target detection model, infrared and visible light image fusion and target detection are performed on image pairs consisting of infrared and visible light images to be fused and detected. The data processing flow of the image fusion and target detection model is as follows: Simple feature extraction operations are performed on the infrared image and the visible light image in the input image pair respectively. The first infrared feature map with 16 channels is extracted from the input infrared image, and the first visible light feature map with 16 channels is extracted from the input visible light image. A difference fusion operation is performed on the first infrared feature map and the first visible light feature map to obtain a first infrared difference feature map and a first visible light difference feature map; Simple feature extraction operations are performed on the first infrared difference feature map and the first visible light difference feature map respectively. A second infrared feature map with 32 channels is extracted from the first infrared difference feature map, and a second visible light feature map with 32 channels is extracted from the first visible light difference feature map. A difference fusion operation is performed on the second infrared feature map and the second visible light feature map to obtain a second infrared difference feature map and a second visible light difference feature map; Deep feature extraction operations are performed on the second infrared difference feature map and the second visible light difference feature map respectively. A third infrared feature map with 64 channels is extracted from the second infrared difference feature map, and a third visible light feature map with 64 channels is extracted from the second visible light difference feature map. A difference fusion operation is performed on the third infrared feature map and the third visible light feature map to obtain a third infrared difference feature map and a third visible light difference feature map; Deep feature extraction operations are performed on the third infrared difference feature map and the third visible light difference feature map respectively. A fourth infrared feature map with 128 channels is extracted from the third infrared difference feature map, and a fourth visible light feature map with 128 channels is extracted from the third visible light difference feature map. A difference fusion operation is performed on the fourth infrared feature map and the fourth visible light feature map to obtain a fourth infrared difference feature map and a fourth visible light difference feature map; Upsampling operations are performed on the second and third infrared difference feature maps respectively, so that the upsampled second and third infrared difference feature maps have the same size as the first infrared difference feature map. Then, the first infrared difference feature map and the upsampled second and third infrared difference feature maps are stitched together along the channel dimension to obtain the stitched infrared feature map. Finally, a simple feature extraction operation is performed on the stitched infrared feature map to obtain the dimensionality-reduced infrared feature map. Upsampling operations are performed on the second and third visible light difference feature maps respectively, so that the upsampled second and third visible light difference feature maps have the same size as the first visible light difference feature map. Then, the first visible light difference feature map and the upsampled second and third visible light difference feature maps are stitched together along the channel dimension to obtain a stitched visible light feature map. Finally, a simple feature extraction operation is performed on the stitched visible light feature map to obtain a dimension-reduced visible light feature map. The dimensionality-reduced infrared feature map and visible light feature map are stitched together along the channel dimension to obtain the first fused feature map; The first fusion feature map is upsampled to the size of the preprocessed image, and the sampled first fusion feature map is input into the image fusion head. A fusion image of infrared and visible light images is obtained using a 5-layer convolutional structure. The fourth infrared difference feature map and the fourth visible light difference feature map are fused using an attention mechanism to obtain a second fused feature map. The second fused feature map is then input into the target detection head to obtain the target detection result. The simple feature extraction operation refers to extracting image features using a simple feature extraction module; wherein, the simple feature extraction module is composed of a stack of convolutional layers, batch normalization layers, and SiLU activation functions; The deep feature extraction operation refers to extracting image features using a deep feature extraction module; wherein, the deep feature extraction module is composed of multiple simple feature extraction modules and residual connections stacked together.

2. The infrared and visible light image fusion and target detection method based on deep learning as described in claim 1, characterized in that, The preprocessing of the acquired image pairs includes the following steps: Adjust the single-channel infrared image in the image pair to a 3-channel infrared image; Scale the infrared and visible light images in the image pair to a preset size; Data augmentation operations are performed on the infrared and visible light images in the image pair respectively.

3. The infrared and visible light image fusion and target detection method based on deep learning as described in claim 1, characterized in that, The process of the difference fusion operation is as follows: Subtracting the visible light feature map from the infrared feature map yields the infrared difference feature; Subtracting the infrared feature map from the visible light feature map yields the visible light difference feature; Global average pooling is performed on the infrared difference features and the visible light difference features respectively to obtain the infrared difference vector and the visible light difference vector; The infrared difference vector and the visible light difference vector are respectively input into the fully connected layer. Then, the processed result of the infrared difference vector output by the fully connected layer is multiplied with the infrared feature map to obtain the infrared features. The visible light feature is obtained by multiplying the visible light difference vector output by the fully connected layer with the visible light feature map. The infrared feature map is added with infrared features and then input into the residual structure to obtain the residual infrared features; The visible light feature map is added to the visible light feature map and then input into the residual structure to obtain the residual visible light feature map. Adding residual infrared features to the infrared feature map yields the infrared difference feature map; adding residual visible light features to the visible light feature map yields the visible light difference feature map.

4. The infrared and visible light image fusion and target detection method based on deep learning as described in claim 1, characterized in that, The process of fusing the fourth infrared difference feature map and the fourth visible light difference feature map using an attention mechanism to obtain a second fused feature map includes: Four simple feature extraction operations are performed on the fourth infrared difference feature map and the fourth visible light difference feature map to obtain the fifth infrared feature map and the fifth visible light feature map. Using an attention mechanism, the attention features of the fifth infrared feature map and the fifth visible light feature map are calculated respectively. Then, the attention features of the fifth infrared feature map are added to the fourth infrared difference feature map to obtain enhanced infrared features. The attention features of the fifth visible light feature map are added to the fourth visible light difference feature map to obtain enhanced visible light features. The enhanced infrared and enhanced visible light features are stitched together along the channel dimension, and then a simple feature extraction operation is performed on the stitched features to obtain the second fused feature map.

5. The infrared and visible light image fusion and target detection method based on deep learning as described in claim 1, characterized in that, The data processing flow of the target detection head is as follows: Perform multiple convolution operations on the second fused feature map to obtain multiple feature maps of different scales; By processing multiple feature maps of different scales obtained from the feature pyramid structure, three prediction feature maps of different sizes are obtained. Prediction operations are performed on prediction feature maps of different sizes to obtain object detection results.

6. The infrared and visible light image fusion and target detection method based on deep learning as described in claim 1, characterized in that, The loss function of the image fusion and object detection model Represented as: in, Indicates detection loss; Indicates fusion loss; and To balance the coefficients of the detection loss term and the fusion loss term; Indicates regression loss; Indicates confidence loss; Indicates classification loss; , and They are , , The balance coefficient; This represents variance-based structural similarity loss; Indicates gradient loss; and They are and The balance coefficient.

7. The infrared and visible light image fusion and target detection method based on deep learning as described in claim 6, characterized in that, The variance-based structural similarity loss is expressed as: in, Represents a visible light image; Represents an infrared image; This represents the generated fused image; N This indicates the number of image blocks after segmenting the infrared and visible light images, respectively. Represents structural similarity loss; Represents the variance of the image; The gradient loss is expressed as: in, S Indicates image size; Represents the L1 norm; This represents the Sobel gradient operator; This indicates the absolute value operation; max This indicates a maximum value selection operation based on the element.

Images