A target detection method and system based on infrared and visible light image fusion

Abstract

The application provides a target detection method and system based on infrared and visible light image fusion, and relates to the technical field of target detection, comprising: acquiring visible light and infrared images to be detected; inputting the acquired visible light and infrared images to be detected into a trained YOLOv8 double-flow network model to output a target detection result; wherein the training of the YOLOv8 double-flow network model comprises: acquiring data of visible light and infrared images of different time periods in a specified area to construct a target joint data set; building a YOLOv8 double-flow network model; training the built YOLOv8 double-flow network model by using the constructed target joint data set to obtain a trained YOLOv8 double-flow network model. The method proposed in the application solves the problem that CNN is difficult to capture long-distance dependence and global relationship between different modalities by building a YOLOv8 double-flow network model, using a fusion module to perform feature fusion and target detection on visible light and infrared images, and improves the accuracy and robustness of target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a target detection method and system based on infrared and visible light image fusion. Background Art

[0002] Remote sensing image target detection has always been a key direction of remote sensing image processing. As the core target of battlefield reconnaissance and the key carrier of maritime trade, ships occupy an important position in remote sensing image interpretation and target detection.

[0003] Traditional ship detection methods generally follow a coarse-to-fine processing pipeline: first, image land and sea separation is performed, followed by preprocessing such as declouding and denoising. Candidate regions that may contain ship targets are extracted, and finally, the ship targets in each candidate region are classified. The extraction of candidate regions is particularly critical, and the main methods for extracting candidate regions include grayscale information statistics, visual saliency analysis, template matching, and classification learning. However, traditional ship detection algorithms overly rely on hand-crafted features, resulting in limited applicability, poor real-time performance, and low efficiency in batch processing of remote sensing imagery. Furthermore, traditional ship detection algorithms focus more on hand-crafted shallow structures and fail to fully utilize deep feature information such as texture, spectrum, and geometry in the image. As a result, their detection results are difficult to meet the needs of practical applications.

[0004] Therefore, it is necessary to study a target detection method based on infrared and visible light image fusion. Summary of the Invention

[0005] The purpose of the present invention is to address the problems existing in the prior art and provide a target detection method based on the fusion of infrared and visible light images. By building a YOLOv8 dual-stream network model, feature fusion and target detection are performed on visible light and infrared images. The feature fusion is performed using a self-attention bimodal feature fusion module, which more effectively captures the complementarity and correlation between different modalities and achieves significant target detection accuracy.

[0006] According to one aspect of the present invention, a target detection method based on infrared and visible light image fusion is provided, comprising:

[0007] Acquire visible light and infrared images to be detected;

[0008] The obtained visible light and infrared images to be detected are input into the trained YOLOv8 two-stream network model to output the target detection results. The training of the YOLOv8 two-stream network model includes:

[0009] Obtain visible light and infrared image data of a specified area at different time periods and construct a target joint dataset;

[0010] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction from the input visible light and infrared images; an aggregation bottleneck module for aggregating the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of the visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0011] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0012] Furthermore, a target joint dataset is constructed, including:

[0013] Select the band of specified resolution;

[0014] The selected bands are converted into different formats, and visible light and infrared images with different contrasts are obtained by controlling the truncation percentage.

[0015] Furthermore, the aggregated visible light feature map and infrared feature map are fused, including:

[0016] Use two feature extractors to extract features from visible light and infrared feature maps respectively;

[0017] Decomposing the extracted visible light and infrared features;

[0018] A self-attention bimodal feature fusion module is used to fuse the decomposed visible light and infrared features to generate a fused feature map;

[0019] Adopting a feature selection module based on the attention mechanism, the attention mechanism is used to extract features from each feature in the fusion feature map;

[0020] A bidirectional high-frequency transmission module based on discrete cosine transform is used to enhance the features of the fused feature map after feature extraction.

[0021] Furthermore, after obtaining the feature-enhanced fusion feature map, the following steps are included:

[0022] The fused feature map is restored to its original size through upsampling;

[0023] The feature map restored to its original size is divided into two parts, each containing visible light and infrared information;

[0024] The split fusion feature maps are added together and projected onto the single-modal feature map before fusion to obtain the final fusion feature map.

[0025] According to one aspect of the present invention, there is provided a target detection system based on infrared and visible light image fusion, comprising:

[0026] The detection data acquisition module is used to obtain the visible light and infrared images to be detected.

[0027] The target detection module inputs the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model and outputs the target detection results. The training of the YOLOv8 two-stream network model includes:

[0028] Obtain visible light and infrared image data of a specified area at different time periods and construct a target joint dataset;

[0029] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction from the input visible light and infrared images; an aggregation bottleneck module for aggregating the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of the visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0030] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0031] According to one aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the target detection method based on infrared and visible light image fusion when executing the computer program.

[0032] According to one aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the target detection method based on infrared and visible light image fusion are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This paper proposes a target detection method based on infrared and visible light image fusion. By building a YOLOv8 dual-stream network model, it performs feature fusion and target detection on visible light and infrared images, and has achieved remarkable experimental results.

[0035] 2. The present invention proposes a target detection method based on the fusion of infrared and visible light images. By acquiring data of visible light and infrared images of a specified area at different time periods, a joint target dataset is constructed, providing data support for target detection experiments.

[0036] 3. The present invention proposes a target detection method based on the fusion of infrared and visible light images. It proposes a fusion module through the neck network of the YOLOv8 dual-stream network model to extract, decompose and fuse the features of visible light feature maps and infrared feature maps, more effectively capturing the complementarity and correlation between different modalities, and solving the problem that CNN has difficulty in capturing long-distance dependencies and global relationships between different modalities.

[0037] 4. The present invention proposes a target detection method based on infrared and visible light image fusion, which can capture the global information and local details of the target through multi-scale fusion of visible light-infrared fusion features, and helps to improve the accuracy and robustness of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flow chart of a target detection method based on infrared and visible light image fusion provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of the YOLOv8 dual-stream network model training provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the SAF module network structure provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the network structure of the Encoder module provided in an embodiment of the present invention;

[0043] Figure 5 The SAF module provided in the embodiment of the present invention uses a self-attention mechanism to learn the binary relationship between visible light and infrared modalities and uses a correlation matrix to weight the input feature map;

[0044] Figure 6 A schematic diagram of the FSM module network structure provided in an embodiment of the present invention;

[0045] Figure 7Schematic diagram of the network structure of the CBAM hybrid attention module provided by an embodiment of the present invention;

[0046] Figure 8 Schematic diagram of the channel attention module network structure of the CAM output provided by an embodiment of the present invention;

[0047] Figure 9 Schematic diagram of the spatial attention structure of the SAM output provided by an embodiment of the present invention;

[0048] Figure 10 Schematic diagram of mAP@0.5-epoch curves of different algorithms in embodiments of the present invention;

[0049] Figure 11 Schematic diagram of mAP@[.5:.95]-epoch curves of different algorithms in embodiments of the present invention;

[0050] Figure 12 A module diagram of a target detection system based on infrared and visible light image fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that:

[0052] The aggregation bottleneck module refers to a cross-stage partial aggregation bottleneck module with two convolutions, also known as the C2f module;

[0053] Fusion module refers to Fusionblock fusion module;

[0054] The self-attention bimodal feature fusion module refers to the (Self Attention Fusion) SAF module;

[0055] The feature selection module based on the attention mechanism refers to the (Feature Selected Module) FSM module;

[0056] The bidirectional high-frequency transmission module based on discrete cosine transform refers to the (Discrete Cosine Transform) DCT-based (Mutual High-frequency Transfer) MHT module.

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] The embodiment of the present invention provides a method for target detection based on infrared and visible light image fusion, such as Figures 1-2 Shown, including:

[0059] Acquire visible light and infrared images to be detected;

[0060] The obtained visible light and infrared images to be detected are input into the trained YOLOv8 two-stream network model to output the target detection results. The training of the YOLOv8 two-stream network model includes:

[0061] Obtain visible light and infrared image data of a specified area at different time periods and construct a target joint dataset;

[0062] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction from the input visible light and infrared images; an aggregation bottleneck module for aggregating the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of the visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0063] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0064] Specifically, in the embodiment of the present invention, the target joint dataset is constructed by using ENVI software, selecting four bands 2, 3, 4, and 8 (Blue, Green, Red, and NIR) with a resolution of 10 m, converting the original data into tiff format, and then using Python's GDAL library to convert the tiff format into jpg format. By controlling the truncation percentage, visible light and infrared images with different contrasts are obtained.

[0065] The embodiment of the present invention also provides a model evaluation index for a target detection method based on the fusion of infrared and visible light images, wherein TP, FP, TN, and FN are basic classification result indicators used to describe the accuracy of the model prediction results. TP (True Positive) is the number of samples that are actually positive and predicted to be positive; FP (False Positive) indicates the number of samples that are actually negative but are mistakenly predicted to be positive, also known as false positives; TN (True Negative) indicates the number of samples that are actually negative and are predicted to be negative; FN (False Negative) indicates the number of samples that are actually positive but are mistakenly predicted to be negative, also known as missed reports. Important evaluation indicators derived from the above basic indicators include:

[0066] P (Precision): Precision, which is the proportion of correctly predicted instances among all positive instances predicted by the model, reflects the accuracy of the model in predicting positive instances, and the calculation formula is as follows:

[0067]

[0068] R (Recall): Recall, also known as the true positive rate, is the proportion of correctly detected positive instances among all true positive instances, reflecting the ability of the model to detect all true instances, and the calculation formula is as follows:

[0069]

[0070] Accuracy: Accuracy, which represents the proportion of correctly classified samples among all samples, and the calculation formula is as follows:

[0071]

[0072] With recall as the horizontal coordinate and accuracy as the vertical coordinate, the curve formed by recall and accuracy is called the P-R curve. AP is the average value of precision at different recall levels, and its value is the area under the P-R curve, while mAP represents the average of all class APs in the entire dataset, and the calculation formula is as follows:

[0073]

[0074] where N represents the number of classes, and mAP has different evaluation indicators, among which mAP@0.5 represents the average precision when the IoU threshold is set to 0.5, and mAP@[.5:.95] is a more comprehensive evaluation indicator that considers various cases of IoU threshold from 0.5 to 0.95, and calculates the average precision mean value of IoU threshold from 0.5 to 0.95.

[0075] Specifically, based on the target joint dataset, a YOLOv8 dual-stream network model is built and trained, and visible light and infrared images are jointly used as the input of the network, and then the backbone network of the YOLOv8 dual-stream network model is used for feature extraction, and then the C2f module is used to output visible light feature maps and infrared feature maps.

[0076] Specifically, a fusion module is proposed based on the neck network of the YOLOv8 two-stream network model to extract, decompose and fuse the visible light feature map and the infrared feature map to obtain the visible light-infrared fusion feature, which includes: receiving input data from different modalities, using the corresponding feature extractor to extract the feature map for each modality, and pooling the feature maps of the two modalities to a smaller size, which not only reduces the amount of calculation but also retains the key information in the feature map; on the pooled feature map, calculating the attention weights of each position in the feature map of one modality and each position in the feature map of the other modality, which reflect the correlation between elements of different modalities; according to the calculated self-attention weights, the feature maps of the two modalities are fused to generate a fused feature map containing cross-modal correlation; the fused feature map is restored to its original size by upsampling, and then the feature map is split into two parts, each part containing information about the correlation between the two modalities; finally, the two fused feature maps are projected onto the original single-modal feature map in an additive manner. The correlation between the two modalities is modeled through the self-attention mechanism, which enhances the representation ability of the single-modal feature map. The SAF module can effectively capture and utilize the correlation between different modalities, thereby improving the performance of the model on cross-modal tasks. At the same time, through pooling and upsampling strategies, the SAF module reduces the computational complexity while maintaining performance, making it more suitable for practical application scenarios.

[0077] like Figure 3 As shown, the embodiment of the present invention also provides a specific network structure of the SAF module, wherein and The feature maps representing the visible light and infrared modes are pooled globally, and then the feature maps of the two modes are flattened and concatenated. The size of the flattened and concatenated feature maps is C×2HW. Represents the feature map for which self-attention is to be calculated.

[0078] Specifically, the SAF module uses the Encoder module in the Transformer to perform self-attention calculations. The Encoder module is as follows: Figure 4 As shown, it is the core component of the self-attention mechanism. It can effectively calculate and update the representation of each token to reflect its relevance to all other tokens, where the model can better understand the global structure of the input data, thereby improving performance on various tasks. In an embodiment of the present invention, the SAF module downsamples the calculated feature map to reduce the computational complexity of the self-attention mechanism. By reducing the size of the feature map, the number of elements involved in the self-attention calculation is significantly reduced, reducing the amount of calculation and improving computational efficiency.

[0079] Specifically, the specific workflow of the Encoder module is as follows:

[0080] The input feature map is positionally encoded before processing, ensuring that the model can take into account the information at each position in the feature map;

[0081] The two-dimensional feature map of the input Encoder is ;

[0082] Positional encoding The size and input features Figure 1 It is directly fused with the input feature map to provide the model with the position information of each pixel, and the position encoding is learnable during training;

[0083] The positional encoding is adjusted as training progresses to better suit the task and data. The overall forward propagation process is formulated as follows:

[0084]

[0085] in, and Represent input and output respectively, and Represents the intermediate feature map of the residual connection, ΦNorm() represents LayerNorm, ΦMLP() represents linear mapping, and ΦAtt() represents multi-head self-attention. The calculation formula for single-head self-attention is as follows:

[0086]

[0087] Among them, Q, K, and V are generated according to the input vector. is the weight parameter that needs to be learned, A represents the attention matrix after Softmax, and O represents the output matrix of the single-head self-attention calculation;

[0088]

[0089] Among them, split() will Divide into equal proportions and , Up() is the upsampling operation, and The sample is the size of the input feature map, and the fused feature map is split and projected onto the original input feature map in a residual manner. and F' ir Represented as the visible light and infrared modal feature maps output by the SAF module.

[0090] Specifically, the SAF module uses the self-attention mechanism to learn the binary relationship between visible and infrared modalities and uses the correlation matrix to weight each position of the input feature map. Figure 5As shown, when calculating the correlation matrix, four matrix blocks can be naturally inferred, two of which are intra-modal correlation matrix blocks (visible light and infrared) and the other two are inter-modal correlation matrix blocks.

[0091] Specifically, a feature selection module (FSM) based on the attention mechanism is used to sort the importance of features within the modality to select rich semantic features to solve the problem that low-information-value areas affect the scores of high-information-value areas. Figure 6 Schematic diagram of the feature selection module structure based on the attention mechanism.

[0092] Specifically, the feature selection module based on the attention mechanism filters low-semantic features by adaptively assigning weights within the modality and enriches the semantic representation within the modality by combining consistent information between modalities. The steps are as follows:

[0093] The weight coefficients for each channel are obtained by normalizing the features derived from the modality;

[0094] Multiply each channel with the corresponding weight coefficient to obtain a sparse weight penalty and suppress the less important channel expressions. Given the modality and r and i, define the adaptive weight of the intra-modal feature, the expression is as follows:

[0095]

[0096] Among them, BN is batch normalization, S is the Sigmoid function, is the weight parameter of a specific module, is the weight parameter of a given mode i, is the weight parameter of the given mode r.

[0097] In this way, we can obtain fine intra-modal features, and the Inter-modal Attention Module uses the CBAM hybrid attention module to further select features. Among them, CBAM (Convolutional Block Attention Module) uses the attention module to enhance the network's ability to process information of different scales, shapes, and directions. It can adaptively learn the importance of features and use this information to weight feature maps, thereby improving the model's expressive power. Figure 7 As shown in Figure 3, CBAM mainly consists of two sequential sub-modules: Channel Attention Module (CAM) and Spatial Attention Module (SAM).

[0098] Specifically, ifFigure 8 As shown, the feature map input by CAM will undergo global maximum pooling and global average pooling operations for each channel. Global maximum pooling can extract the most significant features in the feature map, while global average pooling can take into account the average effect of all features. These two pooling operations can extract global information at the channel level from different perspectives. The outputs of the two pooling operations will be sent to the multi-layer perceptron (MLP) respectively. The MLP will help the model understand which channel features are more important. The outputs of the two MLPs are fused, and the fused features are finally passed through the Sigmoid function to obtain the channel attention weight map. CAM can effectively strengthen the weights of important features and weaken the weights of unimportant features. The calculation formula is as follows:

[0099]

[0100] in, They are represented as global average pooling features and maximum pooling features respectively.

[0101] like Figure 9 As shown in Figure 1, SAM takes as input the feature map output by the channel attention module. The channel attention module weights the different channels. The feature map then undergoes global max pooling and global average pooling along the channel dimension. After pooling, two single-channel feature maps are generated, representing the results of max pooling and average pooling, respectively. These two feature maps are concatenated along the channel dimension to form a dual-channel feature map. This concatenated dual-channel feature map then passes through a 7×7 convolutional layer, which further integrates and refines spatial information while reducing its dimensionality to a single-channel feature map. The 7×7 convolution kernel provides a relatively large receptive field, helping to capture spatial context. The resulting single-channel feature map is then passed through a sigmoid activation function to generate a spatial attention weight map, which represents the importance of different spatial locations in the original feature map. This spatial attention weight map can be directly multiplied with the original input feature map to emphasize important spatial locations and suppress unimportant ones. Through this process, SAM effectively improves the model's focus on critical spatial information. Its calculation formula is as follows:

[0102]

[0103] Among them, F is the input feature map, AvgPool(F) and MaxPool(F) represent the results of global average pooling and global maximum pooling operations on the feature map respectively. Represented as Sigmoid activation function, is a 7×7 convolutional layer, is the spatial attention weight map.

[0104] Specifically, after using the CBAM module, the inter-modal features are inverted into intra-modal features, and finally visible light and infrared features with consistent representation are obtained and combined into the final features. While interacting with the features, rich semantic intra-modal features are obtained by selecting features, and the consistency of inter-modal features is further enhanced in the fusion process.

[0105] Specifically, a DCT-based bidirectional high-frequency transmission module extracts and exchanges high-frequency components of visible light and infrared, converting the spatial domain of an image into the spectral domain. A common method is the discrete Fourier transform (DFT). In this embodiment of the present invention, even if the input signal in the DFT is a real number, the conversion result also contains complex numbers. Through specific operations, the corresponding values ​​containing complex numbers can be calculated. The signal is decomposed into a cosine function; the DCT is then used for operation. When represented as a spectrum, only real values ​​are output. A discrete signal input in the spatial domain of a two-dimensional image of size N×M can be represented in the frequency domain using the DCT. The corresponding two-dimensional DCT is as follows:

[0106]

[0107] Among them, f[i,j] is the pixel value of the two-dimensional input image at (i,j), F[x,y] is the DCT coefficient value at (x,y), is the definition of cosine basis function;

[0108]

[0109] is the definition of the normalization constant, f[i,j] represents a two-dimensional signal converted from the space domain to the frequency domain through DCT, and can be transformed back to the spatial domain through the inverse DCT (IDCT).

[0110] In the DCT domain, low-frequency components are concentrated in the upper left corner, while high-frequency components are concentrated in the lower right corner, allowing only high-frequency components to be extracted. Each feature map in the Mutual High-Frequency Transfer (MHT) module receives input from both the visible and infrared streams. Each feature map undergoes a DCT and is converted from the spatial domain to the spectral domain. The converted spectral domain is located in the upper left corner, while low-frequency and high-frequency features appear in the lower right corner. The high-frequency components of the input feature map are extracted by removing the portion in the upper left corner. If the removed portion is too small, the high-frequency and low-frequency components are not well separated, leading to overfitting of the low-frequency components during model learning. If the removed portion is too large, most of the edge information (high-frequency components that are important for object recognition) is lost, resulting in minimal performance improvement. Therefore, to account for low-frequency overfitting and edge loss, approximately 4% of the total low-frequency components are removed. The spectral domain is then converted back to the spatial domain using the IDCT, resulting in a high-frequency feature map with relatively few low-frequency components. The MHT allows for more efficient information exchange between the visible and infrared streams, thereby improving multispectral object detection performance.

[0111] Specifically, the embodiment of the present invention uses the deep learning framework PyTorch for experiments, with a learning rate of 0.001, a batch size of 16, an IoU threshold of 0.5, and 500 iterations. The specific experimental configuration environment is shown in Table 1:

[0112]

[0113] Specifically, the target detection performance of the YOLOv8 algorithm proposed in the embodiment of the present invention on the visible light image in the target joint dataset is shown in Table 2. YOLOv5, YOLOX, YOLOv7 and YOLOv8 are selected as comparison algorithms.

[0114]

[0115] The curves of mAP@0.5-epoch and mAP@[.5:.95] of the algorithms in Table 2 are as follows Figure 10 and Figure 11 shown.

[0116] It can be seen that the improved YOLOv8 target detection algorithm has the highest rise and rises rapidly. The performance of several other YOLO series algorithms decreases with the lower versions. Among them, the YOLOv5 algorithm curve has the lowest height and the fluctuation is more obvious.

[0117] The implementation of each embodiment of the present invention is based on programmed processing performed by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of each embodiment of the present invention are packaged into various modules. Based on this reality, and in addition to the aforementioned embodiments, an embodiment of the present invention provides a target detection system based on infrared and visible light image fusion. This system is used to implement the target detection method based on infrared and visible light image fusion described in the aforementioned method embodiment.

[0118] See also Figure 12 , the system comprises:

[0119] The detection data acquisition module is used to obtain the visible light and infrared images to be detected.

[0120] The target detection module inputs the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model and outputs the target detection results. The training of the YOLOv8 two-stream network model includes:

[0121] Obtain visible light and infrared image data of a specified area at different time periods and construct a target joint dataset;

[0122] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction of input visible light and infrared images; an aggregation bottleneck module for aggregating the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0123] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0124] An embodiment of the present invention provides a target detection system based on infrared and visible light image fusion, the specific implementation process of which includes: obtaining visible light and infrared images to be detected, inputting the obtained visible light and infrared images to be detected into a trained YOLOv8 two-stream network model, and outputting target detection results; wherein the training of the YOLOv8 two-stream network model includes: collecting visible light and infrared image data of different time periods in a specified area to construct a target joint data set, preprocessing the collected image data, annotating the targets in the preprocessed image data, and dividing the annotated data set into a training set and a validation set; using YOLOv8 to detect the target. The LOv8 algorithm is used as the basic framework to construct a two-stream network model. The backbone network of the two-stream network model is used to extract features of the input visible light and infrared images, and the C2f module is used to output visible light feature maps and infrared feature maps. The Fusionblock fusion module is used to extract, decompose and fuse the visible light feature maps and infrared feature maps, and the self-attention mechanism is used to capture the correlation between the two modalities to obtain visible light-infrared fusion features. The visible light-infrared fusion features are fused at multi-scale and the multi-scale fusion features are input into the detection head of the YOLOv8 two-stream network model for target detection to generate target detection results.

[0125] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides an electronic device, including a memory and a processor, the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the target detection method based on infrared and visible light image fusion proposed in the above-mentioned embodiment.

[0126] This embodiment also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program is used to implement feature fusion and target detection for visible light and infrared images, providing richer information for multimodal feature fusion and improving target detection accuracy. The storage medium can be any non-volatile storage device, such as a hard disk, solid-state drive, flash drive, or optical disk, and is used to store computer program code and necessary data files. The stored computer program includes: a detection data acquisition module and a target detection module; the target detection module includes: a dataset construction unit, a backbone network feature extraction unit, a neck network feature fusion unit, and a multi-scale fusion and target detection unit.

[0127] Finally, it should be noted that the above specific embodiments are merely representative examples of the present invention. Obviously, the present invention is not limited to the above specific embodiments and is susceptible to numerous variations. Any simple modifications, equivalent variations, and modifications to the above specific embodiments based on the technical essence of the present invention shall be deemed to fall within the scope of protection of the present invention.

Claims

1. A target detection method based on infrared and visible light image fusion, characterized in that: include: Acquire visible light and infrared images to be detected; Input the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model to output the target detection results; wherein the training of the YOLOv8 two-stream network model includes: Obtain visible light and infrared image data of a specified area at different time periods to construct a target joint dataset; Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction from the input visible light and infrared images; an aggregation bottleneck module for aggregating the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of the visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results. Among them, the aggregated visible light feature map and infrared feature map are fused, including: using two feature extractors to extract features from the visible light and infrared feature maps respectively; decomposing the extracted visible light and infrared features; using a self-attention bimodal feature fusion module to fuse the decomposed visible light and infrared features to generate a fused feature map, that is, using the self-attention mechanism to learn the binary relationship between the visible light and infrared modalities, and using the correlation matrix to weight each position of the input feature map; using a feature selection module based on the attention mechanism to refine each feature in the fused feature map through the attention mechanism, that is, filtering low-semantic features by adaptively assigning weights within the modality, and combining consistent information between modalities to enrich the semantic representation within the modality; using a bidirectional high-frequency transmission module based on discrete cosine transform to enhance the features of the fused feature map after feature refinement, that is, extracting and exchanging the high-frequency components of visible light and infrared, and converting the spatial domain of the image into the spectral domain; After obtaining the feature-enhanced fusion feature map, the following steps are performed: restoring the fusion feature map to its original size by upsampling; dividing the feature map restored to its original size into two parts, each part containing visible light and infrared information; adding the divided fusion feature maps and projecting them onto the unfused single-modal feature map to obtain the final fusion feature map; The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

2. The target detection method based on infrared and visible light image fusion according to claim 1, characterized in that: Construct the target joint dataset, including: Select the band of specified resolution; The selected bands are converted into different formats, and visible light and infrared images with different contrasts are obtained by controlling the truncation percentage.

3. A target detection system based on infrared and visible light image fusion, applied to the target detection method based on infrared and visible light image fusion according to claim 1, characterized in that: include: The detection data acquisition module is used to obtain the visible light and infrared images to be detected. The target detection module inputs the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model and outputs the target detection results. The training of the YOLOv8 two-stream network model includes: Obtain visible light and infrared image data of a specified area at different time periods and construct a target joint dataset; Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction from the input visible light and infrared images; an aggregation bottleneck module for aggregating the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of the visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results. The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

4. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the target detection method based on infrared and visible light image fusion according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target detection method based on infrared and visible light image fusion according to any one of claims 1 to 2 are implemented.

Images