Target detection method and system based on infrared and visible light image fusion

Abstract

The invention provides a target detection method and system based on infrared and visible light image fusion, and relates to the technical field of target detection, and the method comprises the steps: obtaining to-be-detected visible light and infrared images; inputting the obtained visible light and infrared images to be detected into the trained YOLOv8 double-flow network model, and outputting a target detection result; wherein the training of the YOLOv8 double-flow network model comprises the following steps: acquiring data of visible light and infrared images in different time periods of a specified area, and constructing a target joint data set; a YOLOv8 double-flow network model is built; and training the built YOLOv8 double-flow network model by using the built target joint data set to obtain a trained YOLOv8 double-flow network model. According to the method provided by the invention, the YOLOv8 double-flow network model is established, and the fusion module is used for carrying out feature fusion and target detection on the visible light and infrared images, so that the problem that the CNN is difficult to capture the long-distance dependence and global relationship between different modes is solved, and the accuracy and robustness of target detection are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a target detection method and system based on infrared and visible light image fusion. Background Art

[0002] Remote sensing image target detection has always been a key direction of remote sensing image processing. As the core target of battlefield reconnaissance and the key carrier of maritime trade, ships occupy an important position in remote sensing image interpretation and target detection.

[0003] Traditional ship detection methods basically follow a rough-to-fine processing flow: first, separate the land and sea from the image, then perform preprocessing such as declouding and denoising on the image, extract the candidate areas that may contain ship targets, and finally classify the ship targets in each candidate area. Among them, the extraction of candidate areas is particularly critical. The main methods for extracting candidate areas include four methods based on grayscale information statistics, visual saliency analysis, template matching, and classification learning. However, traditional ship detection algorithms rely too much on manually designed features, resulting in limited algorithm application scope, poor real-time performance, and low efficiency in batch processing of remote sensing images. In addition, traditional ship detection algorithms pay more attention to shallow structures designed by hand, and fail to fully utilize deep feature information such as texture, spectrum, and geometry in the image. Therefore, its detection results are difficult to meet the needs of actual applications.

[0004] Therefore, it is necessary to study a target detection method based on the fusion of infrared and visible light images. Summary of the invention

[0005] The purpose of the present invention is to provide a target detection method based on the fusion of infrared and visible light images to address the problems existing in the prior art. By building a YOLOv8 two-stream network model, feature fusion and target detection are performed on visible light and infrared images. Feature fusion is performed by using a self-attention bimodal feature fusion module to more effectively capture the complementarity and correlation between different modalities, and achieve significant target detection accuracy.

[0006] According to one aspect of the present invention, a target detection method based on infrared and visible light image fusion is provided, comprising:

[0007] Acquire visible light and infrared images to be detected;

[0008] The obtained visible light and infrared images to be detected are input into the trained YOLOv8 two-stream network model to output the target detection result; wherein the training of the YOLOv8 two-stream network model includes:

[0009] Obtain data of visible light and infrared images of the specified area at different time periods and construct a target joint dataset;

[0010] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction of input visible light and infrared images; an aggregation bottleneck module for obtaining visible light feature maps and infrared feature maps based on the extracted features; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0011] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0012] Furthermore, a target joint dataset is constructed, including:

[0013] Select the band of specified resolution;

[0014] The selected bands are converted into different formats, and visible light and infrared images with different contrasts are obtained by controlling the cutoff percentage.

[0015] Furthermore, the aggregated visible light feature map and infrared feature map are fused, including:

[0016] Two feature extractors are used to extract features from the visible light and infrared feature maps respectively;

[0017] Decomposing the extracted visible and infrared features;

[0018] The self-attention bimodal feature fusion module is used to fuse the decomposed visible light and infrared features to generate a fused feature map;

[0019] A feature selection module based on the attention mechanism is used to extract features from each feature in the fusion feature map through the attention mechanism;

[0020] A bidirectional high-frequency transmission module based on discrete cosine transform is used to enhance the features of the fused feature map after feature extraction.

[0021] Furthermore, after obtaining the feature-enhanced fusion feature map, it includes:

[0022] The fused feature map is restored to its original size by upsampling;

[0023] The feature map restored to its original size is divided into two parts, each of which contains visible light and infrared information;

[0024] The segmented fused feature maps are added together and projected onto the single-modal feature maps before fusion to obtain the final fused feature map.

[0025] According to one aspect of the present invention, there is provided a target detection system based on infrared and visible light image fusion, comprising:

[0026] The detection data acquisition module is used to acquire the visible light and infrared images to be detected.

[0027] The target detection module inputs the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model and outputs the target detection result; wherein the training of the YOLOv8 two-stream network model includes:

[0028] Obtain data of visible light and infrared images of the specified area at different time periods and construct a target joint dataset;

[0029] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction of input visible light and infrared images; an aggregation bottleneck module for obtaining visible light feature maps and infrared feature maps based on the extracted features; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0030] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0031] According to one aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the target detection method based on infrared and visible light image fusion when executing the computer program.

[0032] According to one aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the target detection method based on infrared and visible light image fusion are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention proposes a target detection method based on the fusion of infrared and visible light images. By building a YOLOv8 dual-stream network model, the visible light and infrared images are subjected to feature fusion and target detection, and significant experimental results have been achieved.

[0035] 2. The present invention proposes a target detection method based on the fusion of infrared and visible light images. By acquiring data of visible light and infrared images of a specified area at different time periods, a target joint data set is constructed, providing data support for target detection experiments.

[0036] 3. The present invention proposes a target detection method based on the fusion of infrared and visible light images. A fusion module is proposed through the neck network of the YOLOv8 dual-stream network model to extract, decompose and fuse the features of the visible light feature map and the infrared feature map, so as to more effectively capture the complementarity and correlation between different modalities, and solve the problem that CNN is difficult to capture the long-distance dependency and global relationship between different modalities.

[0037] 4. The present invention proposes a target detection method based on infrared and visible light image fusion, which can capture the global information and local details of the target through multi-scale fusion of visible light-infrared fusion features, and help improve the accuracy and robustness of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a target detection method based on infrared and visible light image fusion provided by an embodiment of the present invention;

[0040] Figure 2 A YOLOv8 dual-stream network model training flow chart provided for an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the network structure of the SAF module provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the network structure of the Encoder module provided in an embodiment of the present invention;

[0043] Figure 5 The SAF module provided for the embodiment of the present invention uses a self-attention mechanism to learn the binary relationship between visible light and infrared modalities, and uses a correlation matrix to weight the input feature map;

[0044] Figure 6 A schematic diagram of the network structure of the FSM module provided in an embodiment of the present invention;

[0045] Figure 7A schematic diagram of the network structure of the CBAM hybrid attention module provided in an embodiment of the present invention;

[0046] Figure 8 A schematic diagram of the network structure of the channel attention module output by the CAM provided in an embodiment of the present invention;

[0047] Fig. 9 A schematic diagram of the spatial attention structure of the SAM output provided by an embodiment of the present invention;

[0048] Fig.10 Schematic diagram of mAP@0.5-epoch curves of different algorithms in embodiments of the present invention;

[0049] Fig.11 Schematic diagram of mAP@[.5:.95]-epoch curves of different algorithms in embodiments of the present invention;

[0050] Fig.12 A module diagram of a target detection system based on infrared and visible light image fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that:

[0052] The aggregation bottleneck module refers to a cross-stage partial aggregation bottleneck module with two convolutions, also known as a C2f module;

[0053] Fusion module refers to Fusionblock fusion module;

[0054] The self-attention bimodal feature fusion module refers to the (Self Attention Fusion) SAF module;

[0055] The feature selection module based on the attention mechanism refers to the (Feature Selected Module) FSM module;

[0056] The bidirectional high-frequency transmission module based on discrete cosine transform refers to a (Mutual High-frequency Transfer) MHT module based on (Discrete Cosine Transform) DCT.

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The embodiment of the present invention provides a method for target detection based on infrared and visible light image fusion, such as Figure 1-2 As shown, including:

[0059] Acquire visible light and infrared images to be detected;

[0060] The obtained visible light and infrared images to be detected are input into the trained YOLOv8 two-stream network model to output the target detection result; wherein the training of the YOLOv8 two-stream network model includes:

[0061] Obtain data of visible light and infrared images of the specified area at different time periods and construct a target joint dataset;

[0062] Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction of input visible light and infrared images; an aggregation bottleneck module for obtaining visible light feature maps and infrared feature maps based on the extracted features; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results.

[0063] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0064] Specifically, the target joint dataset is constructed. In an embodiment of the present invention, ENVI software is used to select four bands 2, 3, 4, and 8 (Blue, Green, Red, and NIR) with a resolution of 10 m. After the original data is converted into tiff format, the tiff format is converted into jpg format using the GDAL library of python, and visible light and infrared images with different contrasts are obtained by controlling the truncation percentage.

[0065] The embodiment of the present invention also provides a model evaluation index for a target detection method based on the fusion of infrared and visible light images, wherein TP, FP, TN, and FN are basic classification result indicators, which are used to describe the accuracy of the model prediction results. TP (True Positive) is the number of samples that are actually positive and predicted to be positive; FP (False Positive) indicates the number of samples that are actually negative but are mistakenly predicted to be positive, also known as false positives; TN (True Negative) indicates the number of samples that are actually negative and are predicted to be negative; FN (False Negative) indicates the number of samples that are actually positive but are mistakenly predicted to be negative, also known as false negatives. Important evaluation indicators derived from the above basic indicators include:

[0066] P (Precision): Precision is the proportion of correct predictions among the instances predicted by the model as positive samples, reflecting the accuracy of the model in predicting positive samples. The calculation formula is as follows:

[0067]

[0068] R (Recall): Recall rate, also known as recall rate, is the ratio of positive samples correctly detected by the model to all true positive samples, reflecting the ability of the model to detect all true samples. The calculation formula is as follows:

[0069]

[0070] Accuracy: Accuracy refers to the proportion of samples that are correctly classified among all samples. The calculation formula is as follows:

[0071]

[0072] The recall rate is the horizontal axis and the precision rate is the vertical axis. The curve composed of the recall rate and the precision rate is called the PR curve. AP is the average value of the precision rate calculated at different recall rate levels, and its value is the area under the PR curve. mAP represents the mean AP of all categories in the entire data set. The calculation formula is as follows:

[0073]

[0074] Among them, N represents the number of categories, and mAP has different evaluation indicators. Among them, mAP@0.5 represents the average precision when the IoU threshold is set to 0.5, and mAP@[.5:.95] is a wider range of evaluation indicators, considering various situations from IoU thresholds from 0.5 to 0.95, and calculating the average precision of IoU thresholds from 0.5 to 0.95.

[0075] Specifically, a YOLOv8 two-stream network model is built and trained based on the target joint dataset. The visible light and infrared images are used as the input of the network. Feature extraction is performed through the backbone network of the YOLOv8 two-stream network model, and then the visible light feature map and infrared feature map are output through the C2f module.

[0076] Specifically, a fusion module is proposed based on the neck network of the YOLOv8 two-stream network model to extract, decompose and fuse the features of the visible light feature map and the infrared feature map to obtain the visible light-infrared fusion feature, which includes: receiving input data from different modalities, for each modality, using the corresponding feature extractor to extract the feature map, and pooling the feature maps of the two modalities to a smaller size, which not only reduces the amount of calculation but also retains the key information in the feature map; on the pooled feature map, calculating the attention weights of each position in the feature map of one modality and each position in the feature map of another modality, these weights reflect the correlation between elements of different modalities; according to the calculated self-attention weights, the feature maps of the two modalities are fused to generate a fused feature map containing cross-modal correlation; the fused feature map is restored to its original size by upsampling, and then the feature map is split into two parts, each of which contains information about the correlation between the two modalities; finally, the two fused feature maps after segmentation are projected onto the original single-modal feature map in an additive manner. The correlation between the two modalities is modeled through the self-attention mechanism, which enhances the representation ability of the single-modal feature map. The SAF module can effectively capture and utilize the correlation between different modalities, thereby improving the performance of the model on cross-modal tasks. At the same time, through pooling and upsampling strategies, the SAF module reduces the computational complexity while maintaining performance, making it more suitable for practical application scenarios.

[0077] like Figure 3 As shown, the embodiment of the present invention also provides a specific network structure of the SAF module, wherein and The feature maps representing the visible light and infrared modes are pooled globally, and then the feature maps of the two modes are flattened and concatenated together. The size of the flattened and concatenated feature maps is C×2HW. Represents the feature map for which self-attention is to be calculated.

[0078] Specifically, the SAF module uses the Encoder module in the Transformer to perform self-attention calculations. The Encoder module is as follows: Figure 4 As shown, it is the core component of the self-attention mechanism, which can effectively calculate and update the representation of each token to reflect its relevance to all other tokens, where the model can better understand the global structure of the input data, thereby improving performance on various tasks. In an embodiment of the present invention, the SAF module downsamples the calculated feature map to reduce the computational complexity of the self-attention mechanism, and significantly reduces the number of elements involved in the self-attention calculation by reducing the size of the feature map, thereby reducing the amount of calculation and improving the calculation efficiency.

[0079] Specifically, the specific workflow of the Encoder module is as follows:

[0080] The input feature map is positionally encoded before processing the feature map to ensure that the model can take into account the information of each position in the feature map;

[0081] The two-dimensional feature map input to the Encoder is ;

[0082] Positional encoding The size and input features Figure 1 It is directly fused with the input feature map to provide the model with the location information of each pixel, and the position encoding is learnable during the training process;

[0083] The position encoding will be adjusted as training progresses to better adapt to the task and data. The formula for the overall forward propagation process is as follows:

[0084]

[0085] in, and Represent input and output respectively, and Represents the intermediate feature map of the residual connection, ΦNorm() represents LayerNorm, ΦMLP() represents linear mapping, ΦAtt() represents multi-head self-attention, and the calculation formula of single-head self-attention is as follows:

[0086]

[0087] Among them, Q, K, and V are generated according to the input vector. is the weight parameter that needs to be learned, A represents the attention matrix after Softmax, and O represents the output matrix of single-head self-attention calculation;

[0088]

[0089] Among them, split() will Divide into equal proportions and , Up() is the upsampling operation. and The sampled feature map is the size of the input feature map. The fused feature map is split and then projected to the original input feature map in a residual manner. and F' ir Represented as the visible light and infrared modal feature maps output by the SAF module.

[0090] Specifically, the SAF module uses a self-attention mechanism to learn the binary relationship between visible and infrared modalities and a correlation matrix to weight each position of the input feature map. Figure 5As shown, when calculating the correlation matrix, four matrix blocks can be naturally inferred, two of which are intra-modal correlation matrix blocks (visible light and infrared) and the other two are inter-modal correlation matrix blocks.

[0091] Specifically, a feature selection module (FSM) based on the attention mechanism is used to sort the importance of features within the modality to select rich semantic features to solve the problem that low-information value areas affect the scores of high-information value areas. Figure 6 Schematic diagram of the feature selection module structure based on the attention mechanism.

[0092] Specifically, the feature selection module based on the attention mechanism filters low-semantic features by adaptively assigning weights within the modality, and enriches the semantic representation within the modality by combining consistent information between modalities. The steps are as follows:

[0093] The weight coefficients for each channel are obtained by normalizing the features derived from the modality;

[0094] Multiply each channel with the corresponding weight coefficient to obtain a sparse weight penalty to suppress less important channel expressions. Given the modality and r and i, define the adaptive weight of the intra-modality features as follows:

[0095]

[0096] Among them, BN is batch normalization, S is the Sigmoid function, is the weight parameter of a specific module, is the weight parameter of a given mode i, is the weight parameter for a given mode r.

[0097] In this way, fine intra-modal features can be obtained, and the Inter-modal Attention Module uses the CBAM hybrid attention module to further select features. Among them, CBAM (Convolutional BlockAttention Module) uses the attention module to enhance the network's ability to process information of different scales, shapes, and directions. It can adaptively learn the importance of features and use this information for weighted feature maps, thereby improving the model's expressiveness. Figure 7 As shown in Figure 1, CBAM is mainly composed of two sequential sub-modules: Channel Attention Module (CAM) and Spatial Attention Module (SAM).

[0098] Specifically, Figure 8 As shown in the figure, the feature map input by CAM will undergo global maximum pooling and global average pooling operations for each channel. Global maximum pooling can extract the most significant features in the feature map, while global average pooling can take into account the average effect of all features. These two pooling operations can extract global information at the channel level from different angles. The outputs of the two pooling operations will be sent to the multi-layer perceptron (MLP) respectively. The MLP will help the model understand which channel features are more important. The outputs of the two MLPs will be fused, and the fused features will finally pass through the Sigmoid function to obtain the channel attention weight map. CAM can effectively strengthen the weights of important features and weaken the weights of unimportant features. The calculation formula is as follows:

[0099]

[0100] in, They are represented as global average pooling features and maximum pooling features respectively.

[0101] like Fig. 9 As shown in the figure, the input of SAM is the feature map output by the channel attention module. Through the processing of the channel attention module, different channels are weighted. Then the feature map will be subjected to global maximum pooling and global average pooling operations along the channel dimension respectively. After the pooling operation, two single-channel feature maps will be obtained, representing the results of maximum pooling and average pooling respectively. These two feature maps will be spliced ​​in the channel dimension to form a dual-channel feature map. The spliced ​​dual-channel feature map will pass through a 7×7 convolution layer. The function of the convolution layer is to further fuse and refine the spatial information, and reduce it to a single-channel feature map. At the same time, the 7×7 convolution kernel can provide a relatively large receptive field, which helps to capture spatial context information. The single-channel feature map obtained after the convolution operation will pass through the Sigmoid activation function to generate a spatial attention weight map, which represents the importance of different spatial positions in the original feature map. The spatial attention weight map can be directly multiplied with the original input feature map to achieve the emphasis on important spatial positions and the suppression of unimportant positions. Through this process, SAM can effectively improve the model's attention to key spatial information. Its calculation formula is as follows:

[0102]

[0103] Where F is the input feature map, AvgPool(F) and MaxPool(F) represent the results of global average pooling and global maximum pooling operations on the feature map, respectively. Represented as Sigmoid activation function, is a 7×7 convolutional layer, is the spatial attention weight map.

[0104] Specifically, after using the CBAM module, the inter-modal features are inverted into intra-modal features, and finally visible light and infrared features with consistent representation are obtained and combined into the final features. While interacting with the features, rich semantic intra-modal features are obtained by selecting features, and the consistency of inter-modal features is further enhanced in the fusion process.

[0105] Specifically, the bidirectional high-frequency transmission module based on DCT extracts and exchanges high-frequency components of visible light and infrared, and converts the spatial domain of the image into the spectral domain. The commonly used method is discrete Fourier transform (DFT). In the embodiment of the present invention, even if the input signal in DFT is a real number, the conversion result also contains complex numbers. The corresponding value containing the complex number can be calculated through specific operations, where the signal is decomposed into a cosine function; then DCT is used for operation, and only real values ​​are output when expressed as a spectrum. The discrete signal of the spatial domain input of the two-dimensional image of size N×M can be represented in the frequency domain by DCT. The corresponding two-dimensional DCT is as follows:

[0106]

[0107] Among them, f[i,j] is the pixel value of the two-dimensional input image at (i,j), F[x,y] is the DCT coefficient value at (x,y), is the definition of cosine basis function;

[0108]

[0109] is the definition of the normalization constant, f[i,j] represents a two-dimensional signal transformed from the space domain to the frequency domain through DCT, and can be transformed back to the space domain through the inverse DCT (IDCT).

[0110] In the DCT domain, by utilizing the characteristics that low-frequency components are concentrated in the upper left direction and high-frequency components are concentrated in the lower right direction, only high-frequency components can be extracted. Each feature map of the Mutual High-frequency Transfer (MHT) module is an input received from the visible light stream and the infrared stream. Each feature map received as input will undergo DCT and be converted from the spatial domain to the spectral domain, where the converted spectral domain is located in the upper left corner, and the low-frequency and high-frequency characteristics appear in the lower right direction. The high-frequency components of the input feature map are extracted by removing the upper left corner. Among them, on the one hand, if the removed part is too small, the high-frequency and low-frequency components are not well separated, resulting in overfitting of the low frequency during model learning; on the other hand, if the removed part is too large, most of the edge information (high-frequency components that are very important for target recognition) will be lost, resulting in minimal performance improvement. Therefore, considering the low-frequency overfitting and edge loss, about 4% of the total low-frequency components are removed, and the low-frequency components are relatively removed by converting from the spectral domain back to the spatial domain through IDCT. High-frequency feature maps are obtained, and information is more effectively exchanged between the visible light stream and the infrared stream by using MHT, thereby improving the performance of multispectral target detection.

[0111] Specifically, the embodiment of the present invention uses the deep learning framework PyTorch to conduct experiments, with a learning rate of 0.001, a batch size of 16, an IoU threshold of 0.5, and a number of iterations of 500. The specific experimental configuration environment is shown in Table 1:

[0112] Specifically, the target detection performance of the YOLOv8 algorithm proposed in the embodiment of the present invention on the visible light image in the target joint data set, YOLOv5, YOLOX, YOLOv7 and YOLOv8 are selected as comparison algorithms, and the detection performance results of different algorithms are shown in Table 2:

[0113] The curves of mAP@0.5-epoch and mAP@[.5:.95] of the algorithms in Table 2 are as follows Fig.10 and Fig.11 shown.

[0114] It can be seen that the improved YOLOv8 target detection algorithm has the highest rise height and rises very quickly. The performance of the other YOLO series algorithms decreases with the lower versions. Among them, the YOLOv5 algorithm curve has the lowest height and the fluctuation is more obvious.

[0115] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a target detection system based on infrared and visible light image fusion, which is used to execute a target detection method based on infrared and visible light image fusion in the above method embodiment.

[0116] See also Fig.12 The system comprises:

[0117] The detection data acquisition module is used to acquire the visible light and infrared images to be detected.

[0118] The target detection module inputs the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model and outputs the target detection result; wherein the training of the YOLOv8 two-stream network model includes:

[0119] Obtain data of visible light and infrared images of the specified area at different time periods and construct a target joint dataset;

[0120] Build a YOLOv8 two-stream network model, including: a backbone network, which is used to extract features from the input visible light and infrared images; an aggregation bottleneck module, which is used to aggregate the extracted features to obtain visible light feature maps and infrared feature maps; a fusion module, which is used to fuse the aggregated visible light feature maps and infrared feature maps; a neck network, which is used to obtain visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit, which is used to perform multi-scale fusion of visible light-infrared fusion features; and a detection head, which is used to output target detection results based on the multi-scale fusion results.

[0121] The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

[0122] An embodiment of the present invention provides a target detection system based on infrared and visible light image fusion, and the specific implementation process thereof includes: obtaining visible light and infrared images to be detected, inputting the obtained visible light and infrared images to be detected into a trained YOLOv8 two-stream network model, and outputting target detection results; wherein the training of the YOLOv8 two-stream network model includes: collecting visible light and infrared image data of different time periods in a specified area to construct a target joint data set, preprocessing the collected image data, annotating targets in the preprocessed image data, and dividing the annotated data set into a training set and a verification set; using YOLOv8 to detect the target in the image data; and The LOv8 algorithm is used as the basic framework to build a two-stream network model. The backbone network of the two-stream network model is used to extract features of the input visible light and infrared images, and the C2f module is used to output visible light feature maps and infrared feature maps. The Fusionblock fusion module is used to extract, decompose and fuse the visible light feature maps and infrared feature maps, and the self-attention mechanism is used to capture the correlation between the two modalities to obtain visible light-infrared fusion features. The visible light-infrared fusion features are fused at multi-scale, and the multi-scale fusion features are input into the detection head of the YOLOv8 two-stream network model for target detection to generate target detection results.

[0123] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention further provides an electronic device, including a memory and a processor, the memory being used to store computer-executable instructions, and the processor being used to execute computer-executable instructions, so as to implement a target detection method based on infrared and visible light image fusion as proposed in the aforementioned embodiment.

[0124] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by the processor, it is used to realize feature fusion and target detection of visible light and infrared images, provide richer information for multimodal feature fusion, and improve the accuracy of target detection. The storage medium can be any non-volatile storage device such as a hard disk, a solid-state drive, a flash drive, an optical disk, etc., which is used to store computer program codes and necessary data files. The stored computer program includes: a detection data acquisition module and a target detection module; wherein the target detection module includes: a data set construction unit, a backbone network feature extraction unit, a neck network feature fusion unit, a multi-scale fusion and target detection unit.

[0125] Finally, it should be pointed out that the above specific embodiments are only representative examples of the present invention. Obviously, the present invention is not limited to the above specific embodiments, and there are many variations. Any simple modification, equivalent changes and modifications made to the above specific embodiments based on the technical essence of the present invention should be considered to belong to the protection scope of the present invention.

Claims

1. A target detection method based on infrared and visible light image fusion, characterized in that: include: Acquire visible light and infrared images to be detected; The obtained visible light and infrared images to be detected are input into the trained YOLOv8 two-stream network model to output the target detection result; wherein the training of the YOLOv8 two-stream network model includes: Obtain data of visible light and infrared images of the specified area at different time periods and construct a target joint dataset; Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction of input visible light and infrared images; an aggregation bottleneck module for obtaining visible light feature maps and infrared feature maps based on the extracted features; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results. The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

2. The target detection method based on infrared and visible light image fusion according to claim 1, characterized in that: Construct the target joint dataset, including: Select the band of specified resolution; The selected bands are converted into different formats, and visible light and infrared images with different contrasts are obtained by controlling the cutoff percentage.

3. The target detection method based on infrared and visible light image fusion according to claim 1, characterized in that: The aggregated visible light feature map and infrared feature map are fused, including: Two feature extractors are used to extract features from the visible light and infrared feature maps respectively; Decomposing the extracted visible and infrared features; The self-attention bimodal feature fusion module is used to fuse the decomposed visible light and infrared features to generate a fused feature map; A feature selection module based on the attention mechanism is used to extract features from each feature in the fusion feature map through the attention mechanism; A bidirectional high-frequency transmission module based on discrete cosine transform is used to enhance the features of the fused feature map after feature extraction.

4. The target detection method based on infrared and visible light image fusion according to claim 3, characterized in that: After obtaining the feature-enhanced fusion feature map, it includes: The fused feature map is restored to its original size by upsampling; The feature map restored to its original size is divided into two parts, each of which contains visible light and infrared information; The segmented fused feature maps are added together and projected onto the single-modal feature maps before fusion to obtain the final fused feature map.

5. A target detection system based on infrared and visible light image fusion, characterized in that: include: The detection data acquisition module is used to acquire the visible light and infrared images to be detected. The target detection module inputs the acquired visible light and infrared images to be detected into the trained YOLOv8 two-stream network model and outputs the target detection result; wherein the training of the YOLOv8 two-stream network model includes: Obtain data of visible light and infrared images of the specified area at different time periods and construct a target joint dataset; Build a YOLOv8 two-stream network model, including: a backbone network for feature extraction of input visible light and infrared images; an aggregation bottleneck module for obtaining visible light feature maps and infrared feature maps based on the extracted features; a fusion module for fusing the aggregated visible light feature maps and infrared feature maps; a neck network for obtaining visible light-infrared fusion features based on the fusion results; a multi-scale fusion unit for multi-scale fusion of visible light-infrared fusion features; and a detection head for outputting target detection results based on the multi-scale fusion results. The constructed target joint dataset is used to train the constructed YOLOv8 two-stream network model to obtain a trained YOLOv8 two-stream network model.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the target detection method based on infrared and visible light image fusion according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target detection method based on infrared and visible light image fusion according to any one of claims 1 to 4 are implemented.

Images