Visible light-infrared target detection method based on adaptive frequency domain feature fusion

Abstract

The invention relates to a visible light-infrared target detection method based on adaptive frequency domain feature fusion, and the method comprises the steps: carrying out the labeling processing of a preset visible light image and a preset infrared image, and constructing a visible light-infrared fusion target detection data set; a visible light-infrared fusion target detection network is constructed, the network adopts double-flow backbone structures based on YOLOv8 expansion, a multi-scale channel spatial feature interaction module is embedded between the double-flow backbone structures, and a self-adaptive frequency domain feature fusion module is utilized to perform post-fusion on double-flow features; training the visible light-infrared fusion target detection network by using the data set to obtain a trained visible light-infrared fusion target detection model; and inputting the to-be-detected image pair into the visible light-infrared fusion target detection model to obtain a target detection result. According to the method, the multi-scale feature enhancement and frequency domain adaptive fusion method can be utilized, effective feature information between two kinds of modal data can be fully mined and utilized, and the performance of fusion target detection is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and target detection technology, and specifically to a visible-infrared target detection method with adaptive frequency domain feature fusion. Background Technology

[0002] Visible light images primarily utilize reflected light from objects for imaging, thus providing rich texture and color information under excellent lighting conditions. However, in complex environments, such as at night or on cloudy days with poor lighting, visible light images often perform poorly in target detection. In contrast, infrared images rely mainly on the thermal radiation of objects for imaging and are less sensitive to lighting conditions, thus maintaining high image quality even at night or in low light. However, due to the imaging characteristics of infrared images, such as lower resolution, blurred visual effects, unclear edges, and insufficient texture information, the target detection accuracy of infrared images is usually not high. Combining visible light and infrared image target detection technologies can fully utilize the complementarity of the two image modalities, significantly improving the environmental adaptability of target detection systems that fuse visible light and infrared information. This allows them to meet the perception needs of all-weather, complex scenarios, thereby improving target detection performance.

[0003] Current mainstream target detection methods that fuse visible and infrared images typically assume that the two modalities are well registered. However, in real-world applications, due to differences in imaging mechanisms, camera perspectives, and sensor resolutions, strict registration is often difficult to guarantee. On one hand, the position and proportion of the same target may differ between visible and infrared images, and direct stitching or linear weighted fusion can easily lead to significant alignment errors. On the other hand, visible images emphasize shape and texture details, while infrared images highlight thermal distribution features; insufficient fusion can result in the loss or redundancy of crucial information.

[0004] Therefore, to achieve accurate visible-infrared fusion target detection, adaptive alignment and effective fusion of cross-modal features are needed under conditions of non-strict registration. On the one hand, fine-grained features at different spatial scales should be fully explored to improve the edge representation and discrimination capabilities of targets; on the other hand, complementary and enhanced multi-modal features need to be achieved at the frequency domain level to balance global information and detailed features. The synergistic design of spatial multi-scale feature enhancement and frequency domain adaptive fusion has become an important direction and key issue in current cross-modal target detection research. Summary of the Invention

[0005] In view of the above-mentioned technical problems, this invention proposes a visible-infrared target detection method based on adaptive frequency domain feature fusion. By designing a multi-scale channel spatial module to enhance the spatial feature representation of the target, and innovatively introducing an adaptive frequency domain feature fusion module, the features are converted to the wavelet domain for processing. Low-frequency global information is adaptively fused while high-frequency details are preserved, thereby achieving deep complementarity and efficient fusion of multi-modal features, effectively improving the accuracy and robustness of target detection.

[0006] The technical solution to the technical problem of this invention is: an adaptive frequency domain feature fusion visible light-infrared target detection method, comprising the following steps:

[0007] Step S1: Annotate the preset visible light images and infrared images to construct a visible light-infrared fusion target detection dataset;

[0008] Step S2: Construct a visible light-infrared fusion target detection network. The network adopts a dual-stream backbone structure based on YOLOv8 extension, and embeds a multi-scale channel spatial feature interaction module between the dual-stream backbone structures, and uses an adaptive frequency domain feature fusion module to perform post-fusion of the dual-stream features.

[0009] Step S3: Use the dataset to train the visible light-infrared fusion target detection network to obtain the trained visible light-infrared fusion target detection model;

[0010] Step S4: Input the image to be detected into the visible light-infrared fusion target detection model to obtain the target detection result.

[0011] According to one technical solution of the present invention, step S1 specifically includes:

[0012] Step S11: Construct a target category system and sample labeling specifications based on pre-defined expert experience and knowledge;

[0013] Step S12: Based on the target category system and sample annotation specifications, annotate the target locations and categories in the visible light images and infrared images;

[0014] Step S13: Slice the labeled images to form a visible light-infrared fusion target detection dataset, and divide it into a training set, a validation set, and a test set.

[0015] According to one technical solution of the present invention, in step S13, the labeled image is sliced, the slice size is set to 640*640, the overlap coefficient is set to 20%, and it is divided into training set, validation set and test set according to a ratio of 7:1:2.

[0016] According to one technical solution of the present invention, the dual-stream backbone structure includes a visible light feature detector and an infrared feature detector, both of which are YOLOv8 networks and each includes:

[0017] The backbone network used to extract multi-scale feature maps.

[0018] Feature pyramid network used to fuse high-level semantic information and low-level location information

[0019] And a detection network used to output target classification and localization results.

[0020] According to one technical solution of the present invention, the multi-scale channel spatial feature interaction module is used to receive visible light features and infrared features, and perform the following operations:

[0021] The infrared and visible light features were extracted using deformable convolution and standard convolution, respectively.

[0022] The extracted features are then fused using a channel attention module.

[0023] The fused features are interacted with through a multi-scale spatial attention module to output enhanced visible light and infrared features.

[0024] According to one technical solution of the present invention, the adaptive frequency domain feature fusion module performs the following operations:

[0025] Step S21: Decompose the enhanced visible light features and infrared features into low-frequency components and high-frequency components respectively by wavelet transform;

[0026] Step S22: Adaptively fuse the low-frequency components of the visible light feature and the infrared feature to generate a fused low-frequency sub-band, while retaining the original high-frequency sub-bands of the visible light feature and the infrared feature respectively.

[0027] Step S23: The fused low-frequency sub-band is recombined and integrated with the retained visible light high-frequency sub-band and infrared high-frequency sub-band respectively to obtain the recombined visible light frequency domain features and infrared frequency domain features;

[0028] Step S24: Perform inverse wavelet transform on the recombined visible light frequency domain features and infrared frequency domain features respectively to obtain the fused visible light feature map and infrared feature map.

[0029] According to one technical solution of the present invention, the adaptive fusion of low-frequency components in step S22 specifically includes:

[0030] Based on the low-frequency components of the visible light features and the low-frequency components of the infrared features, a fusion weight α∈[0,1] is generated through a fully connected layer and an activation function;

[0031] Using the fusion weight α, the low-frequency components of the visible light features and the low-frequency components of the infrared features are fused by a weighted sum:

[0032] ;

[0033] in, Sigmoid and ReLU are activation functions, Linear is a fully connected layer, and Concat is a spliced ​​feature after global pooling of visible light low frequency (X''_LL) and infrared low frequency (Y''_LL).

[0034] According to a technical solution of the present invention, after obtaining the fused low-frequency subband... Then, it is further enhanced by convolution:

[0035]

[0036] in, , All kernels are 3×3 convolutional kernels, BN is batch normalization, and ReLU is the activation function.

[0037] According to one technical solution of the present invention, step S3 specifically includes:

[0038] Step S31: Train the visible light-infrared fusion target detection network using the training set;

[0039] Step S32: Use the validation set to monitor network performance and update network parameters using the set loss function and optimization algorithm;

[0040] Step S33: When the network performance converges, stop training to obtain the adaptive frequency domain feature fusion visible light-infrared target detection model.

[0041] According to one technical solution of the present invention, the loss function includes cross-entropy loss function, CIoU loss and distributed focus loss;

[0042] The optimization algorithm is the stochastic gradient descent algorithm.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention provides an adaptive frequency domain feature fusion visible light-infrared target detection method. By introducing a multi-scale channel spatial feature interaction module and combining deformable convolution with multi-scale spatial attention, it can dynamically mine and enhance target features at different scales. In particular, it enhances the ability to express the edge and fine-grained texture of the target and alleviates the problem of detail loss caused by network downsampling.

[0045] This invention designs an adaptive frequency domain feature fusion module that transforms features from the spatial domain to the wavelet domain for processing. In the frequency domain, by adaptively weighting and fusing low-frequency components, the complementarity and enhancement of global semantic information between the two modalities can be effectively achieved; by preserving their respective high-frequency components, the integrity of target detail features (such as edges and corners) is ensured. This "low-frequency fusion, high-frequency preservation" strategy enables deeper and more structured feature fusion.

[0046] The entire method of this invention enables end-to-end training. The adaptive frequency domain fusion weight α is dynamically learned and generated by the network based on the input features, which enables the fusion strategy to be adaptively adjusted according to different scene content, thereby improving the model's adaptability and generalization performance to different environments (such as day and night, sunny and rainy weather).

[0047] This invention can effectively improve the average accuracy (mAP) of visible light-infrared target detection, and exhibits stronger robustness in complex scenes and small target detection tasks, providing an effective technical solution for all-weather intelligent sensing systems. Attached Figure Description

[0048] Figure 1 This diagram illustrates an adaptive frequency domain feature fusion visible-infrared target detection network according to one embodiment of the present invention.

[0049] Figure 2 A schematic diagram illustrating a multi-scale channel spatial feature interaction module according to an embodiment of the present invention;

[0050] Figure 3 This diagram illustrates an adaptive multimodal frequency domain feature fusion module according to one embodiment of the present invention.

[0051] Figure 4 The schematic diagram illustrates a visible-infrared target detection method based on adaptive frequency domain feature fusion according to an embodiment of the present invention. Detailed Implementation

[0052] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0053] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0054] like Figures 1 to 4 As shown, this invention provides an adaptive frequency domain feature fusion method for visible-infrared target detection. Through a multi-scale channel spatial module, it utilizes deformable convolution and multi-scale convolution operations to effectively capture edge information and extract features at different spatial scales, while preserving channel prior information and avoiding ignoring their inherent feature distributions. Furthermore, this invention introduces an adaptive frequency domain feature fusion module to decompose the frequency bands of different modalities, fusing low-frequency components for global information exchange, while retaining the original high-frequency features to enhance the model's sensitivity to target pixels, thus comprehensively improving the visible-infrared fusion target detection capability.

[0055] like Figure 4 As shown, the present invention provides an adaptive frequency domain feature fusion method for visible-infrared target detection, comprising the following steps:

[0056] Step S1: Annotate the preset visible light images and infrared images to construct a visible light-infrared fusion target detection dataset;

[0057] Step S1 aims to construct a high-quality visible-infrared fusion target detection dataset for network training and testing. First, based on practical application scenarios (such as traffic monitoring and security patrols) and expert knowledge, a clear target category system (e.g., pedestrians, vehicles, ships) and detailed sample annotation specifications are defined. Then, according to these specifications, the collected paired visible-infrared and infrared remote sensing images are labeled with targets, typically using horizontal bounding boxes to indicate the target's location and category. To improve data utilization efficiency and adapt to the network input size, the labeled images are sliced, for example, large images are cut into 640-pixel × 640-pixel smaller images, with a certain overlap rate (e.g., 20%) to avoid target loss at the cutting edges. Finally, all processed image pairs are divided into training, validation, and test sets according to a predetermined ratio (e.g., 7:1:2). The training set is used for model parameter learning, the validation set is used to monitor the training process and adjust hyperparameters, and the test set is used for final model performance evaluation. This standardized data processing flow lays a reliable data foundation for subsequent model training and evaluation.

[0058] Step S2: Construct a visible light-infrared fusion target detection network. The network adopts a dual-stream backbone structure based on YOLOv8 extension, and embeds a multi-scale channel spatial feature interaction module between the dual-stream backbone structures, and uses an adaptive frequency domain feature fusion module to perform post-fusion of the dual-stream features.

[0059] Step S2 aims to construct a novel visible-infrared fusion target detection network. For example... Figure 1 As shown, this network is extended based on the YOLOv8 baseline framework.

[0060] The dual-stream backbone network consists of two parallel feature extraction streams, one processing visible light images and the other processing infrared images. Each stream is a standard YOLOv8 backbone network, composed of a series of Conv (convolutional), C2f (cross-stage partial bottleneck module), and SPPF (spatial pyramid pooling fast) modules stacked together to extract multi-level, multi-scale feature maps from the input image. The two streams can have the same structure, share parameters, or have independent parameters.

[0061] Multi-scale channel spatial feature interaction module: This module is embedded in the middle layer of the two-stream backbone network. Its function is to enhance and initially interact with the features of the two modalities before feature fusion. For example... Figure 2 As shown, this module receives intermediate features X and Y from the visible light flow and the infrared flow, and the input optical features are denoted as... Infrared features are denoted as First, feature extraction is performed using deformable convolution (for infrared features) and standard convolution (for visible light features) respectively, yielding X' and Y', represented as: Conv (X), conv2(Y).

[0062] Deformable convolution can adaptively adjust the shape of the receptive field, which is beneficial for capturing irregular geometric deformations of the target.

[0063] Subsequently, the features X' and Y' from the two modalities are concatenated along the channel dimension and fed into an improved channel attention module (CA). This module is divided into two branches, Avg and Max, along the channel dimension and fused and expanded to generate a single-modal channel attention. The fused channel attention is then combined with the previously concatenated features to achieve channel-level feature fusion. Next, the fused channel features are input into a multi-scale spatial attention module (MSSA). MSSA uses multiple depthwise separable convolutional kernels of different scales (e.g., 5x5, 7x7, 11x11, 21x21) in parallel to extract spatial features, which are then fused to generate a multi-scale spatial attention map that emphasizes key spatial regions in the feature map that are relevant to the target. Finally, the features modulated by channel and spatial attention are residually concatenated with the original input features X and Y to output the enhanced features. and , represented as:

[0064] ;

[0065] The process can effectively alleviate the heterogeneity problem between different modalities, enhance the feature representation ability of each modality, and reduce feature redundancy.

[0066] Deformable convolution enhances the model's adaptability to target deformation; multi-scale spatial attention allows for simultaneous attention to local details and broader contextual information, strengthening edge and contour features; and residual connections ensure the backpropagation of original information, preventing gradient vanishing. This provides higher-quality and richer-expressive feature inputs for subsequent frequency domain fusion.

[0067] Adaptive frequency domain feature fusion module: This is key to achieving deep modality fusion, operating after the high-level features output by the backbone network. For example... Figure 3 As shown, its input is the feature enhanced by the multi-scale channel spatial module. and The core idea of ​​this module is to transform features from the spatial domain to the frequency domain (wavelet domain), and then perform more fundamental feature operations in the frequency domain. First, for... and Perform Haar wavelet transform on each feature map to decompose it into one low-frequency subband (LL) and three high-frequency subbands (HL, LH, HH), meaning each mode receives four frequency components: the first mode is ( , , , The second mode is () , , , The low-frequency subband contains the global contour and main energy information of the features, while the high-frequency subband contains the details, edges, and texture information of the features. Subsequently, adaptive fusion is performed on the low-frequency subbands of the two modalities: X′′_LL and Y′′_LL are passed through a fully connected layer and an activation function to generate a fusion weight α ranging from [0,1]. .

[0068] Using this weight α, according to the formula The fused low-frequency subband is generated. The weights α are dynamically learned by the network, determining whether to rely more on visible light or infrared global information based on the input image content. Optionally, a 3×3 convolution enhancement is then applied to Fused_LL to obtain... The most important step is to... The original high-frequency subbands (X′′_HL, X′′_LH, X′′_HH) of X′′ and the original high-frequency subbands (Y′′_HL, Y′′_LH, Y′′_HH) of Y′′ are recombinated to form two new frequency domain feature sets. Next, inverse wavelet transform (IWT) is performed on these two recombined frequency domain feature sets, and the fused low-frequency and high-frequency subbands are merged into the original resolution features X''' and Y'''. Finally, X''' and Y''' are added together to generate the final fused feature. .

[0069] Its functions are: 1) Enhanced information complementarity: Adaptive fusion in the low-frequency part ensures that the fused features possess both the structural contours of visible light and the thermally significant region information of infrared, achieving deep complementarity of global semantics. 2) High detail fidelity: Preserving the original high-frequency components means that the rich texture details of visible light and the clear thermal edges of infrared are fully preserved, avoiding the blurring or loss of high-frequency details during the fusion process, which is crucial for accurate localization. 3) Strong adaptability: The fusion weight α is data-driven, enabling the network to intelligently adjust the fusion strategy according to the current scene (e.g., visible light dominates during the day, infrared dominates at night).

[0070] Neck network and detection head: The final features output by the adaptive frequency domain feature fusion module The data is fed into the neck network of YOLOv8 (usually a Feature Pyramid Network (FPN) + PAN structure) for multi-scale feature fusion and enhancement. Finally, the detection head outputs the final detection result.

[0071] Step S3: Use the dataset to train the visible light-infrared fusion target detection network to obtain the trained visible light-infrared fusion target detection model;

[0072] The network constructed in step S2 is trained using the training and validation sets built in step S1. The training process includes: initializing the backbone network parameters using pre-trained parameters on a large dataset (such as COCO); designing a reasonable loss function, typically including classification loss (such as cross-entropy loss), localization loss (such as CIoU Loss), and distributed focus loss (DFL); employing an optimizer such as stochastic gradient descent (SGD) or Adam, and setting a learning rate decay strategy; and using a validation set to monitor the model's performance during training (such as mAP) to prevent overfitting. Through iterative training, the network parameters are continuously updated until the model performance converges, resulting in an optimized adaptive frequency domain feature fusion visible-infrared target detection model. This training process enables the network to automatically learn the attention parameters in the multi-scale channel spatial module and the adaptive weights α in the frequency domain fusion module.

[0073] Step S4: Input the image to be detected into the visible light-infrared fusion target detection model to obtain the target detection result.

[0074] The trained model is then deployed to real-world applications. For the visible-infrared image pair to be detected, preprocessing is first performed at the training time step (e.g., scaling to network input size, normalization), and then the data is input into the trained model. The model propagates forward, sequentially performing two-stream feature extraction, multi-scale channel spatial interaction, adaptive frequency domain fusion, neck network fusion, and detection head prediction, ultimately outputting the target's category, confidence score, and bounding box coordinates. These results can be used for visualization or as input for downstream tasks (e.g., tracking, counting). Through this process, the method of this invention achieves robust and accurate target detection in complex scenes.

[0075] In some embodiments of the present invention, the multi-scale channel spatial feature interaction module is used to receive visible light features and infrared features, and to perform the following operations:

[0076] The infrared and visible light features were extracted using deformable convolution and standard convolution, respectively.

[0077] The extracted features are then fused using a channel attention module.

[0078] The fused features are interacted with through a multi-scale spatial attention module to output enhanced visible light and infrared features.

[0079] like Figure 2 As shown, the input optical feature X and infrared feature Y first enter the feature extraction path respectively. The infrared feature Y is processed using deformable convolution v2 (DConv2). Deformable convolution learns an additional offset field, allowing the sampling points of the convolution kernel to adaptively shift according to the input content. This characteristic allows it to better adapt to geometric deformations of the target in the image (such as perspective changes and pose changes) and the inherent misalignment between visible light and infrared images, thus extracting a geometrically more robust feature Y′. For the visible light feature X, standard convolution (Conv) is used to extract feature X′.

[0080] Next, X′ and Y′ are concatenated along the channel dimension. The concatenated features are then fed into the multimodal channel attention module (CA). This module performs global average pooling (Avg) and global max pooling (Max) on the concatenated features, and then feeds the two pooling results into the compression-activation mapping (composed of two fully connected layers W1 and W2). W1 performs channel compression and introduces nonlinearity through the ReLU activation function, while W2 performs channel recovery. Finally, the channel weights are generated through the Sigmoid function. This process can evaluate and emphasize feature channels that are important to the current detection task, while suppressing unimportant channels, achieving cross-modal channel-level information filtering and fusion.

[0081] Subsequently, the channel attention weights are multiplied by the concatenated features to obtain the channel-enhanced features. These features are then input into the Multi-Scale Spatial Attention Module (MSSA). MSSA employs a multi-branch structure, with each branch using depthwise separable convolutions with different kernel sizes (e.g., 5×5, 7×7, 11×11, 21×21) to process the input features in parallel. Depthwise separable convolutions reduce computational cost, while large kernel sizes provide a larger receptive field, enabling the module to simultaneously capture spatial information at different scales, from local details to global context. The outputs of each branch are fused and then used to generate a spatial attention weight map via a sigmoid function. This weight map highlights spatial locations in the feature map related to the target (e.g., target regions, edges) while downplaying background regions.

[0082] Finally, the spatial attention weights are multiplied by the channel-enhanced features to obtain the final spatial-channel enhanced features. These features are then residually added to the original input features X and Y respectively to obtain the outputs X'' and Y''. The introduction of residual connections ensures the flow of original feature information, avoids feature degradation that may be caused by the attention mechanism, and stabilizes the training process. In summary, this module, through the collaborative work of deformable convolution, channel attention, and multi-scale spatial attention, significantly improves the model's ability to express target features, especially enhancing the saliency of edges and details, providing a high-quality, high-discriminative feature foundation for subsequent frequency domain fusion.

[0083] In some embodiments of the present invention, such as Figure 3 As shown, the input to the adaptive frequency domain feature fusion module is the enhanced visible light feature X'' and infrared feature Y''. First, two-dimensional discrete Haar wavelet transform (DWT) is performed on X'' and Y'' respectively. Haar wavelet computation is simple and efficient, making it suitable for embedding in deep learning networks. Through DWT, each feature map is decomposed into four sub-bands: low-frequency approximation sub-band (LL), horizontal high-frequency detail sub-band (HL), vertical high-frequency detail sub-band (LH), and diagonal high-frequency detail sub-band (HH). Among them, the LL sub-band is an approximate representation of the original feature map after downsampling, containing the main energy and global structural information; the HL, LH, and HH sub-bands contain detail, edge, and texture information in the horizontal, vertical, and diagonal directions, respectively.

[0084] The next crucial step is adaptive low-frequency fusion. The visible light low-frequency subband X''_LL and the infrared low-frequency subband Y''_LL are each subjected to global average pooling, compressing each subband into a feature vector. These two vectors are then concatenated and fed into a small neural network (typically consisting of two fully connected layers and ReLU and Sigmoid activation functions). This network outputs a scalar fusion weight α∈[0,1]. This α value represents the network's assessment of the relative importance of visible light and infrared low-frequency information in the current input feature pair. For example, during daylight hours, the network may learn a larger α, relying more on visible light information; at night, α may be smaller, relying more on infrared information. Subsequently, according to the formula... Weighted fusion is performed to obtain the fused low-frequency subband Fused_LL. This data-driven adaptive fusion mechanism enables the model to dynamically and flexibly integrate the global semantic information of the two modalities, overcoming the limitations of the fixed-weight fusion strategy.

[0085] To further enhance the expressive power of low-frequency features after fusion, a lightweight convolutional enhancement module can be applied to Fused_LL, such as two 3×3 convolutional layers combined with batch normalization (BN) and ReLU activation function, to obtain the enhanced version. , ,in, , All are convolutions with a kernel size of 3×3, BN is batch normalization, and ReLU is the activation function.

[0086] While fusing low frequencies, the module fully preserves the original high-frequency subbands (X''_HL / LH / HH, Y''_HL / LH / HH) of both X'' and Y''. This is a key design principle: high-frequency details (such as sharp edges, corners, and subtle textures of the target) are crucial for accurate localization, and the high-frequency information of the two modes may have different physical meanings (high frequencies in visible light are mostly textures, while high frequencies in infrared are mostly edges with abrupt temperature changes), and direct mixing could lead to information confusion. Therefore, they are chosen to be preserved.

[0087] Then, feature recombination is performed. The enhanced fused low-frequency subband is then... These, combined with the original high-frequency subbands of visible light and infrared light respectively, form two new sets of frequency domain features: , X''_HL, X''_LH, X''_HH) and ( (Y''_HL, Y''_LH, Y''_HH). This means that both visible and infrared features share the fused, enhanced global information (low frequency) while retaining their own unique detailed information (high frequency).

[0088] Finally, inverse wavelet transform (IWT) is performed on the two sets of recombined frequency domain features to merge the fused low-frequency and high-frequency subbands into the original resolution features X''' and Y'''. To integrate these two complementary feature representations, the module performs a final stitching and fusion operation: the feature maps X''' and Y''' are stitched along the channel dimension to generate the final output of the module—the most fused feature Z'', i.e., Z'' = Concat(X''', Y'''). This operation ensures that the visible light texture details and the infrared thermal saliency features are simultaneously preserved and integrated into a unified feature representation.

[0089] The adaptive frequency domain feature fusion module implements a "harmony in diversity" fusion strategy in the frequency domain—deeply fusion the low-frequency components to achieve semantic complementarity, while independently retaining the high-frequency components to maintain detail fidelity. This ensures that the output fused features contain both robust, complementary global target representations and retain the rich details required for accurate localization, fundamentally improving the accuracy and robustness of cross-modal target detection.

[0090] In some embodiments of the present invention, in step S3, model training requires setting a series of hyperparameters and strategies. During training, the batch size can be set to 8 or 16, depending on the GPU memory. The optimizer uses stochastic gradient descent (SGD), with an initial learning rate set to 0.01, and employs learning rate decay strategies such as cosine annealing or stepwise descent, for example, multiplying the learning rate by 0.1 at 60% and 90% of the total epochs. The total number of training epochs can be set to 50 to 100.

[0091] The loss function is a key guide for training. This method uses the YOLOv8 composite loss function, which mainly consists of three parts: 1) classification loss (L... cls 1) Binary cross-entropy loss (BCE Loss) is used to measure the correctness of the predicted object category within the bounding box. 2) Bounding box regression loss (L... box ): CIoU Loss is used, which considers not only the overlap area (IoU) of the bounding boxes, but also the center point distance and aspect ratio, thus more effectively guiding the accurate regression of the bounding boxes. 3) Distributed Focus Loss (L dfl ): Used to optimize the discrete probability distribution of the bounding box coordinates, making the regression smoother and more accurate. The total loss is the weighted sum of these three factors: ;

[0092] Among them, the weighting coefficient , and It can be adjusted according to the task, for example, set to 0.5, 7.5, 1.5 respectively.

[0093] Network weight initialization affects training stability. The backbone network (the CSPDarknet part of dual-stream YOLOv8) can be initialized using weights pre-trained on large datasets such as COCO, which helps the model converge quickly and achieve better feature extraction capabilities. The parameters of the newly added multi-scale channel spatial module and adaptive frequency domain fusion module in this invention can be randomly initialized using methods such as the Kaiming normal distribution.

[0094] During training, the model performance is periodically evaluated using a validation set, and metrics such as mean average precision (mAP) are monitored. Training can be stopped early to prevent overfitting when validation set performance stops improving or begins to decline. Simultaneously, data augmentation techniques, such as random horizontal flipping, random scaling, color dithering (visible light only), and mosaic enhancement, can be employed to increase the diversity of training data and improve the model's generalization ability.

[0095] Through the carefully designed training process described above, the network can effectively learn the complex mapping relationship from the original image to the target detection result, especially in learning to allocate attention in the multi-scale channel spatial module and dynamically calculate the optimal fusion weight α in the frequency domain fusion module.

[0096] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform an adaptive frequency domain feature fusion visible light-infrared target detection method as described in any of the above technical solutions.

[0097] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0098] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the terminal device. It can also be used to temporarily store data that has been output or will be output.

[0099] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a visible-infrared target detection method based on adaptive frequency domain feature fusion as described in any of the above technical solutions.

[0100] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. They can be implemented using computer-executable program code, thus allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this invention is not limited to any particular hardware and software combination.

[0101] Furthermore, it should be noted that the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0102] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0105] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A visible-infrared target detection method with adaptive frequency domain feature fusion, characterized in that, Includes the following steps: Step S1: Annotate the preset visible light images and infrared images to construct a visible light-infrared fusion target detection dataset; Step S2: Construct a visible light-infrared fusion target detection network. The network adopts a dual-stream backbone structure based on YOLOv8 extension, and embeds a multi-scale channel spatial feature interaction module between the dual-stream backbone structures, and uses an adaptive frequency domain feature fusion module to perform post-fusion of the dual-stream features. Step S3: Use the dataset to train the visible light-infrared fusion target detection network to obtain the trained visible light-infrared fusion target detection model; Step S4: Input the image to be detected into the visible light-infrared fusion target detection model to obtain the target detection result.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Construct a target category system and sample labeling specifications based on pre-defined expert experience and knowledge; Step S12: Based on the target category system and sample annotation specifications, annotate the target locations and categories in the visible light images and infrared images; Step S13: Slice the labeled images to form a visible light-infrared fusion target detection dataset, and divide it into a training set, a validation set, and a test set.

3. The method according to claim 1, characterized in that, In step S13, the labeled image is sliced, with the slice size set to 640*640 and the overlap coefficient set to 20%. The slices are then divided into training, validation, and test sets in a ratio of 7:1:

2.

4. The method according to claim 1, characterized in that, The dual-stream backbone structure includes a visible light feature detector and an infrared feature detector, both of which are YOLOv8 networks and each includes: The backbone network used to extract multi-scale feature maps. Feature pyramid network used to fuse high-level semantic information and low-level location information And a detection network used to output target classification and localization results.

5. The method according to claim 1, characterized in that, The multi-scale channel spatial feature interaction module is used to receive visible light features and infrared features, and perform the following operations: The infrared features and the visible light features are extracted using deformable convolution and standard convolution, respectively. The extracted features are then fused using a channel attention module. The fused features are interacted with through a multi-scale spatial attention module to output enhanced visible light and infrared features.

6. The method according to claim 1, characterized in that, The adaptive frequency domain feature fusion module performs the following operations: Step S21: Decompose the enhanced visible light features and infrared features into low-frequency components and high-frequency components respectively by wavelet transform; Step S22: Adaptively fuse the low-frequency components of the visible light feature and the infrared feature to generate a fused low-frequency sub-band, while retaining the original high-frequency sub-bands of the visible light feature and the infrared feature respectively. Step S23: The fused low-frequency sub-band is recombined and integrated with the retained visible light high-frequency sub-band and infrared high-frequency sub-band respectively to obtain the recombined visible light frequency domain features and infrared frequency domain features; Step S24: Perform inverse wavelet transform on the recombined visible light frequency domain features and infrared frequency domain features respectively to obtain the fused visible light feature map and infrared feature map.

7. The method according to claim 6, characterized in that, In step S22, the low-frequency components are adaptively fused, specifically as follows: Based on the low-frequency components of the visible light features and the low-frequency components of the infrared features, a fusion weight α∈[0,1] is generated through a fully connected layer and an activation function; Using the fusion weight α, the low-frequency components of the visible light features and the low-frequency components of the infrared features are fused by a weighted sum: ； in, Sigmoid and ReLU are activation functions, Linear is a fully connected layer, and Concat is a concatenated feature after global pooling of visible light low frequency (X''_LL) and infrared low frequency (Y''_LL).

8. The method according to claim 6, characterized in that, After obtaining the fused low-frequency subband Then, it is further enhanced by convolution: in, , All kernels are 3×3 convolutional kernels, BN is batch normalization, and ReLU is the activation function.

9. The method according to claim 1, characterized in that, Step S3 specifically includes: Step S31: Train the visible light-infrared fusion target detection network using the training set; Step S32: Use the validation set to monitor network performance and update network parameters using the set loss function and optimization algorithm; Step S33: When the network performance converges, stop training to obtain the adaptive frequency domain feature fusion visible light-infrared target detection model.

10. The method according to claim 9, characterized in that, The loss functions include cross-entropy loss, CIoU loss, and distributed focus loss; The optimization algorithm is the stochastic gradient descent algorithm.

Images