A target detection method, device and equipment for visible light and infrared fusion

Abstract

The application relates to a visible light and infrared fusion target detection method, device and equipment, and belongs to the technical field of image processing. The method comprises the following steps: extracting multi-scale features of a visible light image and an infrared image, and performing non-parameter discrete wavelet transformation to orthogonally separate the features into low-frequency semantic energy and high-frequency detail contour; predicting a spatial offset based on the low-frequency features, and synchronously applying the spatial offset to the low-frequency / high-frequency features of the visible light by using sub-pixel resampling, so that the features are forced to align to the infrared features; performing semantic fusion on the low-frequency features, and guiding the high-frequency features to perform edge fusion by using a spatial gating mechanism based on attention; after the fused features are combined, performing non-parameter discrete inverse wavelet transformation, reconstructing output multispectral fusion features, and inputting the multispectral fusion features into a target prediction subnetwork to perform target prediction. The method significantly improves the target detection precision, maintains low parameter overhead and real-time inference speed, and is highly consistent with the deployment requirements of an unmanned aerial vehicle edge platform.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target detection method, apparatus and equipment that fuses visible light and infrared light. Background Technology

[0002] In recent years, with the rapid development of Unmanned Aerial Vehicles (UAVs) technology, UAV-based visual perception systems have played an increasingly important role in fields such as intelligent security, disaster relief, and traffic monitoring. However, traditional single-visible (Vis) cameras often suffer severe feature loss in harsh environments such as nighttime, fog, or complex lighting (e.g., strong backlighting, heavy shadows), leading to a sharp drop in detection performance. To achieve all-weather, highly robust target perception, multispectral target detection fusing visible light and infrared (IR) images has gradually become a research hotspot in this field. Infrared sensors capture thermal radiation imaging of objects, are unaffected by lighting conditions, and can provide crucial cross-modal complementary information for visible light. Despite the great potential of multispectral fusion, two inherently challenging bottlenecks remain in practical UAV applications.

[0003] First, there's the issue of spatial misalignment. Limited by the drone's payload platform, there's often physical parallax between the visible light and infrared dual-camera lenses. Furthermore, mechanical vibrations and wind loads during drone flight can cause deviations in temporal synchronization and spatial alignment of the dual-modal images. Traditional fusion methods assume strict alignment between the two modalities. When image displacement occurs, this forced fusion leads to severe "ghosting" effects at high-frequency edges, interfering with bounding box regression. To address this, some studies have introduced Deformable Convolution (DCN) to predict spatial offsets using cross-modal feature concatenation, thereby resampling and aligning the feature maps. While DCN performs well in single-modal scenarios, it faces the inherent "modal gap" challenge in cross-modal scenarios: strong background clutter and lighting shadows in visible light images, and thermal noise in infrared images, cause significant drift in the offsets predicted by DCN. The network is prone to incorrectly aligning infrared target edges to shadows or high-frequency background clutter in the visible light mode.

[0004] Secondly, targets from the perspective of drones (such as vehicles and pedestrians) usually have a small pixel ratio. In traditional target detection networks, convolutional downsampling with a stride of 2 will irreversibly lose the structural information of these small targets, further exacerbating the difficulty of multimodal alignment and detection. Summary of the Invention

[0005] Therefore, it is necessary to provide a target detection method, device, and equipment that can significantly improve the target detection accuracy by fusing visible light and infrared light, in response to the above-mentioned technical problems.

[0006] A target detection method that fuses visible light and infrared light, the method comprising: Visible light images and infrared images are input into two independent and parallel backbone networks respectively, multi-scale features are extracted, and frequency domain decoupling is performed by parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. Based on the low-frequency features of visible light and the low-frequency features of infrared, the spatial offset between visible light and infrared is predicted. Subpixel resampling is then used to synchronously apply the spatial offset to the low-frequency features of visible light and the high-frequency features of visible light, thereby forcing the visible light features to align with the infrared features, which serve as the spatial basis. Channel aggregation is performed on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and the aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. After combining the low-frequency and high-frequency fusion features, the multispectral fusion features are reconstructed by parameterless discrete inverse wavelet transform and then input into the target prediction subnetwork to perform the final target prediction.

[0007] In one embodiment, visible light and infrared images are respectively input into two independent and parallel backbone networks to extract multi-scale features. These features are then decoupled in the frequency domain using parameter-free discrete wavelet transforms to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. These features include: Visible light images and infrared images are input into two independent and parallel visible light backbone networks and infrared backbone networks, respectively. The pre-trained weights in the backbone networks are used to extract multi-scale features of the input images and output visible light feature maps and infrared feature maps. The visible light feature map and the infrared feature map are decoupled in the frequency domain without spatial information loss by using parameterless 2D Haar discrete wavelet transform. This results in visible light low-frequency features and infrared low-frequency features that retain the semantic energy of the feature map, as well as visible light high-frequency features and infrared high-frequency features that retain the detailed contours of the feature map.

[0008] In one embodiment, the input feature map is processed by a parameterless 2D Haar discrete wavelet transform. It is decoupled losslessly and efficiently into four spatial resolutions. The frequency domain subbands include low-frequency approximations that preserve the macroscopic topological structure and semantic energy of the feature maps. and high-frequency detail components containing key contour information for bounding box regression ;in, These represent abrupt edges and texture details in the horizontal, vertical, and diagonal directions of the feature map, respectively. and Representing feature maps respectively Height and width.

[0009] In one embodiment, predicting the spatial offset between visible light and infrared based on visible light low-frequency features and infrared low-frequency features includes: infrared low-frequency characteristics Low-frequency characteristics of visible light After concatenation along the channel dimensions, a lightweight offset prediction network is input. Predicts the spatial pixel-level offset between visible light and infrared light in the output. and spatial confidence mask : ; in, Represents the real number field. and These represent the height and width of the low-frequency characteristics, respectively.

[0010] In one embodiment, subpixel resampling is used to simultaneously apply spatial offsets to both low-frequency and high-frequency visible light features, thereby forcing the visible light features to align with the infrared features that serve as the spatial basis, including: Using parameterless subpixel resampling The predicted spatial pixel-level offset Synchronously applied to the low-frequency characteristics of visible light and visible light high-frequency characteristics This forces the visible light features to align with the infrared features that serve as the spatial basis, while multiplying them by a spatial confidence mask during the alignment process. By suppressing the activation of meaningless background regions, aligned low-frequency visible light features are obtained. and aligned visible light high-frequency characteristics , respectively represented as: ; .

[0011] In one embodiment, channel aggregation is performed on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features, including: infrared low-frequency characteristics Aligned visible light low-frequency characteristics Concatenate along the channel dimension, and pass through a... convolutional layers Deep semantic aggregation is performed to obtain low-frequency fusion features. , is represented as: .

[0012] In one embodiment, an attention-based spatial gating mechanism is employed to guide the maximum fusion of infrared high-frequency features and aligned visible light high-frequency features, resulting in high-frequency fused features, including: Global average pooling Extracting aligned visible light high-frequency features The channel-level global context, followed by a convolutional layers Feature dimensionality reduction and excitation are performed to generate a visible light high-frequency confidence mask. , is represented as: ; in, This represents the Sigmoid activation function; By employing an attention-based spatial gating mechanism combined with element-maximum-based operations, high-frequency infrared features are ensured. While preserving the detailed contours, high-frequency information from visible light is adaptively captured to obtain high-frequency fusion features. , is represented as: ; in, This represents an element-wise operation that takes the maximum value at the corresponding spatial and channel positions; when the quality of visible light images is severely degraded... When the value approaches 0, the high-frequency fusion feature will automatically degenerate into the infrared high-frequency feature output, ensuring the performance lower bound in extreme environments.

[0013] In one embodiment, the low-frequency fusion features and high-frequency fusion features are combined, and then subjected to a parameter-free discrete inverse wavelet transform to reconstruct the output multispectral fusion features. These features are then input into the target prediction sub-network to perform the final target prediction, including: Low-frequency fusion features High-frequency fusion features Recombined via parameterless 2D Haar inverse discrete wavelet transform The reconstructed output has multispectral fusion features with full-resolution spatial structure. ; Ultimately, multispectral features will be fused. The target prediction subnetwork is input into the neck and detection head to perform target prediction, and the output is the target's category and bounding box.

[0014] A target detection device that fuses visible light and infrared light, the device comprising: The frequency domain decoupling module is used to input visible light images and infrared images into two independent and parallel backbone networks, extract multi-scale features, and perform frequency domain decoupling through parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. The spatial deformation alignment module is used to predict the spatial offset between visible light and infrared based on the low-frequency features of visible light and the low-frequency features of infrared. It also uses sub-pixel resampling to synchronously apply the spatial offset to the low-frequency features of visible light and the high-frequency features of visible light, thereby forcing the visible light features to align with the infrared features that serve as the spatial basis. The fusion module is used to perform channel aggregation on infrared low-frequency features and aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. The feature reconstruction and target prediction module combines low-frequency and high-frequency fusion features, performs parameterless discrete inverse wavelet transform to reconstruct multispectral fusion features, and inputs them into the target prediction subnetwork to perform the final target prediction.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Visible light images and infrared images are input into two independent and parallel backbone networks respectively, multi-scale features are extracted, and frequency domain decoupling is performed by parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. Based on the low-frequency features of visible light and the low-frequency features of infrared, the spatial offset between visible light and infrared is predicted. Subpixel resampling is then used to synchronously apply the spatial offset to the low-frequency features of visible light and the high-frequency features of visible light, thereby forcing the visible light features to align with the infrared features, which serve as the spatial basis. Channel aggregation is performed on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and the aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. After combining the low-frequency and high-frequency fusion features, the multispectral fusion features are reconstructed by parameterless discrete inverse wavelet transform and then input into the target prediction subnetwork to perform the final target prediction.

[0016] The aforementioned target detection method, apparatus, and device that fuses visible light and infrared features introduces a parameter-free discrete wavelet transform as a frequency domain decoupler with no spatial information loss before multi-scale feature fusion. This achieves zero parameter overhead while orthogonally separating visible light / infrared features into low-frequency semantic energy and high-frequency detail contours, effectively avoiding the loss of small target features from the UAV's perspective. In the alignment stage, the spatial offset is predicted using infrared features as the spatial basis and the more noise-resistant low-frequency components of visible light / infrared are utilized. Sub-pixel resampling is then used to deform and align the visible light features, reducing... It reduces the interference of high-frequency noise and modal gap on feature alignment and eliminates the "ghosting" problem caused by misaligned data during fusion. In the fusion stage, an asymmetric fusion path is designed to address the differences in physical characteristics between low-frequency semantics and high-frequency edges. This includes performing semantic aggregation on low frequencies and guiding high-frequency edges to maximize fusion through an attention-based spatial gating mechanism, achieving adaptive complementarity and reconstruction of advantageous features. Finally, the multispectral fusion feature reconstruction is completed through parameter-free discrete inverse wavelet transform and input into the target prediction sub-network to perform the final target prediction, which can significantly improve the multispectral fusion target detection accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a target detection method that fuses visible light and infrared light in one embodiment. Figure 2 This is a schematic diagram of the overall architecture of a lightweight network (FDAF-Net) based on frequency domain decoupling alignment and fusion in one embodiment; Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In one embodiment, such as Figure 1 and Figure 2 As shown, a target detection method that fuses visible light and infrared light is provided. This method supports seamless integration into most YOLO series networks based on Feature Pyramid Network (FPN) in the form of the FDF-Net framework. The core component of FDF-Net is the Frequency Domain Decoupling Alignment and Fusion (FDAF) module.

[0020] In this embodiment, dual-stream expansion and reconstruction are performed on the lightweight and highly representative YOLOv8s network to specifically demonstrate and verify the significant advantages of the proposed method in terms of accuracy and inference efficiency. Unlike traditional symmetric dual-stream networks, this method adopts an innovative "IR-Anchored" asymmetric architecture. Its core idea is that, from the perspective of an all-weather UAV, the infrared mode is unaffected by ambient lighting (such as day-night cycles, strong backlighting, and dense shadows), and the target's spatial position remains stable and accurate. Therefore, the infrared branch is used as the spatial and semantic basis of the entire network, while the visible light branch is used as an auxiliary information source, allowing its features to be deformed and aligned to the infrared space. This method includes the following steps: Step 1: Input the visible light image and the infrared image into two independent and parallel backbone networks respectively, extract multi-scale features, and decouple them in the frequency domain by parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours.

[0021] Specifically, step 1 includes: First, the visible light image and the infrared image are input into two independent and parallel visible light backbone networks and infrared backbone networks, respectively. Multi-scale features of the input image (i.e., the three key scale nodes of the feature pyramid) are extracted using the pre-trained weights in the backbone networks. Extracted features), output visible light feature map and infrared feature map.

[0022] After extracting multi-scale features, to effectively remove noise interference such as illumination and shadows during the cross-modal alignment and fusion stages, a parameterless 2D Haar discrete wavelet transform is used to perform frequency domain decoupling on the visible light feature map and the infrared feature map without spatial information loss. This yields visible light low-frequency features and infrared low-frequency features that retain the semantic energy of the feature map, as well as visible light high-frequency features and infrared high-frequency features that retain the detailed contours of the feature map. In this embodiment, when using the parameterless 2D Haar discrete wavelet transform for feature map decomposition, to maximize inference efficiency on edge devices, traditional matrix convolution operations are avoided. Instead, spatial slicing with a stride of 2 is used for rapid extraction, and orthogonal decomposition is completed through fixed addition and subtraction operations. After parameterless 2D Haar discrete wavelet transform processing, the input feature map... It is decoupled losslessly and efficiently into four spatial resolutions. The frequency domain subbands include: Low-frequency approximate components It preserves the macroscopic topological structure and semantic energy of the feature map, and filters out high-frequency noise such as sudden changes in illumination, making it the optimal reference frame for calculating relative displacement in space.

[0023] High-frequency detail components : Contains key contour information for bounding box regression, where, These represent abrupt edges and texture details in the horizontal, vertical, and diagonal directions of the feature map, respectively. and Representing feature maps respectively Height and width.

[0024] It should be understood that targets from a UAV's perspective are relatively small. Traditional alignment and fusion methods, if they continue to use pooling or convolution with a stride of 2 when processing these high-dimensional features, will irreversibly lose weak high-frequency edge information. This method, however, introduces a parameter-free 2D Haar Discrete Wavelet Transform (DWT). Haar wavelets possess strict orthogonality and invertibility, enabling the lossless separation of the original signal into low-frequency and high-frequency components while halving the spatial resolution of the feature map, providing a perfect feature carrier for subsequent sub-pixel alignment. Therefore, through discrete wavelet transform, frequency domain stripping of features can be achieved at the cost of zero parameters and extremely low latency. This process not only avoids information loss in the spatial dimension but also preserves "semantic energy" ( ")" and "edge details" The separation of “low-frequency guided alignment” and “high-frequency gated fusion” lays the physical foundation for the subsequent realization of cross-modal noise-resistant alignment.

[0025] Step 2: Based on the low-frequency features of visible light and the low-frequency features of infrared, predict the spatial offset between visible light and infrared. Then, using sub-pixel resampling, apply the spatial offset synchronously to the low-frequency features of visible light and the high-frequency features of visible light, so that the visible light features are forced to align with the infrared features, which serve as the spatial basis.

[0026] Considering that the high-frequency components of visible light images contain a large number of shadow edges and texture clutter generated by complex lighting, while the high-frequency components of infrared images only represent thermal radiation jumps (target contours), if the offset is directly calculated in the full-band space, the offset prediction network is highly susceptible to interference from visible light shadow clutter, misregistering the infrared target edges onto the visible light shadow boundaries, thus causing alignment failure. To overcome this problem, this method utilizes the low-frequency features generated by DWT decomposition and proposes a low-frequency guided spatial deformation alignment mechanism. The core idea of ​​this mechanism is: regardless of the difference in high-frequency details, the low-frequency approximation component of the dual-mode image (infrared low-frequency features) is used for alignment. Low-frequency characteristics of visible light The shadow boundaries and internal textures were filtered out, retaining only the core "semantic energy clusters" of the target. This highly consistent low-frequency energy distribution serves as a robust reference frame against lighting interference and for calculating relative spatial displacement. Specifically, step 2 includes: First, the infrared low-frequency characteristics Low-frequency characteristics of visible light After concatenation along the channel dimensions, a lightweight offset prediction network is input. Predicts the spatial pixel-level offset between visible light and infrared light in the output. (Displacement in the X and Y directions) and spatial confidence mask : ; in, Represents the real number field. and These represent the height and width of the low-frequency characteristics, respectively.

[0027] Secondly, parameterless sub-pixel resampling (hereinafter referred to as sub-pixel warp) is employed. The predicted spatial pixel-level offset Synchronously applied to the low-frequency characteristics of visible light and visible light high-frequency characteristics This forces the visible light features to align with the infrared features that serve as the spatial basis, while multiplying them by a spatial confidence mask during the alignment process. By suppressing the activation of meaningless background regions, aligned low-frequency visible light features are obtained. and aligned visible light high-frequency characteristics , respectively represented as: ; .

[0028] It should be understood that by employing the "low-frequency addressing, full-band alignment" strategy described in step 2, this method not only achieves high-precision spatial registration with low computational overhead, but also fundamentally eliminates the offset drift problem caused by high-frequency noise in visible light.

[0029] Step 3: Perform channel aggregation on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, adopt an attention-based spatial gating mechanism to guide the infrared high-frequency features and the aligned visible light high-frequency features to maximize fusion and obtain high-frequency fused features.

[0030] In this study, considering that the edges of infrared and visible light targets are aligned after high-precision registration in the feature space, traditional direct addition or splicing fusion methods still cause feature contamination between modes. Therefore, this method proposes a frequency-band fusion strategy, designing an asymmetric fusion path to address the differences in physical characteristics between low-frequency semantics and high-frequency edges. Specifically, step 3 includes: Low-frequency semantic fusion: Low-frequency components carry the macroscopic semantic background and energy distribution of the target in an image. This is due to the aligned low-frequency features of visible light. Capable of providing infrared low-frequency characteristics Provides rich contextual clues, splices the two together at the channel dimension, and connects them through a convolutional layers Deep semantic aggregation is performed to obtain low-frequency fusion features. , is represented as: .

[0031] High-frequency edge fusion: High-frequency components contain crucial target boundary information. In nighttime or low-light conditions for drones, infrared high-frequency features... It consistently provides a stable target profile, and the aligned visible light high-frequency characteristics This may include dark light noise. To suppress the interference of invalid features on the effective representation during the fusion process, an attention-based spatial gating mechanism is designed, including: Global average pooling Extracting aligned visible light high-frequency features The channel-level global context, followed by a convolutional layers Feature dimensionality reduction and excitation are performed to generate a visible light high-frequency confidence mask. , is represented as: ; in, This represents the Sigmoid activation function.

[0032] Next, an attention-based spatial gating mechanism is employed, combined with element-maximum-based operations, to ensure the preservation of infrared high-frequency features. While preserving the detailed contours, high-frequency information from visible light is adaptively captured to obtain high-frequency fusion features. , is represented as: ; in, This represents an element-wise operation that takes the maximum value at the corresponding spatial and channel positions; when the quality of visible light images is severely degraded... When the value approaches 0, the high-frequency fusion feature will automatically degenerate into the infrared high-frequency feature output, ensuring the performance lower bound in extreme environments.

[0033] Step 4: After combining the low-frequency fusion features and the high-frequency fusion features, the multispectral fusion features are reconstructed by parameterless discrete inverse wavelet transform and input into the target prediction sub-network to perform the final target prediction.

[0034] Specifically, step 4 includes: fusing low-frequency features High-frequency fusion features Recombined via parameterless 2D Haar inverse discrete wavelet transform The reconstructed output has multispectral fusion features with full-resolution spatial structure. Ultimately, the multispectral features will be fused. The target prediction subnetwork (i.e., the prediction part composed of the neck network and the detection head, as referred to in the field of target detection) performs target prediction and predicts the target category and bounding box.

[0035] In summary, the aforementioned target detection method that fuses visible light and infrared features introduces a parameter-free discrete wavelet transform as a frequency domain decoupler without spatial information loss before multi-scale feature fusion. This achieves zero parameter overhead while orthogonally separating visible light / infrared features into low-frequency semantic energy and high-frequency detail contours, effectively avoiding the loss of small target features from the UAV's perspective. In the alignment stage, the method uses infrared features as the spatial basis and leverages the more noise-resistant low-frequency components of visible light / infrared to predict spatial offsets. Sub-pixel resampling is then used to align the visible light features, reducing distortion. The interference of high-frequency noise and modal gap on feature alignment was eliminated, thus removing the "ghosting" problem caused by misaligned data during fusion. In the fusion stage, an asymmetric fusion path was designed to address the differences in physical characteristics between low-frequency semantics and high-frequency edges. This included performing semantic aggregation on low frequencies and guiding high-frequency edges to maximize fusion through an attention-based spatial gating mechanism, achieving adaptive complementarity and reconstruction of advantageous features. Finally, the multispectral fusion feature reconstruction was completed through parameter-free discrete inverse wavelet transform and input into the target prediction sub-network to perform the final target prediction, which can significantly improve the multispectral fusion target detection accuracy.

[0036] To verify the effectiveness and robustness of the proposed FDF-Net framework, experiments were conducted on the highly challenging UAV multispectral target detection dataset DVTOD. This dataset was collected by a dual-light pod mounted on a UAV in a complex real-world environment and contains 2179 pairs of visible light and infrared images. The core challenges of DVTOD are: (1) the data covers extreme lighting scenarios such as strong backlight during the day, heavy shadows, and extremely dark nighttime; (2) due to the effects of UAV payload parallax and mechanical vibration, the dual-light image pairs exhibit significant spatial misalignment and parallax shift; and (3) some targets (such as pedestrians) have a relatively low pixel count from the aerial perspective.

[0037] To address the practical application needs of UAV edge computing platforms, the experiment comprehensively evaluated the model from two dimensions: detection accuracy and operational efficiency.

[0038] In terms of detection accuracy, the following is adopted: As a core indicator: The average precision across all classes when the Intersection over Union (IoU) threshold is 0.5, used to measure the model’s basic recall and classification capabilities.

[0039] In terms of operational efficiency and lightweight design, the following key indicators are adopted: Parameters (M): Measures the amount of storage space occupied by the model.

[0040] Floating-point operations (FLOPs): Measures the computational complexity of the model.

[0041] All experiments were implemented using the PyTorch deep learning framework, with a single NVIDIA V100-32G GPU as the hardware acceleration platform. In the experiments, FDAF-Net used the lightweight YOLOv8s as its basic two-stream architecture. To accelerate model convergence and fully utilize modal priors, the visible light backbone network was loaded with pre-trained weights from the COCO dataset, while the infrared backbone network was loaded with pre-trained weights from infrared single-modal data; the FDAF module was randomly initialized.

[0042] To prevent overfitting on small multispectral datasets, all weight parameters of the dual-stream backbone network were frozen during the training phase, and gradient updates were performed only on the FDAF module and the detector head. During data preprocessing, the network input resolution was uniformly adjusted to [resolution value missing]. Pixel.

[0043] The model was trained using the AdamW optimizer, with the weight decay factor set to [value missing]. The initial learning rate is set to And a cosine annealing strategy is used to dynamically decay it to its maximum value throughout the entire training cycle. The training batch size of the model was set to 8, and a total of 30 training epochs were conducted. To prevent the model from crashing due to random perturbations in the offset prediction network during the early stages of training, an alignment penalty warm-up strategy was designed: in the first 10 epochs, the weights of the auxiliary alignment loss were linearly increased from 0 to a maximum value of 5.0, thereby ensuring the smoothness and rapid convergence of the entire network training process.

[0044] The quantitative evaluation results of FDAF-Net, unimodal model, and bimodal traditional feature splicing model (Baseline) on the DVTOD dataset are shown in Table 1.

[0045] Table 1 Comparison of DVTOD dataset test results

[0046] Analysis of the test results in Table 1 leads to the following conclusions: 1. The difference in single-mode performance verifies the necessity of using infrared features as a substrate.

[0047] Visible light performance is limited: the mAP@0.50 of single-stream YOLOv8s in the visible light mode is only 35.90%. This confirms that in complex all-weather scenarios of UAVs, visible light images are prone to losing a large number of features, making it difficult to independently support highly robust detection tasks.

[0048] The fundamental advantage of infrared features: a single infrared mode achieves an accuracy of 83.13%, far exceeding that of a single visible light mode. This significant performance difference provides the strongest data support for the asymmetric architecture designed in this method, proving that infrared thermal radiation information is a stable spatial reference in multimodal fusion.

[0049] 2. Limitations of the traditional dual-modal feature splicing model.

[0050] When using the baseline for fusion, the number of model parameters increased from 11.17M to 19.02M, and the computational cost (FLOPs) increased from 28.82G to 66.96G. However, its mAP@0.50 only reached 83.93%, a slight increase of only 0.80% compared to the pure infrared model (83.13%). This phenomenon fully demonstrates that without addressing the "spatial misalignment" problem between the two-stream sensors, simple fusion not only fails to achieve complementary advantages, but the misaligned visible light background noise can also contaminate the features of infrared targets, leading to a serious waste of computational resources.

[0051] 3. Performance and efficiency advantages of FDAF-Net.

[0052] FDAF-Net achieved a best-in-class mAP of 86.85% at 0.50, a significant improvement of 2.92% compared to the baseline. This demonstrates that the FDAF module successfully overcomes spatial misalignment through low-frequency spatial deformation alignment, achieving effective fusion of dual-modal features. Compared to the baseline, FDAF-Net delivers a significant performance improvement while only slightly increasing FLOPs by 1.91G, an increase of less than 3%. This is attributed to the use of parameterless wavelet transform and lightweight sub-pixel alignment operators in the downsampling and fusion stages. Although the number of parameters increases to 28.34M, it demonstrates extremely high cost-effectiveness in inference computation complexity, meeting the needs of edge deployment on drones.

[0053] In summary, the FDF-Net proposed in this application does not rely on excessive computing power. Instead, it achieves a significant performance improvement of 2.92% at the cost of less than 3% increase in computational power (FLOPs) compared to the baseline method, through innovative frequency domain decoupling and low-frequency guided alignment mechanisms. This result fully demonstrates the remarkable effectiveness of the FDF module in efficiently solving the ghosting problem in UAV multispectral detection and achieving cross-modal complementary advantages.

[0054] In summary, the experiments clearly demonstrate that FDFF-Net successfully eliminates ghosting and false detection issues caused by spatial misalignment. This framework significantly improves multispectral detection accuracy while maintaining low parameter overhead and real-time inference speed, making it highly suitable for the deployment requirements of UAV edge platforms.

[0055] In one embodiment, a target detection device that fuses visible light and infrared light is provided, comprising: The frequency domain decoupling module is used to input visible light images and infrared images into two independent and parallel backbone networks, extract multi-scale features, and perform frequency domain decoupling through parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. The spatial deformation alignment module is used to predict the spatial offset between visible light and infrared based on the low-frequency features of visible light and the low-frequency features of infrared. It also uses sub-pixel resampling to synchronously apply the spatial offset to the low-frequency features of visible light and the high-frequency features of visible light, thereby forcing the visible light features to align with the infrared features that serve as the spatial basis. The fusion module is used to perform channel aggregation on infrared low-frequency features and aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. The feature reconstruction and target prediction module combines low-frequency and high-frequency fusion features, performs parameterless discrete inverse wavelet transform to reconstruct multispectral fusion features, and inputs them into the target prediction subnetwork to perform the final target prediction.

[0056] Specific limitations regarding the target detection device that fuses visible light and infrared light can be found in the limitations of the target detection method that fuses visible light and infrared light above, and will not be repeated here. Each module in the aforementioned target detection device that fuses visible light and infrared light can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0057] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a target detection method that fuses visible light and infrared light. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0058] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0059] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: Visible light images and infrared images are input into two independent and parallel backbone networks respectively, multi-scale features are extracted, and frequency domain decoupling is performed by parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. Based on the low-frequency features of visible light and the low-frequency features of infrared, the spatial offset between visible light and infrared is predicted. Subpixel resampling is then used to synchronously apply the spatial offset to the low-frequency features of visible light and the high-frequency features of visible light, thereby forcing the visible light features to align with the infrared features, which serve as the spatial basis. Channel aggregation is performed on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and the aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. After combining the low-frequency and high-frequency fusion features, the multispectral fusion features are reconstructed by parameterless discrete inverse wavelet transform and then input into the target prediction subnetwork to perform the final target prediction.

[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A target detection method that fuses visible light and infrared light, characterized in that, The method includes: Visible light images and infrared images are input into two independent and parallel backbone networks respectively, multi-scale features are extracted, and frequency domain decoupling is performed by parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. Based on the low-frequency features of visible light and the low-frequency features of infrared, the spatial offset between visible light and infrared is predicted, and sub-pixel resampling is used to synchronously apply the spatial offset to the low-frequency features of visible light and the high-frequency features of visible light, so that the visible light features are forced to align with the infrared features as the spatial basis. Channel aggregation is performed on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and the aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. After combining the low-frequency and high-frequency fusion features, the multispectral fusion features are reconstructed by parameterless discrete inverse wavelet transform and then input into the target prediction subnetwork to perform the final target prediction.

2. The target detection method based on visible light and infrared fusion according to claim 1, characterized in that, Visible light and infrared images are input into two independent and parallel backbone networks, respectively. Multi-scale features are extracted and then decoupled in the frequency domain by parameter-free discrete wavelet transform. This yields visible light low-frequency features and infrared low-frequency features that preserve semantic energy, as well as visible light high-frequency features and infrared high-frequency features that preserve detailed contours. These features include: Visible light images and infrared images are input into two independent and parallel visible light backbone networks and infrared backbone networks, respectively. The pre-trained weights in the backbone networks are used to extract multi-scale features of the input images and output visible light feature maps and infrared feature maps. The visible light feature map and the infrared feature map are decoupled in the frequency domain without spatial information loss by using parameterless 2D Haar discrete wavelet transform. This results in visible light low-frequency features and infrared low-frequency features that retain the semantic energy of the feature map, as well as visible light high-frequency features and infrared high-frequency features that retain the detailed contours of the feature map.

3. The target detection method based on visible light and infrared fusion according to claim 2, characterized in that, After the parameterless 2D Haar discrete wavelet transform processing, the input feature map It is decoupled losslessly and efficiently into four spatial resolutions. The frequency domain subbands include low-frequency approximations that preserve the macroscopic topological structure and semantic energy of the feature maps. and high-frequency detail components containing key contour information for bounding box regression ;in, These represent abrupt edges and texture details in the horizontal, vertical, and diagonal directions of the feature map, respectively. and Representing feature maps respectively Height and width.

4. The target detection method based on visible light and infrared fusion according to claim 1, characterized in that, Predicting the spatial offset between visible light and infrared based on visible light low-frequency features and infrared low-frequency features, including: infrared low-frequency characteristics Low-frequency characteristics of visible light After concatenation along the channel dimensions, a lightweight offset prediction network is input. Predicts the spatial pixel-level offset between visible light and infrared light in the output. and spatial confidence mask : ； in, Represents the real number field. and These represent the height and width of the low-frequency characteristics, respectively.

5. The target detection method based on visible light and infrared fusion according to claim 4, characterized in that, By utilizing subpixel resampling, the spatial offset is simultaneously applied to both the low-frequency and high-frequency features of visible light, thereby forcibly aligning the visible light features to the infrared features, which serve as the spatial basis. This includes: Using parameterless subpixel resampling The predicted spatial pixel-level offset Synchronously applied to the low-frequency characteristics of visible light and visible light high-frequency characteristics This forces the visible light features to align with the infrared features that serve as the spatial basis, while multiplying them by a spatial confidence mask during the alignment process. By suppressing the activation of meaningless background regions, aligned low-frequency visible light features are obtained. and aligned visible light high-frequency characteristics , respectively represented as: ； 。 6. The target detection method based on visible light and infrared fusion according to claim 1, characterized in that, Channel aggregation is performed on the infrared low-frequency features and the aligned visible light low-frequency features to obtain low-frequency fused features, including: infrared low-frequency characteristics Aligned visible light low-frequency characteristics Concatenate along the channel dimension, and pass through a... convolutional layers Deep semantic aggregation is performed to obtain low-frequency fusion features. , represented as: 。 7. The target detection method based on visible light and infrared fusion according to claim 1, characterized in that, An attention-based spatial gating mechanism is employed to guide the maximization of fusion between infrared high-frequency features and aligned visible high-frequency features, resulting in high-frequency fused features, including: Global average pooling Extracting aligned visible light high-frequency features The channel-level global context, followed by a convolutional layers Feature dimensionality reduction and excitation are performed to generate a visible light high-frequency confidence mask. , represented as: ； in, This represents the Sigmoid activation function; By employing an attention-based spatial gating mechanism combined with element-maximum-based operations, high-frequency infrared features are ensured. While preserving the detailed contours, high-frequency information from visible light is adaptively captured to obtain high-frequency fusion features. , represented as: ； in, This represents an element-wise operation that takes the maximum value at the corresponding spatial and channel positions; when the quality of visible light images is severely degraded... When the value approaches 0, the high-frequency fusion feature will automatically degenerate into the infrared high-frequency feature output, ensuring the performance lower bound in extreme environments.

8. The target detection method based on visible light and infrared fusion according to claim 1, characterized in that, After combining the low-frequency and high-frequency fusion features, the multispectral fusion features are reconstructed through a parameter-free discrete inverse wavelet transform. These reconstructed features are then input into the target prediction sub-network to perform the final target prediction, including: Low-frequency fusion features High-frequency fusion features Recombined via parameterless 2D Haar inverse discrete wavelet transform The reconstructed output has multispectral fusion features with full-resolution spatial structure. ; Ultimately, multispectral features will be fused. The target prediction subnetwork is input into the neck and detection head to perform target prediction, and the output is the target's category and bounding box.

9. A target detection device that fuses visible light and infrared light, characterized in that, The device includes: The frequency domain decoupling module is used to input visible light images and infrared images into two independent and parallel backbone networks, extract multi-scale features, and perform frequency domain decoupling through parameterless discrete wavelet transform to obtain visible light low-frequency features and infrared low-frequency features that preserve semantic energy, and visible light high-frequency features and infrared high-frequency features that preserve detailed contours. The spatial deformation alignment module is used to predict the spatial offset between visible light and infrared based on the low-frequency features of visible light and the low-frequency features of infrared, and to apply the spatial offset synchronously to the low-frequency features of visible light and the high-frequency features of visible light using sub-pixel resampling, so that the visible light features are forced to align to the infrared features as the spatial basis. The fusion module is used to perform channel aggregation on infrared low-frequency features and aligned visible light low-frequency features to obtain low-frequency fused features. At the same time, an attention-based spatial gating mechanism is used to guide the infrared high-frequency features and aligned visible light high-frequency features to maximize fusion to obtain high-frequency fused features. The feature reconstruction and target prediction module combines low-frequency and high-frequency fusion features, performs parameterless discrete inverse wavelet transform to reconstruct multispectral fusion features, and inputs them into the target prediction subnetwork to perform the final target prediction.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images