Infrared and visible light fusion target detection method and device based on FDEA-Net

Abstract

The application discloses an infrared and visible light fusion target detection method and device based on an FDEA-Net, and the method comprises the following steps: acquiring a video stream of a region to be detected; inputting the video stream into a target detection model based on the FDEA-Net for processing, and outputting position information and a category label of a target; wherein the target detection model based on the FDEA-Net comprises: a backbone network, which is used for obtaining an infrared modality feature map and a visible light modality feature map; a cross-layer frequency domain enhancement module, which is used for outputting enhanced infrared modality feature maps and visible light modality feature maps; a self-calibration attention fusion module, which is used for predicting a spatial offset between modes and a fusion attention weight, calibrating and aligning the modality feature maps according to the spatial offset, adaptively fusing the modality feature maps, and outputting a fusion feature map; a neck network and a detection head, which are used for processing the fusion feature map. The application simultaneously solves the problems of false positives caused by interference information and misalignment between modes.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a target detection method and apparatus based on FDEA-Net that fuses infrared and visible light. Background Technology

[0002] Target detection technology has extremely wide applications in industrial inspection, transportation, and public security. Single visible light imaging detection methods often struggle to guarantee reliability and stability in complex environments such as nighttime, backlighting, low illumination, and obstructed conditions like smoke and light rain. Infrared imaging technology can capture thermal radiation features unconstrained by lighting conditions, but it suffers from missing color information and insufficient fine-grained texture description. Therefore, feature fusion of infrared and visible light modes is widely recognized as an effective way to improve the overall performance of target detection in complex environments.

[0003] In practical deployment, the implementation of cross-modal fusion technology usually faces three key technical bottlenecks: First, modal misalignment. Affected by factors such as differences in imaging mechanisms, equipment parallax, and optical distortion, the same target is prone to pixel-level spatial position shift in dual-modal images. If effective calibration is not performed before fusion, it is very easy to cause target positioning deviation and missed detection. Second, modal interference. When any single mode is affected by noise pollution, image degradation, etc., if a direct stitching or simple weighted fusion method is used, irrelevant interference information is easily introduced into the fusion feature representation, reducing the effectiveness of the features.

[0004] Existing multi-source cross-modal fusion technologies can be broadly categorized as follows: one type relies on traditional geometric registration or extrinsic parameter calibration methods to achieve multi-modal information superposition and fusion at the image or feature level; another type employs early / late fusion strategies to merge multi-modal information before and after backbone network feature extraction; yet another type introduces adaptive alignment and fusion strategies at the feature level through learning-based alignment and fusion mechanisms; simultaneously, multi-scale prediction techniques are also widely used to achieve effective coverage of targets of different sizes. While these methods can achieve certain detection results under specific conditions, they still suffer from problems such as insufficient pre-fusion modal calibration, sensitivity to noise interference, and insufficient coupling between feature alignment and target prediction when facing complex dynamic scenes such as airborne platform movement, rapid changes in illumination, and local occlusion. These issues affect the stability and real-time performance of target detection. Summary of the Invention

[0005] This invention proposes a target detection method and device based on FDEA-Net that integrates infrared and visible light, while solving the problems of false alarms caused by interference information and misalignment between modes.

[0006] On the one hand, to achieve the above objectives, this invention provides a target detection method based on FDEA-Net that fuses infrared and visible light, comprising: Obtain the video stream of the region to be detected; The video stream is input into an FDEA-Net-based target detection model for processing, and the target's location information and category label are output. The target detection model based on FDEA-Net includes: Backbone network: used to extract features from infrared and visible light images in the video stream to obtain infrared modal feature maps and visible light modal feature maps; Cross-layer frequency domain enhancement module: used to perform interference information filtering and target feature enhancement on the infrared mode feature map and the visible light mode feature map of the current layer in the frequency domain, and output the enhanced infrared mode feature map and visible light mode feature map; The self-calibrating attention fusion module is used to predict the spatial offset between modes and the fusion attention weight based on the enhanced infrared modal feature map and visible light modal feature map, calibrate and align the modal feature map according to the spatial offset and perform adaptive fusion, and output the fused feature map. Neck network and detection head: used to process the fused feature map.

[0007] Preferably, acquiring the video stream of the region to be detected includes: Frame sampling is performed on the infrared and visible light dual-modal video streams of the area to be detected at a fixed frame rate. The resolution of the sampled images is uniformly adjusted to match the preset size; The image pixel values ​​are normalized to generate a standardized image sequence, which is the video stream of the region to be detected.

[0008] Preferably, interference information filtering and target feature enhancement are performed on the infrared modal feature map and the visible light modal feature map of the current layer, including: Based on the dimensions of the infrared modal feature map and the visible light modal feature map of the current layer, adjust the dimensions of the cross-modal feature map of the previous layer; The infrared mode feature map and the visible light mode feature map of the current layer are transformed to the frequency domain to obtain the frequency domain infrared feature map and the frequency domain visible light feature map; The cross-modal feature map of the previous layer after resizing is interacted with the frequency domain infrared feature map and the frequency domain visible light feature map respectively to enhance the target feature representation and obtain the enhanced frequency domain infrared feature map and the enhanced frequency domain visible light feature map. The enhanced frequency-domain infrared feature map and the enhanced frequency-domain visible light feature map are converted back to the spatial domain to obtain the spatial domain enhanced feature map.

[0009] Preferably, adjusting the size of the cross-modal feature map of the previous layer includes: By performing convolution operations and bilinear interpolation upsampling, the number of channels, height, and width of the cross-modal feature map of the previous layer are adjusted to match the feature map of the current layer. Figure 1 To.

[0010] Preferably, the cross-modal feature map of the previous layer after resizing is interacted with the frequency-domain infrared feature map and the frequency-domain visible light feature map, respectively, including: By performing a dot product operation, the cross-modal feature map of the previous layer after adjustment is fused with the frequency domain infrared feature map and the frequency domain visible light feature map, respectively.

[0011] Preferably, after obtaining the spatial domain enhancement features, the method further includes: Through a gating mechanism, the spatial domain enhanced features are adaptively fused with the original infrared modal feature map and the visible light modal feature map, and the previous layer cross-modal feature map after adjustment of size is introduced as a residual connection, and finally the enhanced infrared modal feature map and visible light modal feature map are output.

[0012] Preferably, the processing procedure of the self-calibrating attention fusion module includes: The enhanced infrared modal feature map and the visible light modal feature map are spliced ​​together, and the first offset, the second offset, the first attention weight and the second attention weight are predicted by convolution operation; The sampling point coordinates of the enhanced infrared modal feature map are adjusted according to the first offset, and the sampling point coordinates of the enhanced visible light modal feature map are adjusted according to the second offset. Based on the adjusted sampling point coordinates, sampling operations are performed on the enhanced infrared modal feature map and the visible light modal feature map respectively to obtain the calibrated infrared feature map and the calibrated visible light feature map; The calibrated infrared feature map is multiplied by the first attention weight, and the calibrated visible light feature map is multiplied by the second attention weight. The results are then added together to output the fused feature map.

[0013] Preferably, the first offset, the second offset, the first attention weight, and the second attention weight are obtained by splitting the feature tensor output by the convolution operation in the channel dimension.

[0014] Preferably, based on the adjusted sampling point coordinates, sampling operations are performed on the enhanced infrared modal feature map and the visible light modal feature map, respectively, including: ; In the formula, This represents the characteristics of the sampled visible light modes. This represents the characteristics of the sampled infrared modes. The coordinates represent the adjusted visible light characteristics. This represents the coordinates of the adjusted infrared feature. This represents the visible light characteristics output by the cross-layer frequency domain enhancement module. This represents the infrared characteristics output by the cross-layer frequency domain enhancement module.

[0015] On the other hand, to achieve the above objectives, the present invention also provides a target detection device based on FDEA-Net that fuses infrared and visible light, comprising: Sensing input unit: used to acquire the video stream of the region to be detected; Proximal inference unit: used to deploy FDEA-Net-based object detection models; Data processing unit: used to input the video stream into the FDEA-Net-based target detection model for processing, output the target's location information and category label, and encapsulate the output results in a structured manner to obtain encapsulated data; Result storage unit: used to store the encapsulated data; Data link communication unit: used to transmit the encapsulated data to a remote terminal; The target detection model based on FDEA-Net includes: Backbone network: used to extract features from infrared and visible light images in the video stream to obtain infrared modal feature maps and visible light modal feature maps; Cross-layer frequency domain enhancement module: used to perform interference information filtering and target feature enhancement on the infrared mode feature map and the visible light mode feature map of the current layer in the frequency domain, and output the enhanced infrared mode feature map and visible light mode feature map; The self-calibrating attention fusion module is used to predict the spatial offset between modes and the fusion attention weight based on the enhanced infrared modal feature map and visible light modal feature map, calibrate and align the modal feature map according to the spatial offset and perform adaptive fusion, and output the fused feature map. Neck network and detection head: used to process the fused feature map.

[0016] Compared with the prior art, the present invention has the following advantages and technical effects: This invention introduces a cross-layer frequency domain enhancement module and a self-calibrating attention fusion module to address the challenges of interference information and modal misalignment in target detection involving visible light and infrared fusion. The cross-layer frequency domain enhancement module filters interference information in the frequency domain, introducing cross-modal features from the preceding layer to enhance the feature representation capability of a single modality. The self-calibrating attention fusion module predicts the calibration offset for a specific modality and uses sampling operations combined with the offset to calibrate modal features, achieving cross-modal feature alignment. Adaptive cross-modal fusion is achieved through the predicted two attention weights. The method and apparatus of this invention can be widely used in various complex target detection scenarios such as autonomous driving, intelligent security, industrial manufacturing, and aerospace. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of the infrared and visible light fusion target detection method based on FDEA-Net according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall framework of FDEA-Net according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-layer frequency domain enhancement module and the self-calibrating attention fusion module according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the infrared and visible light fusion target detection device based on FDEA-Net according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating data acquisition when the present invention is applied to an unmanned aerial vehicle (UAV) according to an embodiment of the invention; Figure 6 This is a schematic diagram illustrating data acquisition when the present invention is applied to an autonomous vehicle. Figure 7 This is a schematic diagram illustrating data acquisition when the present invention is applied to a robot according to an embodiment of the invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] Example 1 This embodiment proposes a target detection method based on FDEA-Net that fuses infrared and visible light, such as... Figure 1 ,include: Obtain the video stream of the region to be detected; The video stream is input into an FDEA-Net-based target detection model for processing, and the target's location information and category label are output. The target detection model based on FDEA-Net includes: Backbone network: used to extract features from infrared and visible light images in the video stream to obtain infrared modal feature maps and visible light modal feature maps; Cross-layer frequency domain enhancement module: used to perform interference information filtering and target feature enhancement on the infrared mode feature map and the visible light mode feature map of the current layer in the frequency domain, and output the enhanced infrared mode feature map and visible light mode feature map; The self-calibrating attention fusion module is used to predict the spatial offset between modes and the fusion attention weight based on the enhanced infrared modal feature map and visible light modal feature map, calibrate and align the modal feature map according to the spatial offset and perform adaptive fusion, and output the fused feature map. Neck network and detection head: used to process the fused feature map.

[0021] Further, acquiring the video stream of the region to be detected includes: Frame sampling is performed on the infrared and visible light dual-modal video streams of the area to be detected at a fixed frame rate. The resolution of the sampled images is uniformly adjusted to match the preset size; The image pixel values ​​are normalized to generate a standardized image sequence, which is the video stream of the region to be detected.

[0022] Specifically, this embodiment first relies on hardware platforms such as drones, unmanned vehicles, and robots to continuously acquire infrared and visible light dual-modal video streams of the area to be detected. After acquisition, independent frame sampling and image preprocessing operations are performed on the two types of video streams: for the infrared video stream, frame sampling is performed at a fixed frame rate of 30fps, and the resolution of all sampled images is uniformly adjusted to 640×512; for the visible light video stream, frame sampling is performed at a higher frame rate of 60fps, the resolution of its sampled images is first uniformly calibrated to 1080×768, and then the image pixel values ​​are normalized so that all pixel values ​​are mapped to the [0,1] value range.

[0023] Furthermore, such as Figure 3 The process of filtering interference information and enhancing target features on the infrared modal feature map and the visible light modal feature map of the current layer includes: Based on the dimensions of the infrared modal feature map and the visible light modal feature map of the current layer, adjust the dimensions of the cross-modal feature map of the previous layer; The infrared mode feature map and the visible light mode feature map of the current layer are transformed to the frequency domain to obtain the frequency domain infrared feature map and the frequency domain visible light feature map; The cross-modal feature map of the previous layer after resizing is interacted with the frequency domain infrared feature map and the frequency domain visible light feature map respectively to enhance the target feature representation and obtain the enhanced frequency domain infrared feature map and the enhanced frequency domain visible light feature map. The enhanced frequency-domain infrared feature map and the enhanced frequency-domain visible light feature map are converted back to the spatial domain to obtain the spatial domain enhanced feature map.

[0024] Further, adjusting the size of the cross-modal feature map of the previous layer includes: By performing convolution operations and bilinear interpolation upsampling, the number of channels, height, and width of the cross-modal feature map of the previous layer are adjusted to match the feature map of the current layer. Figure 1 To.

[0025] Specifically, convolution operations and bilinear interpolation operations are used to transform the cross-modal features of the previous layer. Adjust to visible light and infrared feature maps , At the same scale, the process can be mathematically represented as: ; In the formula, This indicates a convolution operation with a kernel size of 3. This represents the bilinear interpolation operation. This represents the adjusted multimodal feature map of the (i-1)th layer. In this process, the convolution operation serves to... Adjust to and , Same number of channels C The purpose of bilinear interpolation is to... Adjust to and , same height H With width W .

[0026] Furthermore, in order to filter out interference information, the visible light and infrared modal characteristics are transferred to the frequency domain: ; In the formula, , These represent the infrared feature map and the visible light feature map in the frequency domain, respectively. This represents a 1×1 convolution. DFT This represents the Discrete Fourier Transform.

[0027] Furthermore, the resized cross-modal feature map of the previous layer is interacted with the frequency-domain infrared feature map and the frequency-domain visible light feature map, respectively, including: By performing a dot product operation, the cross-modal feature map of the previous layer after adjustment is fused with the frequency domain infrared feature map and the frequency domain visible light feature map, respectively.

[0028] Specifically, the cross-modal features of the previous layer are used to enhance the single-modal features: ; In the formula, and These represent the enhanced frequency domain infrared feature map and the enhanced frequency domain visible light feature map, respectively.

[0029] The features are then transferred back to the spatial domain: ; in, and These represent the visible light and infrared characteristics that have returned to the spatial domain, respectively. IDFT This represents the inverse discrete Fourier transform.

[0030] Furthermore, after obtaining the spatial domain enhancement features, the process also includes: Through a gating mechanism, the spatial domain enhanced features are adaptively fused with the original infrared modal feature map and the visible light modal feature map, and the previous layer cross-modal feature map after adjustment of size is introduced as a residual connection, and finally the enhanced infrared modal feature map and visible light modal feature map are output.

[0031] Specifically, in order to preserve information in the original features, residual connections are used to introduce features before the frequency domain: ; In the formula, , , They represent the learnable parameters, , These represent the frequency-domain visible light enhancement feature that incorporates the original features and the frequency-domain infrared enhancement feature that incorporates the original features, respectively.

[0032] To ensure the adaptive fusion of frequency domain and spatial domain information, a gating mechanism is used to balance the contributions of frequency domain enhancement information and the original single-mode information. The generation gating function can be mathematically expressed as: ; In the formula, Used to fuse visible light spatial domain features with enhanced frequency domain features. Used to fuse visible light spatial domain features with enhanced frequency domain features. Sigmoid This represents the Sigmoid activation function.

[0033] The fusion of visible light spatial domain features and enhanced frequency domain features can be mathematically expressed as: ; In the formula, , This represents single-mode information that combines the original single-mode features with frequency domain enhancement features.

[0034] Finally, residual connections are used to extract the original information of shallow cross-modal features, which provides spatial priors for subsequent modality fusion: ; In the formula, This represents the visible light characteristics output by the cross-layer frequency domain enhancement module. This represents the infrared characteristics output by the cross-layer frequency domain enhancement module.

[0035] Furthermore, the processing procedure of the self-calibrating attention fusion module includes: The enhanced infrared modal feature map and the visible light modal feature map are spliced ​​together, and the first offset, the second offset, the first attention weight and the second attention weight are predicted by convolution operation; The sampling point coordinates of the enhanced infrared modal feature map are adjusted according to the first offset, and the sampling point coordinates of the enhanced visible light modal feature map are adjusted according to the second offset. Based on the adjusted sampling point coordinates, sampling operations are performed on the enhanced infrared modal feature map and the visible light modal feature map respectively to obtain the calibrated infrared feature map and the calibrated visible light feature map; The calibrated infrared feature map is multiplied by the first attention weight, and the calibrated visible light feature map is multiplied by the second attention weight. The results are then added together to output the fused feature map.

[0036] Specifically, such as Figure 3 A self-calibrating attention fusion module is used to predict the calibration offset for a specific modality, and sampling operations are used to combine the offset with the calibration modal features to achieve cross-modal feature alignment. The process of adaptive fusion across modalities through the predicted two attention weights includes: Concatenate cross-modal features to predict inter-modal offsets and attention weights for adaptive fusion: ; In the formula, This represents the convolution operation used to predict the offset to the attention weights, which consists of a 1×1 convolution followed by a 3×3 convolution; K Represented as the assigned offset and attention weight, K is split into the first offset along the channel dimension. Second offset First attention weight With the second attention weight ; The coordinates of the sampling points used to adjust the characteristics of the visible light modes. The coordinates of the sampling points used to adjust the characteristics of the visible light modes. and Used for adaptive fusion of cross-modal features.

[0037] Furthermore, the process of adjusting the coordinates of the sampling points can be mathematically represented as follows: ; In the formula, The coordinates represent the adjusted visible light characteristics. This represents the coordinates of the adjusted infrared feature. These are the original coordinates of the visible light characteristics. These are the original coordinates of the infrared features. N This is a parameter used to constrain the offset to a reasonable range.

[0038] Further, based on the adjusted sampling point coordinates, sampling operations are performed on the enhanced infrared modal feature map and the visible light modal feature map, respectively, including: ; In the formula, This represents the characteristics of the sampled visible light modes. This represents the characteristics of the sampled infrared modes. The coordinates represent the adjusted visible light characteristics. This represents the coordinates of the adjusted infrared feature. , These represent the enhanced infrared modal feature map and the enhanced visible light modal feature map, respectively.

[0039] Furthermore, the adaptive fusion of cross-modal features using attention weights is expressed mathematically as follows: ; In the formula, This represents the fused cross-modal features.

[0040] The fused features are input into the neck network and the detection head, and the output includes the target category, bounding box coordinates and their confidence scores.

[0041] Example 2 This embodiment provides a target detection device based on FDEA-Net that fuses infrared and visible light, such as... Figure 4 ,include: Sensing input unit: used to acquire the video stream of the region to be detected; Proximal inference unit: used to deploy FDEA-Net-based object detection models; Data processing unit: used to input the video stream into the FDEA-Net-based target detection model for processing, output the target's location information and category label, and encapsulate the output results in a structured manner to obtain encapsulated data; Result storage unit: used to store the encapsulated data; Data link communication unit: used to transmit the encapsulated data to a remote terminal; The target detection model based on FDEA-Net includes: Backbone network: used to extract features from infrared and visible light images in the video stream to obtain infrared modal feature maps and visible light modal feature maps; Cross-layer frequency domain enhancement module: used to perform interference information filtering and target feature enhancement on the infrared mode feature map and the visible light mode feature map of the current layer in the frequency domain, and output the enhanced infrared mode feature map and visible light mode feature map; The self-calibrating attention fusion module is used to predict the spatial offset between modes and the fusion attention weight based on the enhanced infrared modal feature map and visible light modal feature map, calibrate and align the modal feature map according to the spatial offset and perform adaptive fusion, and output the fused feature map. Neck network and detection head: used to process the fused feature map.

[0042] Furthermore, the device also includes: The equipment mounting unit is located at the bottom of the drone platform, the top of the unmanned vehicle platform, the head of the robot, etc. It connects the information acquisition module and the lighting assistance module through a three-axis mechanical stabilization gimbal, aiming to reduce the negative impact on imaging quality during movement.

[0043] The equipment mounting unit is deployed on the bottom of the drone, the top of the unmanned vehicle, and the head of the robot. It uses a three-axis mechanical stabilization gimbal to connect and link with the information acquisition unit and the lighting auxiliary unit. Its core function is to reduce various disturbances and interferences generated during the movement of the equipment, avoid the adverse effects of such factors on the imaging quality from the hardware level, and ensure the stability and clarity of image acquisition.

[0044] The information acquisition unit includes a binocular visible light camera and an infrared thermal imager, enabling multimodal imaging and distance perception of the target scene.

[0045] The lighting auxiliary unit is equipped with a small night light fixture with an IPX4 waterproof rating. It is symmetrically arranged on both sides of the three-axis mechanical stabilization gimbal and can provide high-brightness LED light source supplementary lighting support for image acquisition in complex environments with low light intensity at night.

[0046] Specifically, the power supply, data processing unit, near-side inference unit, result storage and reading unit, and data link communication unit are all placed inside drones, unmanned vehicles, robots, etc.

[0047] The power supply provides power to each functional unit to ensure stable operation.

[0048] The data processing unit is responsible for performing structured encapsulation processing on the detection results output from the front end, generating standardized encapsulated data that integrates timestamps, unique identifiers of corresponding video frames, target spatial location information, and category attribution information. This data will be transmitted synchronously in two channels: one channel is sent to the data link communication unit to support real-time image feedback of the detection results, and the other channel is sent to the result storage and reading unit to complete the historical archiving and preservation of all detection data.

[0049] The proximal inference unit uses the K3588 embedded computing unit, which internally deploys the FDEA-Net model (object detection module) to achieve real-time end-to-end object detection.

[0050] The result storage unit is used to store the algorithm model and model parameters, and to save the detection log, detection results and image data.

[0051] The data link communication unit is used to send identification information to the user terminal in real time.

[0052] like Figure 5 This is a schematic diagram illustrating the use of drones for target detection. Figure 6 This is a schematic diagram illustrating the use of robots for object detection. Figure 7 This is a schematic diagram illustrating the use of unmanned vehicles for target detection.

[0053] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target detection method based on FDEA-Net using infrared and visible light fusion, characterized in that, include: Obtain the video stream of the region to be detected; The video stream is input into an FDEA-Net-based target detection model for processing, and the target's location information and category label are output. The target detection model based on FDEA-Net includes: Backbone network: used to extract features from infrared and visible light images in the video stream to obtain infrared modal feature maps and visible light modal feature maps; Cross-layer frequency domain enhancement module: used to perform interference information filtering and target feature enhancement on the infrared modal feature map and the visible light modal feature map of the current layer in the frequency domain, and output the enhanced infrared modal feature map and visible light modal feature map; The self-calibrating attention fusion module is used to predict the spatial offset between modes and the fusion attention weight based on the enhanced infrared modal feature map and visible light modal feature map, calibrate and align the modal feature map according to the spatial offset and perform adaptive fusion, and output the fused feature map. Neck network and detection head: used to process the fused feature map; The process of filtering interference information and enhancing target features in the infrared modal feature map and the visible light modal feature map of the current layer includes: Based on the dimensions of the infrared modal feature map and the visible light modal feature map of the current layer, adjust the dimensions of the cross-modal feature map of the previous layer; The infrared mode feature map and the visible light mode feature map of the current layer are transformed to the frequency domain to obtain the frequency domain infrared feature map and the frequency domain visible light feature map; The cross-modal feature map of the previous layer after resizing is interacted with the frequency domain infrared feature map and the frequency domain visible light feature map respectively to enhance the target feature representation and obtain the enhanced frequency domain infrared feature map and the enhanced frequency domain visible light feature map. The enhanced frequency-domain infrared feature map and the enhanced frequency-domain visible light feature map are converted back to the spatial domain to obtain the spatial domain enhanced feature map. Adjusting the size of the cross-modal feature map of the previous layer includes: By using convolution operations and bilinear interpolation upsampling operations, the number of channels, height, and width of the cross-modal feature map of the previous layer are adjusted to be consistent with the feature map of the current layer. The resized cross-modal feature map of the previous layer is interacted with the frequency-domain infrared feature map and the frequency-domain visible light feature map, respectively, including: By performing a dot product operation, the cross-modal feature map of the previous layer after adjustment is fused with the frequency domain infrared feature map and the frequency domain visible light feature map, respectively; After obtaining the spatial domain enhancement features, the process further includes: Through a gating mechanism, the spatial domain enhanced features are adaptively fused with the original infrared modal feature map and the visible light modal feature map, and the previous layer cross-modal feature map after adjustment of size is introduced as a residual connection, and finally the enhanced infrared modal feature map and the visible light modal feature map are output. The processing steps of the self-calibrating attention fusion module include: The enhanced infrared modal feature map and the visible light modal feature map are spliced ​​together, and the first offset, the second offset, the first attention weight and the second attention weight are predicted by convolution operation; The sampling point coordinates of the enhanced infrared modal feature map are adjusted according to the first offset, and the sampling point coordinates of the enhanced visible light modal feature map are adjusted according to the second offset. Based on the adjusted sampling point coordinates, sampling operations are performed on the enhanced infrared modal feature map and the visible light modal feature map respectively to obtain the calibrated infrared feature map and the calibrated visible light feature map; The calibrated infrared feature map is multiplied by the first attention weight, and the calibrated visible light feature map is multiplied by the second attention weight. The results are then added together to output the fused feature map.

2. The target detection method based on FDEA-Net with infrared and visible light fusion according to claim 1, characterized in that, Acquiring the video stream of the region to be detected includes: Frame sampling is performed on the infrared and visible light dual-modal video streams of the area to be detected at a fixed frame rate. The resolution of the sampled images is uniformly adjusted to match the preset size; The image pixel values ​​are normalized to generate a standardized image sequence, which is the video stream of the region to be detected.

3. The target detection method based on FDEA-Net with infrared and visible light fusion according to claim 1, characterized in that, The first offset, the second offset, the first attention weight, and the second attention weight are obtained by splitting the feature tensor output by the convolution operation along the channel dimension.

4. The target detection method based on FDEA-Net with infrared and visible light fusion according to claim 1, characterized in that, Based on the adjusted sampling point coordinates, sampling operations are performed on the enhanced infrared modal feature map and the visible light modal feature map, respectively, including: ； In the formula, This represents the characteristics of the sampled visible light modes. This represents the characteristics of the sampled infrared modes. The coordinates represent the adjusted visible light characteristics. This represents the coordinates of the adjusted infrared feature. This represents the visible light characteristics output by the cross-layer frequency domain enhancement module. This represents the infrared characteristics output by the cross-layer frequency domain enhancement module.

5. A target detection device based on FDEA-Net infrared and visible light fusion, used to implement the target detection method based on FDEA-Net infrared and visible light fusion as described in any one of claims 1-4, characterized in that, include: Sensing input unit: used to acquire the video stream of the region to be detected; Proximal inference unit: used to deploy FDEA-Net-based object detection models; Data processing unit: used to input the video stream into the FDEA-Net-based target detection model for processing, output the target's location information and category label, and encapsulate the output results in a structured manner to obtain encapsulated data; Result storage unit: used to store the encapsulated data; Data link communication unit: used to transmit the encapsulated data to a remote terminal; The target detection model based on FDEA-Net includes: Backbone network: used to extract features from infrared and visible light images in the video stream to obtain infrared modal feature maps and visible light modal feature maps; Cross-layer frequency domain enhancement module: used to perform interference information filtering and target feature enhancement on the infrared modal feature map and the visible light modal feature map of the current layer in the frequency domain, and output the enhanced infrared modal feature map and visible light modal feature map; The self-calibrating attention fusion module is used to predict the spatial offset between modes and the fusion attention weight based on the enhanced infrared modal feature map and visible light modal feature map, calibrate and align the modal feature map according to the spatial offset and perform adaptive fusion, and output the fused feature map. Neck network and detection head: used to process the fused feature map.

Images