A lightweight unmanned aerial vehicle-borne optical infrared image fusion pedestrian detection method based on feature level offset correction

Abstract

The application belongs to the technical field of remote sensing image processing, and particularly relates to a lightweight unmanned aerial optical infrared image fusion pedestrian detection method based on feature level offset correction. First, an RGB image and an infrared image are respectively input into a YOLO backbone network to obtain multi-level feature maps. Then, the features of each level of the two modalities are respectively input into a feature level space correction module for displacement field prediction. Then, a standard normalized grid coordinate is defined, and the displacement field is added to the standard grid to obtain a resampling grid. Then, the resampling grid is used for bilinear interpolation resampling of Frgb to obtain offset corrected RGB features. Finally, the offset corrected RGB features are channel spliced and fused with infrared features, and the obtained fusion features are sent to a detection head for classification and regression. The application can overcome the problems of strong data dependence, large calculation overhead and lack of physical modeling leading to insufficient generalization ability of existing RGBT offset correction methods based on deep learning in the scene of unmanned aerial small target detection.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction. Background Technology

[0002] Pedestrian detection is an important research area in UAV remote sensing, with wide applications in traffic monitoring, search and rescue, and more. UAV-based imagery can capture distant scenes and provide a wide-ranging bird's-eye view, thus offering spatial flexibility for pedestrian detection. Meanwhile, infrared thermal imagers can capture the thermal radiation emitted by objects, facilitating pedestrian detection in low-light conditions and ensuring temporal flexibility under varying lighting conditions. Therefore, UAV-based RGBT fusion detection has attracted widespread attention due to its temporal and spatial flexibility.

[0003] However, the combination of drones and RGBT (Radio-Infrared Photodetector) presents new challenges. Visible light and thermal imaging sensors differ in their imaging characteristics, resulting in visible light images and thermal infrared images not being perfectly aligned. Furthermore, in practical hardware deployments, the two sensors cannot be perfectly aligned. These two factors lead to the same target appearing in slightly different positions in the two images, i.e., a positional shift in the target across multimodal images. Drone-captured images cover a wide area, and due to the extremely small size of the target, even a small positional shift within the entire image is significant for target detection. This shift can lead to severe feature map mismatch and chaotic bounding boxes in the detection results.

[0004] Current state-of-the-art RGBT fusion detection methods rely entirely on deep learning to address the aforementioned issues, adaptively correcting positional offsets through deformable convolutions or deformable Transformers. However, these methods have significant limitations when applied to small target detection scenarios on UAVs. First, deep network alignment modules typically require large amounts of data to learn complex offset patterns, while RGBT small target data from a UAV perspective is relatively scarce, making it difficult to support sufficient model convergence. Second, deformable operations themselves have high computational overhead, contradicting the UAV platform's requirement for real-time lightweight operation. More importantly, methods entirely dependent on data-driven approaches lack modeling of physical principles, leading to a sharp decline in generalization ability when encountering extreme flight attitudes or environmental conditions not covered by training data. Therefore, this invention provides a lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction. Summary of the Invention

[0005] The purpose of this invention is to provide a lightweight UAV-borne optical infrared image fusion pedestrian detection method based on feature-level offset correction, which can overcome the shortcomings of existing technologies. Specifically, existing deep learning-based RGBT offset correction methods suffer from strong data dependence, high computational overhead, and insufficient generalization ability due to lack of physical modeling in UAV small target detection scenarios.

[0006] The specific technical solution adopted by this invention is as follows: A lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction includes the following steps: Step 1: First, input the RGB image and infrared image into the two branches of the YOLO backbone network to obtain multi-level feature maps; then, input the features of each level of the two modes into the feature-level spatial correction module to predict the displacement field. In step 1, specifically, the input to each feature-level spatial correction module is the extracted optical features. and infrared features Then, the displacement field is predicted by optical flow estimation:

[0007] in, This is the optical flow estimation function, used to calculate the pixel-level displacement relationship between two modal features. It is the core optical flow estimation sub-network of the feature-level spatial correction module. And the displacement has been normalized to

[0008] .

[0009] in, These are specific offsets at the pixel level, corresponding to the normalized offsets in the horizontal / vertical directions of the feature map (i,j) position, respectively. The subscripts i / j specify the pixel spatial location. It is a general term for the horizontal / vertical offset of all pixel positions.

[0010] Step 2: Define standard normalized grid coordinates in the feature-level spatial correction module, and add the displacement field to the standard grid to obtain the resampled grid; in step 2: Define standard normalized grid coordinates:

[0011] in: ; Matrix form: .

[0012] Then add the displacement field to the standard grid:

[0013] Obtain the resampling grid: .

[0014] W is the total number of pixels in the horizontal direction of the feature map, and H is the total number of pixels in the vertical direction. Together, they define the spatial resolution of the feature map.

[0015] Step 3, based on the resampling grid pairs obtained in Step 2 Bilinear interpolation resampling is performed to obtain offset-corrected RGB features, which are then channel-separated and fused with infrared features. The resulting fused features are then fed into the detection head for classification and regression.

[0016] In step 3, specifically, for the output position : set up Map back to pixel coordinates:

[0017] in It is the resampling grid position p normalized to The coordinates, and , It is the actual pixel coordinates after inverse normalization, which completes the transformation from normalized abstract coordinates to physical pixel coordinates of the feature map, so that the subsequent selection of neighboring pixels and bilinear interpolation calculations have a verifiable pixel location basis. The coordinates of the four nearest neighbors:

[0018] Bilinear weights:

[0019] in, ; Output offset corrected RGB features ; , is the sub-pixel offset in the horizontal / vertical direction, with a value of 0-1, and is the core calculation basis for interpolation weight; c is the feature channel index, which distinguishes feature information in different dimensions and ensures independent channel-level correction. It is the interpolation weight, These are the neighboring pixel coordinates, which correspond one-to-one with each other, and together they complete the weighted calculation of sub-pixel level feature values; the fused feature is obtained by channel stitching and fusing these coordinates with the infrared features. Its expression is:

[0020] in, This indicates a splicing operation. This is to ensure that the number of channels in the fused feature after splicing is consistent with the number of channels in the input feature. Finally, the fused features are sent to the detection head for classification and regression.

[0021] The technical effects achieved by this invention are as follows: This invention proposes a lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction, enabling efficient UAV-borne RGBT small pedestrian target detection under RGBT target offset. It fundamentally solves the feature fusion inaccuracy problem caused by intermodal geometric misalignment. While maintaining detection accuracy, this method significantly reduces the dependence on the scale of labeled data and computational resources, making it more suitable for real-time, lightweight, and small target detection scenarios faced by UAV platforms.

[0022] Extensive experimental results demonstrate that the method presented in this invention is more suitable for RGBT fusion target detection tasks than other RGBT fusion target detection methods. The method proposed in this invention represents a significant step forward in target detection using UAV remote sensing images, providing an effective technical means for achieving multi-source data fusion target detection. Attached Figure Description

[0023] Figure 1 This is a general framework diagram of a lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction according to the present invention. Figure 2 This is a schematic diagram of the feature-level spatial correction module FSAM of the present invention. Detailed Implementation

[0024] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0025] like Figures 1-2 As shown, a lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction includes the following steps: Step 1: First, input the RGB image and infrared image into two branches of the YOLO backbone network respectively to obtain multi-level feature maps; such as... Figure 2 As shown, the features of each level of the two modes are then input into the feature-level spatial correction module for displacement field prediction. In step 1, specifically, the input to each feature-level spatial correction module is the extracted optical features. and infrared features Then, the displacement field is predicted by optical flow estimation:

[0026] in And the displacement has been normalized to

[0027] .

[0028] Step 2, as follows Figure 2 As shown, in the feature-level spatial correction module, standard normalized grid coordinates are defined, and the displacement field is added to the standard grid to obtain the resampled grid; in step 2: Define standard normalized grid coordinates:

[0029] in: ; Matrix form: .

[0030] Then add the displacement field to the standard grid:

[0031] Obtain the resampling grid: .

[0032] Step 3, as follows Figure 2 As shown, based on the resampling grid pairs obtained in step 2 Bilinear interpolation resampling is performed to obtain offset-corrected RGB features, which are then channel-separated and fused with infrared features. The resulting fused features are then fed into the detection head for classification and regression.

[0033] In step 3, specifically, for the output position : set up Map back to pixel coordinates:

[0034] The coordinates of the four nearest neighbors:

[0035] Bilinear weights:

[0036] in, ; Output offset corrected RGB features The fused feature is obtained by channel splicing and fusion of this feature with infrared features. Its expression is:

[0037] in, This indicates a splicing operation. This is to ensure that the number of channels in the fused feature after splicing is consistent with the number of channels in the input feature. Finally, the fused features are sent to the detection head for classification and regression.

[0038] This method is based on the lightweight, single-stage target detection architecture YOLO. While maintaining its real-time inference advantages, it innovatively introduces a physics-based feature-level offset correction mechanism. This method does not rely on learnable deformation operations; instead, it explicitly and accurately calculates the local positional deviation of the same target between visible light and thermal infrared modes through lightweight feature-level optical flow estimation. This optical flow estimation module fuses multi-level feature information, enabling it to sensitively capture positional offsets caused by viewpoint differences, non-rigid deformation between modes, and imaging conditions. Furthermore, through coordinate mapping and feature alignment, it achieves pixel-level spatial correction of modal features, thus providing the detection head with more accurate and effective fused features. This completely physics-based offset correction method does not rely on large-scale alignment data training, significantly reducing dependence on labeled data while maintaining low computational complexity, making it more suitable for the dual requirements of efficiency and generalization ability of UAV platforms.

[0039] This invention proposes a Feature-level Spatial Alignment Module (FSAM) based on physical priors, using the lightweight single-stage target detection architecture YOLO as a foundation to address spatial misalignment caused by intermodal parallax and geometric deformation from a UAV perspective. This module abandons the reliance on large-scale training data and complex models inherent in traditional learnable deformation operations, instead employing a lightweight and interpretable physics-driven alignment strategy. Specifically, FSAM first introduces a lightweight bi-branch feature pyramid to extract multi-scale features from RGB and thermal infrared modes. Then, at each feature level, a feature-level optical flow estimation sub-network is constructed based on the assumption of motion continuity between adjacent frames and feature similarity constraints. This sub-network uses the optical flow estimate from the previous scale as initial values ​​and employs an iterative refinement strategy to accurately calculate the local displacement field of the same target between the two modes. The resulting optical flow field directly reflects the pixel-level offset caused by differences in sensor physical position, viewpoint changes, and non-rigid deformation of the target. Finally, the displacement field is used to perform spatial resampling and coordinate correction on the RGB feature map to achieve physical alignment at the feature level, thereby ensuring the consistency of the two modes in spatial position before fusion, and thus providing more accurate and effective fusion features for the detection head, improving the positioning accuracy of the bounding box of the detection result.

[0040] Taking the practical application scenario of pedestrian detection using optical-infrared fusion on a drone platform as an example, on the publicly available remote sensing dataset RGBTDronePerson, a comparison of various existing RGBT fusion target detection methods is shown in Table 1. It can be seen that the method proposed in this embodiment has better accuracy.

[0041] Table 1. Detection performance indicators of various existing RGBT fusion target detection methods and the proposed method.

[0042] Extensive experimental results demonstrate that the method presented in this invention is more suitable for RGBT fusion target detection tasks than other RGBT fusion target detection methods. The method proposed in this invention represents a significant step forward in target detection using UAV remote sensing images, providing an effective technical means for achieving multi-source data fusion target detection.

[0043] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction, characterized in that: Includes the following steps: Step 1: First, input the RGB image and infrared image into the two branches of the YOLO backbone network to obtain multi-level feature maps; then, input the features of each level of the two modes into the feature-level spatial correction module to predict the displacement field. Step 2: Define standard normalized grid coordinates in the feature-level spatial correction module, and add the displacement field to the standard grid to obtain the resampled grid; Step 3, based on the resampling grid pairs obtained in Step 2 Bilinear interpolation resampling is performed to obtain offset-corrected RGB features, which are then channel-separated and fused with infrared features. The resulting fused features are then fed into the detection head for classification and regression.

2. The lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction according to claim 1, characterized in that: In step 1, specifically, the input to each feature-level spatial correction module is the extracted optical features. and infrared features Then, the displacement field is predicted by optical flow estimation: in, This is the optical flow estimation function, used to calculate the pixel-level displacement relationship between two modal features. It is the core optical flow estimation sub-network of the feature-level spatial correction module. And the displacement has been normalized to ； in, These are specific offsets at the pixel level, corresponding to the normalized offsets in the horizontal / vertical directions of the feature map (i,j) position, respectively. The subscripts i / j specify the pixel spatial location. It is a general term for the horizontal / vertical offset of all pixel positions.

3. The lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction according to claim 1, characterized in that: In step 2: Define standard normalized grid coordinates: in: ; Matrix form: ； W is the total number of pixels in the horizontal direction of the feature map, and H is the total number of pixels in the vertical direction. Together, they define the spatial resolution of the feature map.

4. The lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction according to claim 1, characterized in that: In step 2, the displacement field is added to the standard grid: Obtain the resampling grid: .

5. A lightweight UAV-borne optical-infrared image fusion pedestrian detection method based on feature-level offset correction according to claim 1, characterized in that: In step 3, specifically, for the output position : set up Map back to pixel coordinates: in It is the resampling grid position p normalized to The coordinates, and , It is the actual pixel coordinates after inverse normalization, which completes the transformation from normalized abstract coordinates to physical pixel coordinates of the feature map, so that the subsequent selection of neighboring pixels and bilinear interpolation calculations have a verifiable pixel location basis. The coordinates of the four nearest neighbors: Bilinear weights: in, ; Output offset corrected RGB features ; , is the sub-pixel offset in the horizontal / vertical direction, with a value of 0-1, and is the core calculation basis for interpolation weight; c is the feature channel index, which distinguishes feature information in different dimensions and ensures independent channel-level correction. It is the interpolation weight, These are the coordinates of neighboring pixels, and the two correspond one-to-one to complete the weighted calculation of sub-pixel level feature values. The fused feature is obtained by channel splicing and fusing the corrected RGB features and infrared features. Its expression is: in, This indicates a splicing operation. This is to ensure that the number of channels in the fused feature after splicing is consistent with the number of channels in the input feature. Finally, the fused features are sent to the detection head for classification and regression.

Images