Visible light and infrared fusion target detection method and system for all-day unmanned aerial vehicle scene

Abstract

The invention discloses a visible light and infrared fusion target detection method and system for an all-time unmanned aerial vehicle scene, and belongs to the technical field of unmanned aerial vehicle aviation and intelligent image processing. The method comprises the following steps: analyzing an original visible light image to calculate average brightness and root-mean-square contrast, and quantifying scene illumination conditions; distortion correction and registration are carried out on the original image; a parallel backbone network is adopted to extract visible light and infrared twinborn feature maps, the number of channels allocated to two modal features is dynamically adjusted through 1 * 1 convolution according to illumination condition parameters, and then a dual-light fusion network containing a gated convolution block is utilized to perform weighted fusion; and target prediction is carried out through the progressive feature pyramid network. Through an illumination adaptive fusion strategy and targeted data enhancement, the precision and robustness of target detection in a complex all-day scene are significantly improved, and the method is suitable for real-time target detection tasks of the unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of unmanned aerial vehicle aviation and intelligent image processing technology, and particularly relates to a visible-infrared fusion target detection method and system for all-weather unmanned airfield scenes. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, its application in target reconnaissance, intelligence collection, security monitoring and personnel search and rescue fields is increasingly widespread. These application scenarios often require unmanned aerial vehicles to have all-weather working capability. However, relying solely on a single visible light camera in night or poor light conditions is difficult to work effectively, because visible light images mainly capture environmental reflected light, which will lose a lot of information in low light or no light.

[0003] In order to overcome this limitation, visible light and infrared dual light fusion technology has emerged. Visible light images can provide rich scene color and texture details, while infrared images can stably work at night by capturing the thermal radiation of objects themselves, regardless of light conditions. Therefore, the dual imaging system of unmanned aerial vehicles combining the two can obtain more comprehensive and rich target and scene information, significantly improving the adaptability of the target detection system under various light conditions.

[0004] In recent years, target detection methods based on deep learning have made significant progress. In the field of visible light and infrared fusion detection, existing methods are mainly divided into two-stage methods of "fusion first and then detection" and direct fusion detection methods of "end-to-end". The two-stage method has poor real-time performance and lacks task-oriented fusion process because the fusion and detection tasks are separated. Therefore, the end-to-end fusion detection method is more popular due to its efficiency.

[0005] However, the existing end-to-end fusion detection method generally has a fatal assumption: the input visible light and infrared images are strictly pixel-aligned and of high quality. However, in actual unmanned aerial vehicle flight tasks, due to camera vibration, attitude changes, and inherent field-of-view differences between visible light and infrared sensors, the collected image pairs often have weak alignment, blurred jitter, and local data loss. These real-world image quality degradation problems can seriously weaken the performance of existing fusion detection models, resulting in a significant reduction in detection accuracy and robustness. Therefore, how to design an all-weather target detection method that can effectively deal with weak alignment and image noise, and can adaptively adjust the fusion strategy according to the scene light conditions, is a technical problem that needs to be solved. SUMMARY

[0006] The present application aims at the deficiencies of the prior art, and provides a visible light-infrared fusion target detection method and system for all-weather unmanned aerial scenes, aiming to solve the problem of reduced detection performance caused by image weak alignment and jitter blur in the prior art.

[0007] The technical scheme is as follows: S1: acquiring and analyzing visible light original images and infrared original images collected by a camera of an unmanned aerial vehicle to determine one or more parameters representing illumination conditions of a scene; S2: performing distortion correction and registration on the visible light original images and the infrared original images to generate weakly aligned visible light images and infrared images within a preset offset range; S3: using two parallel full convolution backbone networks to extract multi-layer twin feature maps from the weakly aligned visible light images and infrared images respectively, and fusing the visible light feature maps and the infrared feature maps according to the scene illumination condition parameters to generate fusion feature maps; S4: inputting the fusion feature maps into a target prediction network to generate a detection result of a target.

[0008] Further improvement of the above technical scheme is that the scene illumination condition parameters include average brightness and root mean square contrast calculated based on the visible light original images; and the fusing of the visible light feature maps and the infrared feature maps is based on a comparison result of the average brightness and the root mean square contrast with respective preset thresholds, and the number of channels dynamically allocated to the visible light feature maps and the infrared feature maps is adjusted.

[0009] Further, the average brightness of the visible light original images is calculated by the following formula: In the formula, represents the average brightness of the visible light original images, and represent the width and height of the visible light original images, and represent the horizontal coordinate and vertical coordinate of a pixel of the visible light original images, represents the three-channel average brightness of the pixel with coordinates in the visible light original images; The root mean square contrast of the visible light original images is calculated by the following formula: In the formula, represents the root mean square contrast of the visible light original images.

[0010] Further, the number of channels of the visible light and infrared feature maps is optimized according to the average brightness and root mean square contrast of the visible light original image, and is expressed as follows: wherein, and respectively represent the number of channels of the visible light and infrared feature maps, represents a fixed feature map channel number parameter and is a multiple of 4, and respectively represent threshold parameters of the average brightness and root mean square contrast.

[0011] Further, the distortion correction is realized by the following formula: wherein, represents the image after distortion correction, represents the original image, represents the camera intrinsic matrix, , and represent the radial distortion coefficients, and represent the tangential distortion coefficients, represents the distortion correction function; In the corrected visible light image, the visible light image is aligned with the infrared image corresponding image area by cropping and scaling.

[0012] Further, the weak alignment refers to that the maximum offset of the corresponding pixel points of the visible light image and the infrared image does not exceed 8 pixels; when training the neural network model for realizing the method, data noise is introduced to the weakly aligned visible light image or infrared image, and the data noise is selected from at least one of pixel offset, dithering blur or single channel removal.

[0013] Further, the number of channels of the visible light and infrared feature maps is optimized according to the average brightness and root mean square contrast of the visible light original image, and is expressed as follows:

[0014] Further, the full convolution backbone network adopts the backbone network of the YOLO detection algorithm, including five stages of convolution blocks P1 to P5, and the multi-layer visible light feature maps and the multi-layer infrared feature maps are respectively output from the P3, P4 and P5 stages; The gated convolutional block has a dual-branch structure, including: a content branch, which consists of a layer normalization layer, a linear layer, and a 3*3 convolutional layer; and a gated branch, which consists of a layer normalization layer, a linear layer, a 3*3 depthwise separable convolutional layer, and an activation layer. The outputs of the content branch and the gated branch are gated and modulated by element-wise multiplication. The modulation result is then added to the original input through residual connection to form the final output.

[0015] Furthermore, the target prediction network is an improved progressive feature pyramid network, which enhances the fused feature map across levels through upsampling, downsampling, and concatenation operations for final classification and regression prediction; and uses a 3*3 convolution with a stride of 2 to achieve downsampling and a transposed convolution to achieve upsampling.

[0016] The system for implementing the visible light and infrared fusion target detection method in the above-mentioned all-weather UAV scenarios includes: The scene lighting estimation module is used to analyze the raw visible light image and estimate the scene lighting conditions based on the analysis results. The original image distortion correction and registration module is used to perform distortion correction and registration on the original visible light and infrared images, generating weakly aligned image pairs; The image twin feature extraction and adaptive fusion network module is used to extract multi-layer twin feature maps from the weakly aligned image pairs respectively, and to adaptively fuse the twin feature maps according to the estimation results of the scene illumination estimation module to generate a dual-light fusion feature map. The image fusion feature complementarity and target prediction network module is used to process the dual-light fusion feature map and predict the final target information; A training image data augmentation and noise addition module is used to apply one or more types of data noise to the weakly aligned image pairs.

[0017] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows: This invention introduces a scene illumination analysis module, enabling dynamic adaptive adjustment of the fusion weights for visible light and infrared information. This module quantitatively analyzes the average brightness and contrast of the visible light image, allowing the system to intelligently determine real-time illumination conditions: in sufficient light, it automatically prioritizes preserving the rich texture details of the visible light image; in dim light or at night, it relies on the stable thermal radiation information of the infrared image. This intelligent fusion mechanism based on illumination perception completely overcomes the limitations of traditional fixed-weight fusion, resulting in a 3.5% improvement in daytime detection accuracy and a 2.7% improvement at night in real-world drone scenarios.

[0018] This invention innovatively introduces a data augmentation strategy that simulates real noise during the training phase. By actively injecting noise such as pixel offset and jitter blur into the training data, it forces the neural network to learn robust feature extraction and alignment capabilities under non-ideal conditions, significantly enhancing the model's generalization and environmental adaptability. This makes it less prone to misjudgment when facing complex images in the real world, effectively reducing the false alarm rate by 6.8%, and solving the problem of weak image alignment and quality degradation caused by factors such as vibration and attitude changes during actual flight of UAVs.

[0019] This invention features deep optimization at the network architecture level, employing a series of lightweight designs, including a gated fusion network centered on depthwise separable convolutions and an improved progressive feature pyramid network. This significantly reduces the computational complexity of the model without sacrificing high accuracy. This efficient network structure design fundamentally guarantees real-time inference speeds exceeding 30 FPS for 640x640 resolution images, meeting the stringent real-time requirements of UAV platforms and enabling commercial deployment on UAV platforms with limited computing resources.

[0020] This invention organically combines intelligent perception, robust training, and efficient network design, systematically solving the three core challenges of target detection in all-weather UAV scenarios: accuracy, reliability, and real-time performance. It has significant practical value and technological advancements. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of the visible light and infrared fusion target detection method for all-weather UAV scenarios according to the present invention.

[0022] Figure 2 This is a schematic diagram of the dual-light fusion network structure in the image twin feature extraction and adaptive fusion network module of this invention. Detailed Implementation

[0023] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0024] Example 1: As Figure 1 As shown, the all-weather visible light and infrared fusion target detection system for UAV scenarios provided in this aspect specifically includes: The scene illumination estimation module is used to calculate the average brightness and root mean square contrast of the original visible light image, and then optimize the number of feature map channels for visible light and infrared based on the thresholds of average brightness and root mean square contrast. The original image distortion correction and registration module is used to correct the original visible light and infrared images captured by the UAV camera, and then register and weakly align the visible light and infrared image regions. The training image data enhancement and noise adding module is used to introduce pixel offset, dithering blur or single channel removal of three kinds of data noise in the weakly aligned visible light and infrared images. The image twin feature extraction and adaptive fusion network module is used to extract multi-layer twin feature maps of the visible light and infrared images, and then reduce dimensions according to the feature map channel numbers obtained by the scene illumination estimation module, and finally fuse the feature maps based on adaptive weights. The image fusion feature complementarity and target prediction network module is used to share information of the fused feature maps and predict the final target information.

[0025] Based on the above system, the all-weather unmanned aerial scene visible light infrared fusion target detection method is as follows: Step one: scene illumination analysis After obtaining the visible light original image collected by the unmanned aerial vehicle, the scene illumination estimation module calculates the average brightness and root mean square contrast of the gray image of the visible light original image, and then according to the threshold of the average brightness and root mean square contrast, the fixed feature map channel number is distributed to the visible light and infrared feature map channel number, so as to adjust the contribution of the visible light and infrared to the feature map according to the scene illumination estimation.

[0026] The average brightness of the visible light original image is represented by formula (1): (1) In formula (1), represents the average brightness of the visible light original image, and represent the width and height of the visible light original image respectively, and represent the horizontal coordinate and vertical coordinate of the pixel of the visible light original image respectively, represents the three-channel average brightness of the pixel of the visible light original image at coordinate .

[0027] The root mean square contrast of the visible light original image is represented by formula (2): (2) In formula (2), represents the root mean square contrast of the visible light original image.

[0028] According to the average brightness and root mean square contrast of the visible light original image, the feature map channel numbers of the visible light and infrared are obtained, which are represented by formula (3): (3) In formula (3), and represent the feature map channel numbers of the visible light and infrared respectively, represents the fixed number of feature map channels parameter and is a multiple of 4, and respectively represent the threshold parameters of average brightness and root mean square contrast; wherein, typical values of are 128, 256 and 512, a typical value of is 100, a typical value of is 50 (when the maximum gray value of the image is 256). If the average brightness and the root mean square contrast of the visible light original image are both higher than the threshold parameters, the ratio of the number of feature channels of visible light and infrared is 3:1; if both are lower than the threshold parameters, the ratio of the number of feature channels of visible light and infrared is 1:3; otherwise, the ratio of the number of feature channels of visible light and infrared is 1:1.

[0029] Step two: image preprocessing After the visible light and infrared original images collected by the unmanned aerial vehicle camera are obtained by the original image distortion correction and registration module, the imaging distortion of the visible light original image and the infrared original image is corrected according to the camera distortion coefficient, and then the corresponding image area of the infrared image is cropped and scaled in the visible light image after distortion correction, to obtain weakly aligned visible light and infrared images, wherein the weak alignment refers to that the maximum offset of the corresponding pixel points of the visible light and infrared images does not exceed 8 pixels.

[0030] The distortion correction of the visible light original image and the infrared original image can be expressed by formula (4): (4) In formula (4), represents the image after distortion correction, represents the original image, represents the camera intrinsic matrix, , and represent the radial distortion coefficients, and represent the tangential distortion coefficients, represents the distortion correction function.

[0031] The cropping and scaling of the visible light image after distortion correction is expressed by formula (5): (5) In formula (5), represents the weakly aligned visible light image, represents the visible light image after distortion correction, , , and represent the top-left corner and the bottom-right corner pixel coordinates of the cropped visible light image after distortion correction, and denotes the width and height of the infrared image, denotes a quadratic interpolation scale function.

[0032] In order to make the model better cope with real scenes, in the training stage of the neural network model, the application also includes a data enhancement step. Before the weakly aligned image pair is sent into the network, random data noise is applied to the visible light image or the infrared image, and the noise types include pixel shift (randomly moving the image within a small range), blur (simulating motion blur), or single-channel removal (simulating local information loss). This step forces the model to learn to robustly extract and fuse features even when the image quality is poor, thereby greatly improving the generalization ability and robustness of the final model.

[0033] The training image data enhancement and noise addition module introduces three kinds of data noise, pixel shift, blur, or single-channel removal, in the weakly aligned visible light and infrared images, for either visible light or infrared image, to support the model parameter training process of the image twin feature extraction and adaptive fusion network module and the image fusion feature complementation and target prediction network module. This module can be omitted in the actual prediction process of visible light infrared fusion target detection.

[0034] The pixel shift data noise of the training image is represented by formula (6): (6) In formula (6), denotes the image after adding pixel shift, denotes the weakly aligned image, and denote the pixel coordinates of the image; and denote the maximum pixel shift in the horizontal and vertical coordinates of the image, and the constraints and ; and denote the shift direction in the horizontal and vertical coordinates of the image, and the constraints and .

[0035] The blur data noise of the training image is represented by formula (7): (7) In formula (7), denotes the image after adding blur, denotes the weakly aligned image, and denote the pixel shift in the horizontal and vertical coordinates of the image.

[0036] Single-channel data noise removal of the training image refers to replacing a local image region of any image in visible light or infrared with a mean value of pixels in the region to represent image data loss in the region.

[0037] Step three: adaptive feature fusion The image twin feature extraction and adaptive fusion network module first extracts multi-layer twin feature maps of visible light and infrared images by using two full convolution backbone networks with the same structure and different parameters. Then, the visible light and infrared twin features are reduced in dimension and spliced according to the number of feature map channels obtained by the scene illumination estimation module. Finally, the dual light fusion network is used to calculate the gate convolution block weight of the visible light and infrared feature maps and perform feature fusion.

[0038] The full convolution backbone network adopts the backbone network of the YOLO detection algorithm, which is composed of five convolution blocks P1-P5. The P3, P4 and P5 stages of the two full convolution backbone networks output multi-layer twin feature maps of visible light and infrared images respectively and , and , and , the channel number of each layer feature map satisfies .

[0039] The feature map dimension reduction is calculated according to the result of formula (3), and 1*1 convolution is used to reduce the dimension of the multi-layer twin feature maps of visible light and infrared images to reduce the parameter amount, that is, there are three different structures of 1*1 convolution network for different cases of formula (3), and the one-to-one corresponding feature maps are obtained and , and , and , the channel number of each layer feature map satisfies .

[0040] As shown in Figure 2 , the dual light fusion network adopts a full convolution fusion network, which is composed of a splicing layer and three gate convolution blocks with depth separable convolution stacked in series. Each gate convolution block has a double branch, wherein the content branch is composed of layer normalization, linear layer, 3*3 convolution layer, the gate branch is composed of layer normalization, linear layer, 3*3 depth separable convolution layer and activation layer, the outputs of the two branches are multiplied to perform gate modulation, the modulation result is processed through a linear layer, and then the processed features are added to the original input through residual connection to form the final output. The feature extraction backbone network P3, P4 and P5 stage each has an independent feature fusion network , and the fusion obtains one-to-one corresponding feature maps , and Equation (8) is shown as follows: (8) Step four: target prediction The image fusion feature complementarity and target prediction network module first uses information based on the improved progressive feature pyramid network shared fusion feature map to improve the learning ability of texture and semantic features, and finally uses the standard network prediction head based on YOLO to obtain the target category, detection frame coordinates and confidence through classification and regression.

[0041] Based on the improved progressive feature pyramid network, the feature map is up-sampled, down-sampled, spliced and convoluted. On the basis of the standard progressive feature pyramid network, 2*2 convolution and 4*4 convolution are replaced by 3*3 convolution with a stride of 2 to realize 2 times down-sampling and 2 layers of 3*3 convolution with a stride of 2 to realize 4 times down-sampling, thereby improving the learning ability of semantic features; in the up-sampling process, the bilinear interpolation is replaced by the transposed convolution to improve the reconstruction ability of the texture features. Based on the improved progressive feature pyramid network, the shallow features Gradual fusion of deep features and After fusion, the feature maps , and are obtained, and equation (9) is shown as follows: (9) In equation (9), and represent two consecutive networks based on the improved progressive feature pyramid network, and represent the intermediate features obtained after the fusion of features and .

[0042] As described above, although the present application has been described and expressed with reference to specific preferred embodiments, it should not be construed as a limitation on the present application itself. Various changes can be made in form and details without departing from the spirit and scope of the present application as defined in the appended claims.

Claims

1. A method for visible light and infrared fusion target detection in all-weather UAV scenarios, characterized in that, Includes the following steps: S1: Acquire and analyze raw visible light and infrared images captured by the drone camera to determine one or more parameters characterizing the scene's lighting conditions; S2: Perform distortion correction and registration on the original visible light image and the original infrared image to generate a weakly aligned visible light image and infrared image within a preset offset range; S3: Two parallel fully convolutional backbone networks are used to extract multi-layer twin feature maps from the weakly aligned visible light image and infrared image respectively, and the visible light feature map and infrared feature map are fused according to the scene lighting condition parameters to generate a fused feature map. S4: Input the fused feature map into the target prediction network to generate the detection result of the target.

2. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 1, characterized in that, The scene lighting condition parameters include the average brightness and root mean square contrast calculated based on the original visible light image; The fusion of the visible light feature map and the infrared feature map is achieved by dynamically adjusting the number of channels allocated to the visible light feature map and the infrared feature map based on the comparison results of the average brightness and root mean square contrast with their respective preset thresholds.

3. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 2, characterized in that, The average brightness of the original visible light image is calculated using the following formula: In the formula, This represents the average brightness of the original visible light image. and These represent the width and height of the original visible light image, respectively. and These represent the x and y coordinates of the pixels in the original visible light image, respectively. Represents the coordinates in the raw visible light image The average brightness of a pixel across its three channels; The root mean square contrast of the original visible light image is calculated using the following formula: In the formula, This represents the root mean square contrast of the original visible light image.

4. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 3, characterized in that, The number of feature map channels for visible light and infrared light is optimized based on the average brightness and root mean square contrast of the original visible light image, as shown below: In the formula, and These represent the number of channels in the visible light and infrared feature maps, respectively. This indicates a fixed number of feature map channels, which is a multiple of 4. and These represent the threshold parameters for average brightness and root mean square contrast, respectively.

5. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 1, characterized in that, The distortion correction is achieved through the following formula: In the formula, This represents the image after distortion correction. Represents the original image. Represents the camera intrinsic parameter matrix. , and Represents the radial distortion coefficient. and Indicates the tangential distortion coefficient. Represents the distortion correction function; In the corrected visible light image, registration is performed by cropping and scaling to align the corresponding rectangular imaging regions of the visible light image and the infrared image.

6. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 1, characterized in that, The weak alignment refers to the maximum offset of corresponding pixels in the visible light image and the infrared image not exceeding 8 pixels; when training the neural network model used to implement the method, data noise is introduced into the weakly aligned visible light image or infrared image, and the data noise is selected from at least one of pixel offset, jitter blur or single-channel removal.

7. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 1, characterized in that, The fusion step specifically includes: employing two parallel fully convolutional backbone networks to extract multi-layer visible light feature maps and multi-layer infrared feature maps from the weakly aligned visible light image and infrared image, respectively; performing channel dimensionality reduction on the multi-layer visible light feature maps and multi-layer infrared feature maps using 1x1 convolution operations according to the scene illumination condition parameters; concatenating the dimensionality-reduced visible light feature maps and infrared feature maps and inputting them into the dual-light fusion network, which contains at least one gated convolutional block composed of depthwise separable convolutions to calculate adaptive weights and generate the fused feature map.

8. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 7, characterized in that, The fully convolutional backbone network adopts the YOLO detection algorithm and includes five stages of convolutional blocks P1 to P5. The multi-layer visible light feature map and multi-layer infrared feature map are output from stages P3, P4 and P5 respectively. The gated convolutional block has a dual-branch structure, including: The content branch consists of layer normalization, linear layers, and 3*3 convolutional layers; The gated branch consists of layer normalization, linear layers, 3*3 depthwise separable convolutional layers, and activation layers; The outputs of the content branch and the gated branch are gated and modulated by element-wise multiplication. The modulation result is then added to the original input through residual connection to form the final output.

9. The visible light and infrared fusion target detection method for all-weather UAV scenarios according to claim 1, characterized in that, The target prediction network is an improved progressive feature pyramid network, which enhances the fused feature map across levels through upsampling, downsampling, and concatenation operations for final classification and regression prediction. Furthermore, the downsampling is achieved using a 3*3 convolution with a stride of 2, and the upsampling is achieved using a transposed convolution.

10. A system for implementing the visible light and infrared fusion target detection method for all-weather UAV scenarios as described in claim 1, characterized in that, include: The scene lighting estimation module is used to analyze the raw visible light image and estimate the scene lighting conditions based on the analysis results. The original image distortion correction and registration module is used to perform distortion correction and registration on the original visible light and infrared images, generating weakly aligned image pairs; The image twin feature extraction and adaptive fusion network module is used to extract multi-layer twin feature maps from the weakly aligned image pairs respectively, and to adaptively fuse the twin feature maps according to the estimation results of the scene illumination estimation module to generate a dual-light fusion feature map. The image fusion feature complementarity and target prediction network module is used to process the dual-light fusion feature map and use an improved progressive feature pyramid network prediction head to predict the final target information.

Images