An unmanned aerial vehicle low-illumination visible light image enhancement method and device based on infrared structure guidance and two-stage target supplement

Abstract

The application belongs to the technical field of image processing, and discloses a UAV low-illumination visible light image enhancement method and device based on infrared structure guidance and two-stage target supplement, which comprises the following steps: using the relatively stable edge and contour information of an infrared image under weak light conditions to guide and correct the features of a low-illumination visible light image, and combining brightness residual modulation and Gamma correction to complete the first-stage illumination enhancement; further introducing infrared significant target information to perform second-stage brightness supplement and saliency enhancement on personnel, heat source targets and other infrared high-response regions. The application improves the image distinguishability, target highlighting capability and result stability under a low-illumination scene on the basis of maintaining the naturalness of the result, and can be applied to night monitoring, unmanned system visual perception, security monitoring and complex environment imaging scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and apparatus for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation. Background Technology

[0002] Low-light image enhancement plays a crucial role in scenarios such as nighttime surveillance, visual perception of unmanned systems, imaging of complex environments, and multimodal image processing. By enhancing low-light visible light images, image brightness, contrast, and detail discernibility can be improved, thereby providing more stable input for subsequent target detection, scene understanding, image fusion, and recognition analysis.

[0003] In recent years, existing low-light image enhancement methods mainly include histogram-based methods, Retinex-based methods, and deep learning-based methods. Deep learning-based methods, in particular, can learn the mapping relationship between low-light and normal-light images through samples, achieving some success in brightness restoration. However, most existing methods still rely on a single visible light image for processing. When the scene is at night or in severely insufficient lighting, the edge, texture, and target contour information in the visible light image are significantly attenuated, easily leading to problems such as insufficient texture restoration, local blurring, and structural distortion. Furthermore, some methods use overall gain adjustment or direct regression to enhance the results, which can easily cause local over-enhancement, bright area overflow, dark area noise amplification, and color imbalance, making it difficult to balance brightness restoration and structure preservation. In addition, existing methods often lack utilization of external stable structural information. In low-light scenes, if the structural prior in infrared images cannot be effectively utilized, the enhancement results remain limited in terms of edge sharpness and target saliency.

[0004] Therefore, it is necessary to propose a method and device for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation, so as to solve the above problems. Summary of the Invention

[0005] To address the problems of insufficient utilization of structural information, unstable brightness recovery, easy over-enhancement of local areas, and weak ability to highlight salient targets in existing low-light image enhancement methods, this invention provides a method and device for enhancing UAV low-light visible light images based on infrared structure guidance and two-stage target supplementation. This method first utilizes the relatively stable edge and contour information of infrared images under low-light conditions to guide and correct the features of the low-light visible light image, and completes the first stage of illumination enhancement by combining brightness residual modulation and Gamma correction. Subsequently, based on the first stage enhancement result, infrared salient target information is further introduced to perform a second stage of brightness supplementation and saliency enhancement for personnel, heat source targets, and other infrared high-response areas. Through processing steps such as dual-modal feature extraction, infrared guided feature correction, shared coding, residual brightness generation, and salient target supplementation fusion, scene texture and target structure can be maintained while improving image brightness, and the discernibility of salient target areas in nighttime scenes can be further improved. This invention improves image discernibility, target highlighting ability, and result stability in low-light scenes while maintaining the naturalness of the results, and can be applied to scenarios such as nighttime surveillance, visual perception of unmanned systems, security monitoring, and imaging in complex environments.

[0006] In a first aspect, to achieve the above objectives, the present invention provides a method for enhancing low-light visible light images of unmanned aerial vehicles (UAVs) based on infrared structure guidance and two-stage target supplementation, comprising: Acquire low-light visible light and infrared images of the same scene; The low-light visible light image and infrared image are input into the low-light enhancement model for image enhancement to obtain the final enhanced image. The low-light enhancement model includes a first-stage enhancement network and a second-stage supplementary fusion network. The first-stage enhancement network includes a dual-modal feature extraction module, an infrared guided feature correction module, and a shared coding and residual prediction module connected in sequence. The second-stage supplementary fusion network includes a salient target region determination module and an infrared salient target supplementary fusion module connected in sequence.

[0007] Optionally, the first-stage enhancement network processing procedure specifically includes: The low-light visible light image and infrared image are input into the dual-modal feature extraction module for feature extraction to obtain the initial visible light features and infrared structural features. The initial visible light features and infrared structural features are input into the infrared guided feature correction module. The initial visible light features are guided and corrected based on the infrared structural features to obtain the corrected features. The correction features are input into the shared coding and residual prediction module. The correction features are shared coded and the brightness residual map is predicted. Based on the brightness residual map, the low-illuminance visible light image is subjected to residual brightness modulation and gamma correction to obtain the first-stage enhancement result image.

[0008] Optionally, the processing procedure of the dual-modal feature extraction module specifically includes: Convolution operations are used to perform basic texture encoding on the input low-light visible light image to obtain the initial visible light features; Edge information and local texture information in different directions of infrared images are extracted by multiple parallel directional sensing branches. The multiple parallel directional sensing branches include convolution with a kernel size of 1×3, convolution with a kernel size of 3×1, and convolution with a kernel size of 3×3. The output features of each branch are fused and nonlinearly activated to obtain infrared structural features.

[0009] Optionally, the processing procedure of the infrared guidance feature correction module specifically includes: Frequency decomposition of the initial features of visible light yields low-frequency and high-frequency features of visible light. Frequency decomposition of infrared structural features yields low-frequency and high-frequency infrared features. Based on the infrared high-frequency characteristics, proportional modulation parameters and offset modulation parameters for the visible light high-frequency characteristics are generated. The visible light high-frequency characteristics are then corrected based on the proportional modulation parameters and offset modulation parameters to obtain candidate high-frequency correction results. The visible light high-frequency features and the infrared high-frequency features are spliced ​​together to generate a gating weight map; By using a gated weight map, an adaptive selection is made between candidate high-frequency correction results and original visible light high-frequency features to obtain the final high-frequency correction features; The low-frequency visible light features are convolved and mapped, and then recombined with the final high-frequency correction features to obtain the corrected features.

[0010] Optionally, the processing procedure of the shared coding and residual prediction module specifically includes: The corrected features are input into a shared encoding network to obtain shared representation features; wherein, the shared encoding network is composed of multiple convolutional layers, and intermediate features from different layers are added and fused through a cross-layer aggregation structure; A brightness residual map is generated based on the shared representation features. The input low-light image is then subjected to residual brightness modulation and gamma correction using the brightness residual map to obtain the first-stage enhancement result image.

[0011] Optionally, the second-stage supplementary fusion network processing procedure specifically includes: The enhancement results of the first-stage enhancement network are combined with the infrared image and input into the salient target region determination module to predict the supplementary salient target mask; The salient target supplementation mask is input into the infrared salient target supplementation fusion module. Based on the predicted salient target supplementation mask and the infrared image, the enhancement result of the first-stage enhancement network is supplemented with brightness to obtain the final enhanced image.

[0012] Optionally, the processing procedure of the salient target region determination module specifically includes: The enhancement results of the first-stage enhancement network and the infrared image are input into the supplementary mask prediction network. After multi-layer convolutional coding, joint features are obtained, and the supplementary mask for salient targets is predicted based on the joint features.

[0013] Optionally, the processing procedure of the infrared salient target supplementation and fusion module specifically includes: The enhancement results of the first-stage enhancement network are converted to the luminance-chrominance space and the luminance and chrominance components are extracted. The luminance component is supplemented according to the salient target supplementation mask and infrared image to obtain the supplemented luminance component. The supplemented luminance component is then recombined with the chrominance component to obtain the final enhanced image.

[0014] Optionally, the training process of the low-light enhancement model specifically includes: Acquire training data, which includes low-light visible light images, normal-brightness visible light images, and infrared images of the same scene; An initial low-light enhancement model is constructed, and the training data is input into the initial low-light enhancement model for image enhancement. The model is then trained according to the target loss function to obtain the trained low-light enhancement model. During the training process, the target loss function of the first-stage enhancement network includes structural loss, exposure loss, color angle loss, structure-aware weighted loss, and pixel-level RGB reconstruction loss. For the second-stage supplementary fusion network, significant target region weights are constructed based on the high-response regions in the infrared image. The supplementary mask prediction results and the fused output image are constrained based on the significant target region weights.

[0015] Secondly, to achieve the above objectives, the present invention provides a low-light visible light image enhancement device for unmanned aerial vehicles based on infrared structure guidance and two-stage target supplementation, comprising: The data acquisition module is used to acquire low-light visible light images and infrared images of the same scene; The image enhancement module is used to input the low-light visible light image and infrared image into the low-light enhancement model for image enhancement to obtain the final enhanced image. The low-light enhancement model includes a first-stage enhancement network and a second-stage supplementary fusion network. The first-stage enhancement network includes a dual-modal feature extraction module, an infrared guided feature correction module, and a shared coding and residual prediction module connected in sequence. The second-stage supplementary fusion network includes a salient target region determination module and an infrared salient target supplementary fusion module connected in sequence.

[0016] The technical effects of this invention are as follows: This invention introduces infrared images as structural guidance information, providing more stable contour and edge references for visible light enhancement under low-light conditions, thus improving the structural clarity of the first-stage enhancement result. The invention uses a combination of luminance residual modulation and Gamma correction to generate the first-stage enhancement result, rather than directly adjusting the gain of the entire image. Therefore, it can better preserve the color information and local details of the original image while improving brightness. Furthermore, this invention includes a second-stage salient target supplementation and fusion module, which improves the discernibility of personnel, heat sources, and other salient target areas by utilizing high-response infrared regions while maintaining visible light chromaticity information. During the training phase, this invention uses online synthesis of low-light samples and mixed training with normal-light samples, which helps improve the model's adaptability to input images under different nighttime brightness conditions. The output of this invention is an enhanced three-channel color image, which can be directly used as input for subsequent image fusion, target detection, and scene analysis tasks, demonstrating good engineering applicability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the composition of the UAV low-light visible light image enhancement network device in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall structure of the low-light visible light image enhancement network in an embodiment of the present invention; Figure 3 This is a schematic diagram of the network architecture of the dual-modal feature extraction module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network architecture of the infrared guidance feature correction module in an embodiment of the present invention; Figure 5 This is a schematic diagram of the network architecture of the residual prediction module in an embodiment of the present invention; Figure 6 This is a schematic diagram of the network architecture of the supplementary mask prediction module and the infrared salient target supplementary module in an embodiment of the present invention.

[0019] Labeling explanations: 101. Computer; 102. Drone; 103. Visible light camera; 104. Infrared camera; 105. Pedestrian in low light scene. Detailed Implementation

[0020] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0021] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0022] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0023] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Example 1 like Figure 1 - Figure 6As shown, this embodiment provides a method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation. The method includes: acquiring low-light visible light images and infrared images of the same scene; inputting the low-light visible light images and infrared images into a low-light enhancement model for image enhancement to obtain the final enhanced image; wherein the low-light enhancement model includes a first-stage enhancement network and a second-stage supplementation fusion network. The first-stage enhancement network includes a dual-modal feature extraction module, an infrared guidance feature correction module, and a shared coding and residual prediction module connected in sequence; the second-stage supplementation fusion network includes a salient target region determination module and an infrared salient target supplementation fusion module connected in sequence. This embodiment, based on the color information of the visible light image, introduces structure guidance information from the infrared image, and through the first-stage illumination enhancement and the second-stage infrared salient target supplementation, achieves a synergistic improvement in the brightness, structure, and target saliency of the low-light image, thereby improving the brightness recovery capability, structure preservation capability, and result stability of the enhanced image.

[0026] like Figure 1 As shown, this embodiment provides a UAV low-light visible light image enhancement device based on infrared structure guidance and two-stage target supplementation, including: a computer 101, a UAV 102, a visible light camera 103, an infrared camera 104, and a pedestrian 105 in a low-light scene. The visible light camera 103 and the infrared camera 104 acquire visible light and infrared images of the same scene. The computer 101 performs image registration and executes a low-light image enhancement algorithm, performing two-stage processing on the low-light visible light image and outputting the enhanced visible light image. The low-light image enhancement algorithm includes the following modules: a data processing module, a dual-modal feature extraction module, an infrared-guided feature correction module, a shared coding and residual prediction module, a salient target region determination and supplementary mask prediction module, and an infrared salient target supplementation and fusion module. The data processing module simultaneously receives low-light visible light images and infrared images, and sets parameters to degrade the normal-light visible light image to a low-light visible light image. The dual-modal feature extraction module extracts the basic features of both modalities. The infrared-guided feature correction module uses infrared structural information to correct the visible light features. The shared coding and residual prediction module outputs a brightness residual map. The salient target region determination and supplementary mask prediction module outputs a brightness residual map modulated by the residual. The corrected enhanced image is then supplemented by the infrared salient target supplementation and fusion module, which combines the infrared salient response region to process the first-stage enhancement result and outputs the final enhanced image.

[0027] A method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation is used to obtain the enhanced low-light images. The specific implementation steps are as follows: Step 1: As Figure 1As shown, images in both visible and infrared modes were captured by drones, primarily targeting pedestrians, to construct an RGB-IR image dataset I. d =[I RGB1 I IR1 I RGB2 I IR2 , ...I RGBK I IRK ], where dataset I d The total number of elements in the middle is 2K (in this embodiment) This refers to 380 pairs of RGB and IR image data, obtained by registering raw images captured by a drone. The image size is... , For image channels, Image height, Image width (in this embodiment, the visible light image size is...) The size of the infrared image is During the training phase, clear visible light images were used as the supervised target images, and [the following was utilized]: Degradation method for online construction of low-light input images, assuming a clear visible light image is used. The low-light input image is Then we have: In the above formula, Represents a clear visible light image. This represents the generated low-light input image. This represents the degradation parameters used to construct the low-light sample. In this embodiment, The value range is set to 1.5 to 3.5. Meanwhile, the infrared image corresponding to this visible light image is denoted as... In addition to synthesizing low-light samples, some normal brightness samples are also directly input into the network during training to suppress the network's tendency to over-enhance normal images and improve the model's adaptability to different brightness scenes.

[0028] Step 2.1: The overall network structure is as follows Figure 2 As shown, firstly, the low-light visible light image is input into the first-stage enhancement network to obtain initial visible light features; the infrared image is input into the infrared structural feature extraction branch to obtain infrared structural features. Then, the infrared structural features are used to guide and correct the initial visible light features, resulting in corrected features. Next, the corrected features are input into the shared coding network to obtain shared representation features for the enhancement task, and a brightness residual map is generated based on the shared representation features. The first stage performs residual brightness modulation on the input low-light image based on the brightness residual map, and then... The image is corrected to obtain the first-stage enhanced image. In the second stage, the first-stage enhanced image and the infrared image are input into a salient target supplementation fusion network. A salient target supplementation mask is constructed in this network to supplement the brightness of high infrared response areas. While maintaining visible light chromaticity information, the final enhanced image is output, making people, heat source targets, and other salient target areas clearer in night scenes.

[0029] Step 2.2: Dual-modal feature extraction module as follows Figure 3 As shown, feature extraction is performed on the input low-light visible light image and infrared image respectively to obtain the initial visible light features and infrared structural features used for the first stage of enhancement. Among them, the initial visible light features... It can be represented as: In the above formula, This represents the visible light shallow layer feature extraction function. This represents the input low-light visible light image. This represents the initial features of visible light. In one specific implementation, the shallow visible light feature extraction uses a convolutional layer with a kernel size of 3×3, an output channel count of 16, and an activation function of a linear rectified function with a leakage slope to obtain basic texture features suitable for subsequent enhancement processing.

[0030] Infrared structural features It can be represented as: In the above formula, This represents the infrared structural feature extraction function. This indicates the input infrared image. This represents the extracted infrared structural features. In a specific implementation, the infrared structural feature extraction module includes three parallel convolutional branches, which extract horizontal edge information, vertical edge information, and local composite texture information, respectively. The process can be represented as follows: In the above formula, This represents a convolution operation with a kernel size of 1×3. This represents a convolution operation with a kernel size of 3×1. This indicates a convolution operation with a kernel size of 3×3. , , These represent the output features of the three directional branches, respectively. Represents a non-linear activation function. This indicates the infrared structural features after fusion.

[0031] Step 2.3: Infrared guidance feature correction module as follows Figure 4 As shown, the initial visible light features and infrared structural features are input into the feature correction module. First, the initial visible light features are decomposed into low-frequency and high-frequency features. Simultaneously, the infrared structural features are decomposed into low-frequency and high-frequency infrared features. The process can be represented as follows: In the above formula, This represents the average pooling operation. This indicates the low-frequency characteristics of visible light. Represents the high-frequency characteristics of visible light. Indicates low-frequency infrared characteristics. This represents the high-frequency infrared characteristics. In one specific implementation, the average pooling operation uses 5×5 average pooling.

[0032] Then, based on the infrared high-frequency characteristics, proportional modulation parameters and offset modulation parameters for the visible light high-frequency characteristics are generated. This process can be expressed as follows: In the above formula, This represents a 1×1 convolution mapping used to generate the scaling parameters. This represents the 1×1 convolution mapping used to generate the offset modulation parameters. Indicates the proportional modulation parameter. This represents the offset modulation parameter. Based on the proportional modulation parameter and the offset modulation parameter, the high-frequency characteristics of visible light are corrected to obtain the candidate high-frequency correction result, which can be expressed as: In the above formula, This indicates the candidate high-frequency correction result.

[0033] Next, the visible light high-frequency features and the infrared high-frequency features are concatenated to generate a gated weight map. This process can be represented as follows: In the above formula, This indicates a feature concatenation operation. This indicates a convolution operation with a kernel size of 3×3. This represents the Sigmoid activation function. This represents the gated weight map. The gated weight map is used for adaptive selection between candidate high-frequency correction results and original visible light high-frequency features. The final high-frequency correction features can be represented as: In the above formula, This represents the final high-frequency correction feature. Finally, the processed high-frequency features are recombined with the low-frequency processing results to obtain the corrected features, which can be represented as: In the above formula, This represents a 1×1 convolution mapping applied to low-frequency features. This represents the correction feature. Using the above method, the relatively stable structural information of infrared images under low-light conditions can be utilized to compensate for edge and texture responses in visible light features.

[0034] Step 2.4: Shared coding and residual prediction module, as shown Figure 5 As shown, the correction features Inputting into a shared coding network yields shared representation features. , can be represented as: In the above formula, Indicates shared encoding function, Indicates the correction feature, This represents shared representation features. In one specific implementation, the shared encoding network consists of multiple 3×3 convolutional layers, and intermediate features from different layers are added and fused through a cross-layer aggregation structure to enhance the expressive power of multi-scale texture information. A brightness residual map is generated based on the shared representation features. , can be represented as: In the above formula, This represents the brightness residual prediction function. This represents the brightness residual map. The brightness residual map describes the enhancement magnitude corresponding to each pixel position in the input image. In a specific implementation, the brightness residual map is a three-channel image, the convolution kernel size is set to 3×3, the number of output channels is set to 3, and the activation function is a linear rectified function to ensure that the brightness residual is non-negative, so that this branch mainly performs brightening operations without negatively weakening the input brightness. Based on the brightness residual map, residual brightness modulation is performed on the input low-light image to obtain the initial enhanced image of the first stage. , can be represented as: In the above formula, This indicates that the input image is a low-light image. Represents the brightness residual map. This represents the initial enhanced image for the first stage. Then, the initial enhanced image for the first stage is processed... Correction yields the first-stage enhanced image. , can be represented as: In the above formula, Represents the numerically stable term. express Correction factor, This represents the image resulting from the first stage of enhancement. To obtain a stable output from the first stage, further adjustments can be made... After truncation, we get: In the above formula, This represents the truncation function. This represents the first-stage enhancement result image after truncation. The first stage mainly completes global illumination restoration and texture enhancement, and provides the enhanced base visible light representation and brightness prior for the salient targets in the second stage.

[0035] Step 2.5: The salient target region identification and supplementary mask prediction module, as shown in the example... Figure 6 As shown, the second stage uses the enhanced image from the first stage and the infrared image as joint input to identify high-response infrared regions in the night scene and construct a supplementary mask for salient targets. The enhanced image from the first stage, the infrared image, and the auxiliary brightness information generated from the first stage are jointly input into the supplementary mask prediction network, and joint features are obtained through multi-layer convolutional coding. , can be represented as: In the above formula, This represents the joint feature extraction function in the second stage. This represents the first-stage enhancement result image after truncation. Represents an infrared image. This represents the joint features of the second stage. A supplementary mask for salient targets is predicted based on these joint features. , can be represented as: In the above formula, This represents the convolutional mapping used to generate the supplementary mask. express Activation function This refers to a salient target supplementary mask. The salient target supplementary mask represents the weight of the infrared brightness information introduced at each pixel location. The mask value is larger for people, heat sources, and other high infrared response areas; and smaller for ordinary background areas. In this way, the second stage can adaptively locate the target areas that need to be highlighted without repeatedly brightening the overall image.

[0036] Step 2.6: Infrared salient target supplementation and fusion module, as shown Figure 6 As shown, the second stage of infrared luminance supplementation is performed on the first-stage enhanced image based on the salient target supplementation mask. To preserve the color information of the visible light image, the first-stage enhanced image is first converted to a luminance-chrominance space, and the luminance and chrominance components are extracted, which can be represented as: In the above formula, This represents a color space conversion function. This represents the luminance component of the image obtained from the first stage of enhancement. , This represents the chromaticity component of the image resulting from the first stage of enhancement. Infrared supplementation of the luminance component based on the salient target supplementation mask can be expressed as: In the above formula, This represents the brightness component after supplementation with salient targets in the second stage. Represents an infrared image. This represents a mask for supplementing salient targets. The supplemented luminance components are then reconstructed with the chrominance components of the first-stage enhancement image to obtain the final enhanced image, which can be represented as: In the above formula, R This represents a function that reconstructs a three-channel image from the luminance and chrominance components. This represents the final enhanced image. To ensure output stability, the final enhanced image can also be truncated, limiting pixel values ​​to the range of 0 to 1. Through this method, the first-stage enhancement network is responsible for restoring the overall illumination in low-light scenes, while the second-stage supplementary fusion network is responsible for improving the visibility of people, heat sources, and other infrared-salient areas.

[0037] Step 3: During the training phase, a clear visible light image is used as the supervised target image. The output image of the first-stage enhancement network and the supervised target image are used together to calculate the loss function. The output image of the second-stage supplementary fusion network is used to constrain the brightness preservation and structure recovery of the infrared salient target region. During the first-stage training, structural loss, exposure loss, color angle loss, structure-aware weighted loss, and pixel-level RGB reconstruction loss are used to jointly constrain the network. Among them, structural loss is used to constrain the edge and texture gradient of the enhancement result to be consistent with the supervised target image; exposure loss is used to constrain the brightness distribution of the enhancement result; color angle loss and pixel-level RGB reconstruction loss are used to constrain the color direction and pixel reconstruction accuracy of the enhancement result; and structure-aware weighted loss is used to enhance the recovery quality of complex texture regions and edge regions. During the second-stage training, the first-stage enhancement result image and the infrared image are input together into the salient target supplementary fusion network. The weights of the salient target region are constructed based on the high-response regions in the infrared image, and the target brightness enhancement ability and gradient recovery ability of the supplementary mask prediction result and the fused output image are constrained to ensure that the salient target region maintains higher discernibility in the enhanced image.

[0038] Set the network training parameters as follows: learning rate The batch size is 10, and the training block size is [missing information]. Training cycle The optimizer uses the Adam optimizer, and global gradient clipping is used during training with a clipping norm set to 5.0. The weights of each loss term can be set as exposure loss weights. Color angle loss weight Pixel-level RGB reconstruction loss Structural loss weights Structure-aware weighted loss weights .

[0039] Step 4: Use the trained low-light enhancement network to infer the image to be enhanced, converting the low-light visible light image to the image to be enhanced. and corresponding infrared images Inputting the data into the network first yields the brightness residual map. The input low-light image is then subjected to residual brightness modulation based on the brightness residual map to obtain the initial enhanced image for the first stage. Subsequently, on conduct Correction and truncation processes yield the first-stage enhanced image. Then, the first-stage enhanced image is processed. With infrared images The common input to the second-stage salient target supplementation network first predicts the salient target supplementation mask. Then, based on the mask, the brightness of the infrared salient target area is supplemented in the brightness space to obtain the final enhanced image. The first-stage enhancement process can be represented as: The supplementary mask for the salient target in the second stage can be represented as: The final enhanced image after the second stage of salient target addition can be represented as: In the above formula, This indicates a low-light visible light image that needs to be enhanced. This represents the corresponding infrared image. This represents the brightness residual map output by the network. Represents the numerically stable term. express Correction factor, This represents the first-stage enhancement result image after truncation. This represents the joint feature extraction function in the second stage. This indicates that the second phase of salient targets requires the addition of corresponding masks. , , These represent the luminance and chrominance components of the first-stage enhanced image in the luminance space, respectively. This represents a function that reconstructs a three-channel image from the luminance and chrominance components. This represents the truncation function. This represents the final enhanced image output. The output image is a three-channel RGB image with the same size as the input image. It can be saved as an image file or used as input for subsequent image fusion, object detection, and scene recognition tasks. Low-light image enhancement processing is complete when the brightness, edge, and texture information of low-light areas and the discernibility of infrared salient target areas in the output image are improved.

[0040] In summary, this embodiment introduces infrared images as structural guidance information, providing more stable contour and edge references for visible light enhancement under low-light conditions, which helps improve the structural clarity of the first-stage enhancement result. This embodiment uses a combination of luminance residual modulation and correction to generate the first-stage enhancement result, rather than directly adjusting the gain of the entire image. Therefore, it can better preserve the color information and local details of the original image while improving brightness. This embodiment further sets up a second-stage salient target supplementation and fusion module, which improves the discernibility of personnel, heat source targets, and other salient target areas by utilizing the high-response infrared region while maintaining visible light chromaticity information. The training phase of this embodiment uses online synthesis of low-light samples and mixed training with normal brightness samples, which helps improve the model's adaptability to input images under different nighttime brightness conditions. The output of this embodiment is an enhanced three-channel color image, which can be directly used as input for subsequent image fusion, target detection, and scene analysis tasks, demonstrating good engineering applicability.

[0041] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation, characterized in that, include: Acquire low-light visible light and infrared images of the same scene; The low-light visible light image and infrared image are input into the low-light enhancement model for image enhancement to obtain the final enhanced image. The low-light enhancement model includes a first-stage enhancement network and a second-stage supplementary fusion network. The first-stage enhancement network includes a dual-modal feature extraction module, an infrared guided feature correction module, and a shared coding and residual prediction module connected in sequence. The second-stage supplementary fusion network includes a salient target region determination module and an infrared salient target supplementary fusion module connected in sequence.

2. The method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 1, characterized in that, The first stage of the network enhancement process specifically includes: The low-light visible light image and infrared image are input into the dual-modal feature extraction module for feature extraction to obtain the initial visible light features and infrared structural features. The initial visible light features and infrared structural features are input into the infrared guided feature correction module. The initial visible light features are guided and corrected based on the infrared structural features to obtain the corrected features. The correction features are input into the shared coding and residual prediction module. The correction features are shared coded and the brightness residual map is predicted. Based on the brightness residual map, the low-illuminance visible light image is subjected to residual brightness modulation and gamma correction to obtain the first-stage enhancement result image.

3. The method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 2, characterized in that, The processing procedure of the dual-modal feature extraction module specifically includes: Convolution operations are used to perform basic texture encoding on the input low-light visible light image to obtain the initial visible light features; Edge information and local texture information in different directions of infrared images are extracted by multiple parallel directional sensing branches. The multiple parallel directional sensing branches include convolution with a kernel size of 1×3, convolution with a kernel size of 3×1, and convolution with a kernel size of 3×3. The output features of each branch are fused and nonlinearly activated to obtain infrared structural features.

4. The method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 2, characterized in that, The processing procedure of the infrared guidance feature correction module specifically includes: Frequency decomposition of the initial features of visible light yields low-frequency and high-frequency features of visible light. Frequency decomposition of infrared structural features yields low-frequency and high-frequency infrared features. Based on the infrared high-frequency characteristics, proportional modulation parameters and offset modulation parameters for the visible light high-frequency characteristics are generated. The visible light high-frequency characteristics are then corrected based on the proportional modulation parameters and offset modulation parameters to obtain candidate high-frequency correction results. The visible light high-frequency features and the infrared high-frequency features are spliced ​​together to generate a gating weight map; By using a gated weight map, an adaptive selection is made between candidate high-frequency correction results and original visible light high-frequency features to obtain the final high-frequency correction features; The low-frequency visible light features are convolved and mapped, and then recombined with the final high-frequency correction features to obtain the corrected features.

5. A method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 2, characterized in that, The processing procedure of the shared coding and residual prediction module specifically includes: The corrected features are input into a shared encoding network to obtain shared representation features; wherein, the shared encoding network is composed of multiple convolutional layers, and intermediate features from different layers are added and fused through a cross-layer aggregation structure; A brightness residual map is generated based on the shared representation features. The input low-light image is then subjected to residual brightness modulation and gamma correction using the brightness residual map to obtain the first-stage enhancement result image.

6. The method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 1, characterized in that, The second stage of the supplementary fusion network processing specifically includes: The enhancement results of the first-stage enhancement network are combined with the infrared image and input into the salient target region determination module to predict the supplementary salient target mask; The salient target supplementation mask is input into the infrared salient target supplementation fusion module. Based on the predicted salient target supplementation mask and the infrared image, the enhancement result of the first-stage enhancement network is supplemented with brightness to obtain the final enhanced image.

7. The method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 6, characterized in that, The processing procedure of the salient target region determination module specifically includes: The enhancement results of the first-stage enhancement network and the infrared image are input into the supplementary mask prediction network. After multi-layer convolutional coding, joint features are obtained, and the supplementary mask for salient targets is predicted based on the joint features.

8. A method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 6, characterized in that, The processing procedure of the infrared salient target supplementation and fusion module specifically includes: The enhancement results of the first-stage enhancement network are converted to the luminance-chrominance space and the luminance and chrominance components are extracted. The luminance component is supplemented according to the salient target supplementation mask and infrared image to obtain the supplemented luminance component. The supplemented luminance component is then recombined with the chrominance component to obtain the final enhanced image.

9. A method for enhancing low-light visible light images of UAVs based on infrared structure guidance and two-stage target supplementation according to claim 1, characterized in that, The training process of the low-light enhancement model specifically includes: Acquire training data, which includes low-light visible light images, normal-brightness visible light images, and infrared images of the same scene; An initial low-light enhancement model is constructed, and the training data is input into the initial low-light enhancement model for image enhancement. The model is then trained according to the target loss function to obtain the trained low-light enhancement model. During the training process, the target loss function of the first-stage enhancement network includes structural loss, exposure loss, color angle loss, structure-aware weighted loss, and pixel-level RGB reconstruction loss. For the second-stage supplementary fusion network, significant target region weights are constructed based on the high-response regions in the infrared image. The supplementary mask prediction results and the fused output image are constrained based on the significant target region weights.

10. A low-light visible light image enhancement device for unmanned aerial vehicles based on infrared structure guidance and two-stage target supplementation, characterized in that, include: The data acquisition module is used to acquire low-light visible light images and infrared images of the same scene; The image enhancement module is used to input the low-light visible light image and infrared image into the low-light enhancement model for image enhancement to obtain the final enhanced image. The low-light enhancement model includes a first-stage enhancement network and a second-stage supplementary fusion network. The first-stage enhancement network includes a dual-modal feature extraction module, an infrared guided feature correction module, and a shared coding and residual prediction module connected in sequence. The second-stage supplementary fusion network includes a salient target region determination module and an infrared salient target supplementary fusion module connected in sequence.

Images