Sparsity-based image enhancement method and system under low-illumination condition

Abstract

The invention discloses a sparsity-based image enhancement method and system under a low-illumination condition, and relates to the technical field of image processing. According to the method, the sparsity of the image is considered, the attention mechanism and the feedforward neural network are combined, the low-illumination unmanned aerial vehicle image enhancement model is constructed, the features and details of the unmanned aerial vehicle image under the low-illumination condition can be effectively extracted, and the problem of color distortion or lack of details of the unmanned aerial vehicle image is solved; the image quality and definition of the enhanced unmanned aerial vehicle image are improved, identification, detection or monitoring of the unmanned aerial vehicle is facilitated, and further utilization of the unmanned aerial vehicle is assisted; the system is simple in structure, the corresponding low-illumination unmanned aerial vehicle image enhancement model is constructed, the unmanned aerial vehicle image under the low-illumination condition is effectively enhanced, and the night image quality of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image enhancement method and system under low illumination conditions based on sparsity. Background Art

[0002] With the continuous development of unmanned aerial vehicle (UAV) technology, the problem of night image enhancement has become a challenging issue attracting much attention in UAV vision applications. In many UAV missions, such as night cruising, security patrol, and night search and rescue, UAVs usually need to capture images under low illumination conditions, which results in a decline in image quality, presenting problems such as low contrast, color distortion, noise amplification, and detail loss, restricting the effectiveness and safety of UAV operations.

[0003] Low illumination image enhancement technology has broad application potential in the UAV field, which can improve the quality of UAV night images and enhance the efficiency and accuracy of task execution. The application scope of this technology is not limited to night aerial photography, but also includes the following fields: UAV night monitoring: UAVs are used for monitoring and patrolling at night, and low illumination image enhancement technology can help improve the quality of monitoring images and increase the reliability of the monitoring system. Night search and rescue: In night search and rescue missions, UAVs can use low illumination image enhancement technology to enhance the visibility of the search area and improve the success rate of search and rescue operations. Night military applications: UAVs are used for military reconnaissance and surveillance at night, and low illumination image enhancement technology can enhance the images transmitted back by UAVs, providing more intelligence and situation reports.

[0004] To solve the problem of image enhancement under low illumination conditions, previous methods mainly relied on traditional image processing techniques, such as histogram equalization, filtering, and increasing exposure. However, when dealing with images under low illumination conditions, these methods usually introduce additional noise or lose the detail information of the images, so the enhancement effect is limited. In recent years, image enhancement methods based on deep learning have made remarkable progress and become the mainstream technology for current low illumination image enhancement. However, under low illumination conditions, the information in the images is usually severely affected by uneven illumination and noise interference. Therefore, an innovative method is needed to handle the problem of image enhancement under low illumination conditions of UAVs to improve image quality and retain detail information. However, existing image enhancement methods based on deep learning do not consider the sparsity of low illumination images, resulting in problems such as color distortion or incomplete detail restoration in the enhanced pictures. Summary of the Invention

[0005] The purpose of the present invention is to provide an image enhancement method and system under low illumination conditions based on sparsity to improve the above technical problems.

[0006] To achieve the above object of the invention, the embodiments of the present invention provide the following technical solutions:

[0007] An image enhancement method based on sparsity under low illumination conditions includes:

[0008] S1. Obtain a low-illumination UAV image to be enhanced; the low-illumination UAV image is an image captured by a UAV under low illumination conditions;

[0009] S2. Preprocess the low-illumination UAV image to be enhanced;

[0010] S3. Based on image sparsity, construct a low-illumination UAV image enhancement model;

[0011] S4. Input the preprocessed low-illumination UAV image into the low-illumination UAV image enhancement model, and output the enhanced UAV image.

[0012] Further, the preprocessing of the low-illumination UAV image to be enhanced includes:

[0013] S2-1. Randomly crop the low-illumination UAV image to be enhanced to obtain a cropped low-illumination UAV image;

[0014] S2-2. Perform data augmentation on the cropped low-illumination UAV image to obtain the preprocessed low-illumination UAV image; the data augmentation includes random flipping, rotation, and random cropping.

[0015] Further, the low-illumination UAV image enhancement model includes a background modeling-detail restoration submodel, a global feature fusion module, and a Conv3 layer connected in series;

[0016] The background modeling-detail restoration submodel includes seven background modeling-detail restoration modules, respectively serving as the first background modeling-detail restoration submodule to the seventh background modeling-detail restoration submodule; the output of the first background modeling-detail restoration submodule is respectively input into the second background modeling-detail restoration submodule and the seventh background modeling-detail restoration submodule; the output of the second background modeling-detail restoration submodule is respectively input into the third background modeling-detail restoration submodule and the sixth background modeling-detail restoration submodule; the output of the third background modeling-detail restoration submodule is respectively input into the fourth background modeling-detail restoration submodule and the fifth background modeling-detail restoration submodule; the output of the fourth background modeling-detail restoration submodule is respectively input into the fifth background modeling-detail restoration submodule and the global feature fusion module; the output of the fifth background modeling-detail restoration submodule is respectively input into the sixth background modeling-detail restoration submodule and the global feature fusion module; the output of the sixth background modeling-detail restoration submodule is respectively input into the seventh background modeling-detail restoration submodule and the global feature fusion module; the output of the seventh background modeling-detail restoration submodule is input into the global feature fusion module;

[0017] The global feature fusion module includes an upsampling layer, a first convolutional layer, a second convolutional layer, a first sparse self-attention layer, and a third convolutional layer connected in series in sequence; both the first convolutional layer and the third convolutional layer use a 1×1 convolutional kernel; the second convolutional layer uses a 3×3 convolutional kernel.

[0018] Furthermore, each background modeling-detail restoration sub-module includes a position encoding sub-module, a convolutional-self attention sub-module, and a feed-forward neural network sub-module connected in series in sequence; the position encoding sub-module includes a fourth convolutional layer and a DWConv layer with a 3×3 convolutional kernel connected in series; the convolutional-self attention sub-module includes a dilated convolutional layer and a second sparse self-attention layer in parallel; the dilated convolutional layer includes a Dilated Conv3 layer and a first activation layer connected in series; both the first sparse self-attention layer and the second sparse self-attention layer use the sparse self-attention mechanism; the feed-forward neural network sub-module includes a normalization layer, a fifth convolutional layer, a convolutional sub-module, and a sixth convolutional layer connected in series; the convolutional sub-module includes a seventh convolutional layer, a convolutional layer, and a ninth convolutional layer in parallel; the convolutional layer includes an eighth convolutional layer and a second activation layer connected in series; both the first activation layer and the second activation layer use the GELU activation function; the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer all use a 1×1 convolutional kernel; the seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer all use a 3×3 convolutional kernel.

[0019] Furthermore, the training process of the low-light UAV image enhancement model includes:

[0020] S3-1. Obtain low-light UAV training images and their labels, and preprocess the low-light UAV training images using the same method as in S2 to obtain the preprocessed low-light UAV training images;

[0021] S3-2. Input the preprocessed low-light UAV training images and their labels into the background modeling-detail restoration sub-model, and output the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data;

[0022] S3-3. Input the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data into the global feature fusion module, and output the fused training feature map data;

[0023] S3-4. Input the fused training feature map data into the Conv3 layer, and output the convolution-processed fused training feature map data;

[0024] S3-5. Add the convolution-processed fused training feature map data and the preprocessed low-light UAV training images, and output the enhanced UAV training images;

[0025] S3-6. Based on the enhanced UAV training images, construct the corresponding training loss function;

[0026] S3-7. Based on the training loss function, use the backpropagation method to calculate the gradients of the low-light UAV image enhancement model, and use the Adam optimization algorithm to adjust the parameters of the low-light UAV image enhancement model;

[0027] S3-8. Repeat S3-1 to S3-7 until the training error is less than the error threshold or the preset number of iterations is reached, and complete the training of the low-light UAV image enhancement model.

[0028] Further, S3-2 includes the following steps:

[0029] S3-2-1. Input the preprocessed low-light UAV training images into the first background modeling-detail restoration sub-module, and output the first initial training feature map data;

[0030] S3-2-2. Input the first initial training feature map data into the second background modeling-detail restoration sub-module, and output the second initial training feature map data;

[0031] S3-2-3. Input the second initial training feature map data into the third background modeling-detail restoration sub-module, and output the third initial training feature map data;

[0032] S3-2-4. Input the third initial training feature map data into the fourth background modeling-detail restoration sub-module, and output the first training feature map data;

[0033] S3-2-5. Input the third initial training feature map data and the first training feature map data into the fifth background modeling-detail restoration sub-module, and output the second training feature map data;

[0034] S3-2-6. Input the second initial training feature map data and the second training feature map data into the sixth background modeling-detail restoration sub-module, and output the third training feature map data;

[0035] S3-2-7. Input the first initial training feature map data and the third training feature map data into the seventh background modeling-detail restoration sub-module, and output the fourth training feature map data.

[0036] Further, S3-2-5 includes the following steps:

[0037] S3-2-5-1. Concatenate the third initial training feature map data and the first training feature map data to obtain the concatenated initial training feature map data;

[0038] S3-2-5-2. Input the concatenated initial training feature map data into the position encoding sub-module of the fifth background modeling-detail restoration sub-module, and output the encoded initial training feature map data;

[0039] S3-2-5-3. Input the encoded initial training feature map data into the convolution-self-attention sub-module of the fifth background modeling-detail restoration sub-module, and output the initial training key feature map data;

[0040] S3-2-5-4. Input the initial training key feature map data into the feed-forward neural network sub-module of the fifth background modeling-detail restoration sub-module, and output the second training feature map data.

[0041] Furthermore, S3-3 includes the following steps:

[0042] S3-3-1. Take the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data as the training feature map data ; where represents the permutation serial number;

[0043] S3-3-2. Input the training feature map data into the upsampling layer, and output the sampled training feature map data;

[0044] S3-3-3. Input the sampled training feature map data into the first convolutional layer, and output the training feature map data with unified dimensions;

[0045] S3-3-4. Input the training feature map data with unified dimensions into the second convolutional layer, and output the encoded training feature map data;

[0046] S3-3-5. Input the encoded training feature map data into the first sparse self-attention layer, and output the enhanced training feature map data;

[0047] S3-3-6. Input the enhanced training feature map data into the third convolutional layer, and output the fused training feature map data.

[0048] An image enhancement system under low illumination conditions based on sparsity includes:

[0049] A drone image acquisition module, used to acquire the low illumination drone image to be enhanced;

[0050] An image preprocessing module for preprocessing the low-light UAV images to be enhanced;

[0051] A low-light UAV image enhancement module for constructing a low-light UAV image enhancement model based on image sparsity; inputting the preprocessed low-light UAV images into the low-light UAV image enhancement model, and outputting the enhanced UAV images.

[0052] The beneficial effects of the present invention are as follows:

[0053] This method considers the sparsity of the image, combines the attention mechanism and the feedforward neural network to construct a low-light UAV image enhancement model, which can effectively extract the features and details of the UAV images under low-light conditions, solve the problems of color distortion or lack of details in UAV images, improve the image quality and clarity of the enhanced UAV images, facilitate the recognition, detection or monitoring of UAVs, etc., and assist in the further utilization of UAVs;

[0054] The structure of this system is simple. By constructing the corresponding low-light UAV image enhancement model, the effective enhancement of UAV images under low-light conditions can be realized, and the quality of UAV night images can be improved. Description of the Drawings

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, so they should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0056] Figure 1 It is the flowchart of the method in the embodiment of the present invention;

[0057] Figure 2 It is the structural schematic diagram of the background modeling-detail restoration sub-model in the embodiment of the present invention;

[0058] Figure 3 It is the structural schematic diagram of the global feature fusion module in the embodiment of the present invention;

[0059] Figure 4 It is the structural schematic diagram of the background modeling module in the embodiment of the present invention;

[0060] Figure 5 It is the system structure diagram in the embodiment of the present invention. Detailed Embodiments

[0061] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0062] Please refer to Figure 1 , a method for image enhancement under low illumination conditions based on sparsity provided in this embodiment includes:

[0063] S1. Obtain a low-illumination UAV image to be enhanced;

[0064] S2. Preprocess the low-illumination UAV image to be enhanced;

[0065] The preprocessing of the low-illumination UAV image to be enhanced includes:

[0066] S2-1. Randomly crop the low-illumination UAV image to be enhanced, and randomly crop the low-illumination UAV image with a size of to obtain a UAV image with a size of , that is, obtain the cropped low-illumination UAV image; where represents the height of the low-illumination UAV image, represents the width of the low-illumination UAV image, represents the height and width of the cropped low-illumination UAV image;

[0067] S2-2. Perform data augmentation on the cropped low-illumination UAV image to obtain the preprocessed low-illumination UAV image; the data augmentation includes random flipping, rotation, and random cropping.

[0068] S3. Based on image sparsity, construct a low-illumination UAV image enhancement model;

[0069] As Figure 2 shown, the low-illumination UAV image enhancement model includes a series-connected background modeling-detail restoration submodel, a global feature fusion module, and a Conv3 layer;

[0070] The background modeling-detail restoration submodel includes seven background modeling-detail restoration modules, which are respectively used as the first to seventh background modeling-detail restoration submodules. The output of the first background modeling-detail restoration submodule is respectively input into the second and seventh background modeling-detail restoration submodules. The output of the second background modeling-detail restoration submodule is respectively input into the third and sixth background modeling-detail restoration submodules. The output of the third background modeling-detail restoration submodule is respectively input into the fourth and fifth background modeling-detail restoration submodules. The output of the fourth background modeling-detail restoration submodule is respectively input into the fifth background modeling-detail restoration submodule and the global feature fusion module. The output of the fifth background modeling-detail restoration submodule is respectively input into the sixth background modeling-detail restoration submodule and the global feature fusion module. The output of the sixth background modeling-detail restoration submodule is respectively input into the seventh background modeling-detail restoration submodule and the global feature fusion module. The output of the seventh background modeling-detail restoration submodule is input into the global feature fusion module.

[0071] As Figure 3 shown, the global feature fusion module includes an upsampling layer, a first convolutional layer, a second convolutional layer, a first sparse self-attention layer, and a third convolutional layer connected in series. Both the first convolutional layer and the third convolutional layer use 1×1 convolutional kernels. The second convolutional layer uses a 3×3 convolutional kernel.

[0072] As Figure 4 shown, each background modeling-detail restoration submodule includes a position encoding submodule, a convolutional-self attention submodule, and a feed-forward neural network submodule connected in series. The position encoding submodule includes a fourth convolutional layer and a 3×3 DWConv layer connected in series. The convolutional-self attention submodule includes a dilated convolutional layer and a second sparse self-attention layer in parallel. The dilated convolutional layer includes a Dilated Conv3 layer and a first activation layer connected in series. Both the first sparse self-attention layer and the second sparse self-attention layer use the sparse self-attention mechanism. The feed-forward neural network submodule includes a normalization layer, a fifth convolutional layer, a convolutional submodule, and a sixth convolutional layer connected in series. The convolutional submodule includes a seventh convolutional layer, a convolutional layer, and a ninth convolutional layer in parallel. The convolutional layer includes an eighth convolutional layer and a second activation layer connected in series. Both the first activation layer and the second activation layer use the GELU activation function. The fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer all use 1×1 convolutional kernels. The seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer all use 3×3 convolutional kernels.

[0073] The training process of the low-light UAV image enhancement model includes:

[0074] S3-1. Obtain low-light UAV training images and pair corresponding labels; preprocess the low-light UAV training images using the same method as in S2 to obtain preprocessed low-light UAV training images; where the label is the GroundTruth image corresponding to the low-light UAV training image, and the same processing is performed on the low-light UAV training image.

[0075] S3-2. Input the preprocessed low-light UAV training images and their labels into the background modeling-detail restoration submodel, and output the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data.

[0076] The S3-2 includes the following steps:

[0077] S3-2-1. Input the preprocessed low-light UAV training images into the first background modeling-detail restoration submodule, and output the first initial training feature map data.

[0078] S3-2-2. Input the first initial training feature map data into the second background modeling-detail restoration submodule, and output the second initial training feature map data.

[0079] S3-2-3. Input the second initial training feature map data into the third background modeling-detail restoration submodule, and output the third initial training feature map data.

[0080] S3-2-4. Input the third initial training feature map data into the fourth background modeling-detail restoration submodule, and output the first training feature map data.

[0081] S3-2-5. Input the third initial training feature map data and the first training feature map data into the fifth background modeling-detail restoration submodule, and output the second training feature map data.

[0082] S3-2-6. Input the second initial training feature map data and the second training feature map data into the sixth background modeling-detail restoration submodule, and output the third training feature map data.

[0083] S3-2-7. Input the first initial training feature map data and the third training feature map data into the seventh background modeling-detail restoration submodule, and output the fourth training feature map data.

[0084] The S3-2-5 includes the following steps:

[0085] S3-2-5-1. Concatenate the third initial training feature map data and the first training feature map data to obtain the concatenated initial training feature map data;

[0086] S3-2-5-2. Input the concatenated initial training feature map data into the position encoding sub-module of the fifth background modeling-detail restoration sub-module, and output the encoded initial training feature map data. Use the fourth convolutional layer with a convolution kernel of 1×1 to increase the image dimension of the concatenated initial training feature map data, obtaining the processed initial training feature map data with a size of ; Perform position encoding through the DWConv layer (depthwise separable convolution) with a convolution kernel of 3×3 to obtain the encoded initial training feature map data . .

[0087] The formula corresponding to S3-2-5-2 is:

[0088] ;

[0089] ;

[0090] where, represents the concatenated initial training feature map data, represents the fourth convolutional layer, represents the DWConv layer.

[0091] S3-2-5-3. Input the encoded initial training feature map data into the convolution-self-attention sub-module of the fifth background modeling-detail restoration sub-module, and output the initial training key feature map data;

[0092] The formula corresponding to the convolution-self-attention sub-module is:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] where, , , represent the query vector, key vector, and value vector respectively, represents the activation function, represents the operation of average division by channel dimension, represents the sparse self-attention, represents the dimension of the key vector, Represents the output data of the second sparse self-attention layer, Represents the output data of the dilated convolutional layer, Represents the dilated convolution operation corresponding to the Dilated Conv3 layer, Represents the activation function. The initial training key feature map data includes the output data of the second sparse self-attention layer and the output data of the dilated convolutional layer .

[0098] S3-2-5-4. Input the initial training key feature map data into the feed-forward neural network sub-module of the fifth background modeling-detail restoration sub-module, and output the second training feature map data ;

[0099] Add the output data of the second sparse self-attention layer and the output data of the dilated convolutional layer , then input it into the normalization layer. Use the fifth convolutional layer with a convolution kernel of 1×1 to increase the dimension of the feature map output by the normalization layer, and input it into the convolutional sub-module. In the convolutional sub-module, it enters three branches respectively. The first branch extracts features through the seventh convolutional layer with a convolution kernel of 3×3; the second branch extracts features through the eighth convolutional layer with a convolution kernel of 3×3, and then uses the GELU activation function to process the feature map output by the eighth convolutional layer; the third branch extracts features through the ninth convolutional layer with a convolution kernel of 3×3; perform element-wise multiplication on the feature maps output by the three branches, and then restore to the original dimension through the sixth convolutional layer with a convolution kernel of 1×1. That is, the formula corresponding to S3-2-5-4 is:

[0100] ;

[0101] ;

[0102] ;

[0103] Among them, Represents the initial training key feature map data, Represents the concatenation function, Represents the intermediate function, , respectively represent 1×1 convolution and 3×3 convolution, Represents the normalization layer, Represents the activation function, Represents 1×1 convolution, Represents the second training feature map data, Represents element-wise multiplication.

[0104] Among them, the same method as that of S3-2-5-1 to S3-2-5-4 is adopted for S3-2-6 to S3-2-7, and the same method as that of S3-2-2-2 to S3-2-2-4 is adopted for S3-2-1 to S3-2-4.

[0105] S3-3. Input the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data into the global feature fusion module, and output the fused training feature map data;

[0106] The S3-3 includes the following steps:

[0107] S3-3-1. Take the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data as the training feature map data ; where represents the permutation serial number;

[0108] S3-3-2. To unify the feature maps of different layers, input the training feature map data into the upsampling layer for upsampling, and output the upsampled training feature map data; when is 1, the training feature map data is the first training feature map data, and so on for 2, 3, 4.

[0109] S3-3-3. Input the upsampled training feature map data into the first convolutional layer with a convolution kernel of 1×1 for dimension unification, and output the dimension-unified training feature map data;

[0110] S3-3-4. Input the dimension-unified training feature map data into the second convolutional layer with a convolution kernel of 3×3 for position encoding, and output the encoded training feature map data;

[0111] S3-3-5. Input the encoded training feature map data into the first sparse self-attention layer for feature enhancement, and output the enhanced training feature map data;

[0112] S3-3-6. Input the enhanced training feature map data into the third convolutional layer with a convolution kernel of 1×1 for convolution, and output the fused training feature map data .

[0113] Thus, the formula corresponding to S3-3 is:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] Among them, represents a 3×3 convolution, represents an intermediate function, represents upsampling, represents the dimension of the key vector.

[0119] The global feature fusion module can fuse the global information of the preprocessed low-light UAV training images, adaptively enhance the background and detail features corresponding to the preprocessed low-light UAV training images, and can further improve the image enhancement ability of the low-light UAV image enhancement model.

[0120] S3-4. Input the fused training feature map data into the Conv3 layer, and output the fused training feature map data after convolution ;

[0121] S3-5. Add the fused training feature map data after convolution and the preprocessed low-light UAV training image, and output the enhanced UAV training image , and the corresponding formula is:

[0122] ;

[0123] S3-6. Based on the enhanced UAV training image, construct the corresponding training loss function , and the corresponding formula is:

[0124] ;

[0125] Among them, represents the total number of enhanced UAV training images, represents the low-light UAV training image corresponding GroundTruth image, represents the absolute value, represents the summation function;

[0126] S3-7. Based on the training loss function, use the backpropagation method to calculate the gradient of the low-light UAV image enhancement model, and use the Adam optimization algorithm to adjust the parameters of the low-light UAV image enhancement model;

[0127] S3-8. Repeat S3-1 to S3-7 until the training error is less than the error threshold or reaches the preset number of iterations, and complete the training of the low-light UAV image enhancement model.

[0128] S4. Input the preprocessed low-light UAV images into the low-light UAV image enhancement model, and output the enhanced UAV images.

[0129] As Figure 5 shown, an image enhancement system under low-light conditions based on sparsity includes:

[0130] A UAV image acquisition module for acquiring low-light UAV images to be enhanced;

[0131] An image preprocessing module for preprocessing the low-light UAV images to be enhanced;

[0132] A low-light UAV image enhancement module for constructing a low-light UAV image enhancement model based on image sparsity; inputting the preprocessed low-light UAV images into the low-light UAV image enhancement model, and outputting the enhanced UAV images.

[0133] In summary, the method of the present invention considers the sparsity of images, combines the attention mechanism and the feed-forward neural network to construct a low-light UAV image enhancement model, which can effectively extract the features and details of UAV images under low-light conditions, solve the problems of color distortion or lack of details in UAV images, improve the image quality and clarity of the enhanced UAV images, facilitate the recognition, detection or monitoring of UAVs, and assist in the further utilization of UAVs; the system structure of the present invention is simple, constructs a corresponding low-light UAV image enhancement model, realizes the effective enhancement of UAV images under low-light conditions, and improves the quality of UAV night images.

[0134] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An image enhancement method under low illumination conditions based on sparsity, characterized in that, Including: S1. Obtain a low-light UAV image to be enhanced; The low-light UAV image is an image captured by a UAV under low-light conditions; S2. Preprocess the low-light UAV image to be enhanced; S3. Based on image sparsity, construct a low-light UAV image enhancement model; S4. Input the preprocessed low-light UAV image into the low-light UAV image enhancement model, and output the enhanced UAV image.

2. The image enhancement method under low illumination conditions based on sparsity according to claim 1, characterized in that, The preprocessing of the low-light UAV image to be enhanced includes: S2-1. Randomly crop the low-light UAV image to be enhanced to obtain a cropped low-light UAV image; S2-2. Perform data augmentation on the cropped low-light UAV image to obtain the preprocessed low-light UAV image; the data augmentation includes random flipping, rotation, and random cropping.

3. The image enhancement method under low illumination conditions based on sparsity according to claim 1, wherein The low-light UAV image enhancement model includes a sequentially connected background modeling-detail restoration submodel, a global feature fusion module, and a Conv3 layer; The background modeling-detail restoration submodel includes seven background modeling-detail restoration submodules, which are respectively used as the first background modeling-detail restoration submodule to the seventh background modeling-detail restoration submodule; the output of the first background modeling-detail restoration submodule is respectively input into the second background modeling-detail restoration submodule and the seventh background modeling-detail restoration submodule; the output of the second background modeling-detail restoration submodule is respectively input into the third background modeling-detail restoration submodule and the sixth background modeling-detail restoration submodule; the output of the third background modeling-detail restoration submodule is respectively input into the fourth background modeling-detail restoration submodule and the fifth background modeling-detail restoration submodule; the output of the fourth background modeling-detail restoration submodule is respectively input into the fifth background modeling-detail restoration submodule and the global feature fusion module; The output of the fifth background modeling-detail restoration submodule is respectively input into the sixth background modeling-detail restoration submodule and the global feature fusion module; The output of the sixth background modeling-detail restoration submodule is respectively input into the seventh background modeling-detail restoration submodule and the global feature fusion module; The output of the seventh background modeling-detail restoration submodule is input into the global feature fusion module; The global feature fusion module includes an upsampling layer, a first convolutional layer, a second convolutional layer, a first sparse self-attention layer, and a third convolutional layer connected in sequence; both the first convolutional layer and the third convolutional layer use a 1×1 convolutional kernel; the second convolutional layer uses a 3×3 convolutional kernel.

4. The method for image enhancement under low illumination conditions based on sparsity according to claim 3, wherein Each background modeling-detail restoration submodule includes a position encoding submodule, a convolutional-self attention submodule, and a feed-forward neural network submodule connected in sequence; the position encoding submodule includes a fourth convolutional layer and a DWConv layer with a 3×3 convolutional kernel connected in sequence; the convolutional-self attention submodule includes a dilated convolutional layer and a second sparse self-attention layer in parallel; the dilated convolutional layer includes a Dilated Conv3 layer and a first activation layer connected in sequence; Both the first sparse self-attention layer and the second sparse self-attention layer adopt the sparse self-attention mechanism; the feed-forward neural network sub-module includes a concatenated normalization layer, a fifth convolutional layer, a convolutional sub-module, and a sixth convolutional layer; the convolutional sub-module includes parallel seventh convolutional layer, convolutional layer, and ninth convolutional layer; the convolutional layer includes a concatenated eighth convolutional layer and a second activation layer; both the first activation layer and the second activation layer adopt the GELU activation function; the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer all adopt 1×1 convolutional kernels; the seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer all adopt 3×3 convolutional kernels.

5. The method for image enhancement under low illumination conditions based on sparsity according to claim 4, wherein The training process of the low-light UAV image enhancement model includes: S3-1. Obtain low-light UAV training images and their labels, and preprocess the low-light UAV training images using the same method as S2 to obtain preprocessed low-light UAV training images; S3-2. Input the preprocessed low-light UAV training images and their labels into the background modeling-detail restoration sub-model, and output the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data; S3-3. Input the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data into the global feature fusion module, and output the fused training feature map data; S3-4. Input the fused training feature map data into the Conv3 layer, and output the convolved fused training feature map data; S3-5. Add the convolved fused training feature map data and the preprocessed low-light UAV training images, and output the enhanced UAV training images; S3-6. Based on the enhanced UAV training images, construct the corresponding training loss function; S3-7. Based on the training loss function, use the backpropagation method to calculate the gradient of the low-light UAV image enhancement model, and use the Adam optimization algorithm to adjust the parameters of the low-light UAV image enhancement model; S3-8. Repeat S3-1 to S3-7 until the training error is less than the error threshold or reaches the preset number of iterations, and complete the training of the low-light UAV image enhancement model.

6. The method for image enhancement under low illumination conditions based on sparsity according to claim 5, wherein, The S3-2 includes the following steps: S3-2-1. Input the preprocessed low-light UAV training images into the first background modeling-detail restoration sub-module, and output the first initial training feature map data; S3-2-2. Input the first initial training feature map data into the second background modeling-detail restoration sub-module, and output the second initial training feature map data; S3-2-3. Input the second initial training feature map data into the third background modeling-detail restoration sub-module, and output the third initial training feature map data; S3-2-4. Input the third initial training feature map data into the fourth background modeling-detail restoration sub-module, and output the first training feature map data; S3-2-5. Input the third initial training feature map data and the first training feature map data into the fifth background modeling-detail restoration sub-module, and output the second training feature map data; S3-2-6. Input the second initial training feature map data and the second training feature map data into the sixth background modeling-detail restoration sub-module, and output the third training feature map data; S3-2-7. Input the first initial training feature map data and the third training feature map data into the seventh background modeling-detail restoration sub-module, and output the fourth training feature map data.

7. The method for image enhancement under low illumination conditions based on sparsity according to claim 6, wherein The S3-2-5 includes the following steps: S3-2-5-1. Concatenate the third initial training feature map data and the first training feature map data to obtain the concatenated initial training feature map data; S3-2-5-2. Input the concatenated initial training feature map data into the position encoding sub-module of the fifth background modeling-detail restoration sub-module, and output the encoded initial training feature map data; S3-2-5-3. Input the encoded initial training feature map data into the convolution-self-attention sub-module of the fifth background modeling-detail restoration sub-module, and output the initial training key feature map data; S3-2-5-4. Input the initial training key feature map data into the feed-forward neural network sub-module of the fifth background modeling-detail restoration sub-module, and output the second training feature map data.

8. The method for image enhancement under low illumination conditions based on sparsity according to claim 5, wherein, The S3-3 includes the following steps: S3-3-1. Use the first training feature map data, the second training feature map data, the third training feature map data, and the fourth training feature map data as the training feature map data ; where represents the permutation serial number S3-3-2. Input the training feature map data into the upsampling layer, and output the sampled training feature map data; S3-3-3. Input the sampled training feature map data into the first convolutional layer, and output the training feature map data with unified dimensions; S3-3-4. Input the training feature map data with unified dimensions into the second convolutional layer, and output the encoded training feature map data; S3-3-5. Input the encoded training feature map data into the first sparse self-attention layer, and output the enhanced training feature map data; S3-3-6. Input the enhanced training feature map data into the third convolutional layer, and output the fused training feature map data.

9. An image enhancement system under low illumination conditions based on sparsity, which is used to implement an image enhancement method under low illumination conditions based on sparsity according to any one of claims 1 to 8, characterized in that It includes: A drone image acquisition module, which is used to acquire low-light drone images to be enhanced; An image preprocessing module, which is used to preprocess the low-light drone images to be enhanced; A low-light drone image enhancement module, which is used to construct a low-light drone image enhancement model based on image sparsity; Input the preprocessed low-light drone images into the low-light drone image enhancement model, and output the enhanced drone images.

Images