A two-stage color correction method for images of unmanned aerial vehicles

Abstract

This invention discloses a two-stage color correction method for UAV images, relating to the field of UAV image color correction technology. The method includes the following steps: aerial photography of a target area with several color swatches, ensuring all color swatches are included in all UAV-captured images; constructing a weighted color correction matrix using multiple linear regression to perform standard color correction on the UAV-captured images; building and training a deep learning model based on the standard color-corrected image; acquiring the color-distorted image to be corrected and inputting it into the trained deep learning model, outputting a prediction residual image; subtracting the prediction residual image from the color-distorted image to obtain the final corrected image. This application improves color correction accuracy and efficiency by constructing a weighted color correction matrix based on multiple color swatches and using the standard color-corrected image obtained through the weighted color correction matrix as a benchmark to build and train a deep learning model.

Description

Technical Field

[0001] This application relates to the field of color correction technology for UAV images, and in particular to a two-stage color correction method for UAV images. Background Technology

[0002] In recent years, UAV remote sensing images have been widely used in agriculture, environmental monitoring, and urban planning. However, due to unavoidable factors such as changes in lighting conditions, camera limitations, and atmospheric conditions, the acquired images generally suffer from severe color inconsistencies, directly affecting data quality and the accuracy of subsequent analysis. In agricultural applications, vegetation indices are typically calculated using multispectral or RGB images to assess crop growth, health status, pest and disease prevalence, and yield prediction. However, in practical agricultural applications, color inconsistencies can directly lead to errors in vegetation index calculations. For example, a passing cloud darkening the image might be misinterpreted as a decline in crop health, resulting in incorrect agricultural operation instructions. Therefore, there is an urgent need to explore a new color correction method for UAV images.

[0003] In the prior art, Chinese patent CN118657697A discloses a method and system for color correction of field rice images acquired by drones based on improved GPR, including the following steps: using a drone to capture images of a 24-color chart and rice; extracting the mean values ​​of the R, G, and B channels within a region color patch based on the 24-color chart image; using the mean values ​​of the R, G, and B channels within the region color patch to train and optimize a Gaussian process regression model to obtain a color correction model based on improved Gaussian process regression; and using the color correction model based on improved Gaussian process regression to perform color correction on the 24-color chart and rice image to obtain the final corrected field rice image.

[0004] However, the aforementioned existing technologies rely solely on a single 24-color chart and employ a single-stage Gaussian process regression model. They fail to effectively integrate the advantages of traditional correction and deep learning, making it difficult to adapt to the complex lighting and altitude variations encountered in drone operations. They cannot balance correction accuracy and efficiency in complex scenarios, and the color correction accuracy and work efficiency need to be improved. Summary of the Invention

[0005] This application provides a two-stage color correction method for UAV images to address the problems that existing UAV image color correction technologies have not effectively integrated the advantages of traditional correction and deep learning, making it difficult to adapt to the complex lighting and altitude changes in UAV operations, and failing to balance correction accuracy and efficiency in complex scenarios, thus requiring improvements in color correction accuracy and work efficiency.

[0006] On the one hand, this application provides a two-stage color correction method for UAV images, including the following steps: Step 1: Use a drone to take aerial photos of the target area with several color swatches, and obtain several drone-captured images. All drone-captured images contain all the color swatches.

[0007] Step 2: Based on several color swatches, a weighted color correction matrix is ​​constructed using multiple linear regression technology. The weighted color correction matrix is ​​then used to perform standard color correction on the images captured by the UAV to obtain a standard color-corrected image.

[0008] Step 3: Using the standard color-corrected image as a reference, construct and train a deep learning model.

[0009] Step 4: Obtain the color distortion image to be corrected and input it into the trained deep learning model. Output the prediction residual image and subtract the prediction residual image from the color distortion image to obtain the final corrected image.

[0010] In one possible implementation, in step one, the shooting conditions of several drone-captured images cover different light intensities and different shooting altitudes.

[0011] In one possible implementation, each color swatch consists of several color blocks, and the true color value of each color block is measured in advance using a colorimeter.

[0012] In step two, based on the true color value of each color block in several color swatches and the captured color value of the corresponding color block in the image taken by the drone, an independent color correction matrix for each color swatch is constructed using multiple linear regression technology. After weighting and aggregating all the independent color correction matrices, a weighted color correction matrix is ​​obtained.

[0013] In one possible implementation, in step two, the error term of each color swatch is converted into the weights of the corresponding independent color correction matrix.

[0014] The error term for each color swatch is defined as the sum of the squared differences between the true color value of each color patch in the swatch and the predicted color value of the independent color correction matrix.

[0015] In one possible implementation, in step three, the deep learning model employs a graph-to-graph regression deep convolutional neural network.

[0016] The graph-to-graph regression deep convolutional neural network comprises five modules: the first module is the input layer; the second module contains a convolutional layer and a ReLU nonlinear layer; the third module contains several repeating structures, each of which includes a convolutional layer, a ReLU nonlinear layer and a batch normalization layer; the fourth module contains a convolutional layer; and the fifth module is the regression layer.

[0017] In one possible implementation, in step three, all pooling layers are omitted before the convolutional layers of the graph-to-graph regression deep convolutional neural network, and a zero-padding operation is introduced.

[0018] In one possible implementation, in step three, the graph-to-graph regression deep convolutional neural network trains the residual mapping using a residual learning framework.

[0019] In one possible implementation, step four is followed by: Step 5: Independently evaluate the color correction effects of the weighted color correction matrix and the deep learning model, and assess the impact of different light intensities and shooting heights on the colors of images captured by the drone.

[0020] In one possible implementation, in step five, Euclidean distance is used to quantify the color error between the color correction estimate and the actual color value; the smaller the color error, the more accurate the color correction effect.

[0021] The two-stage color correction method for UAV images disclosed in this application has the following advantages: By constructing a weighted color correction matrix based on multiple color palettes, and using the standard color-corrected image obtained through the weighted color correction matrix as a benchmark, a deep learning model is constructed and trained. This two-stage approach improves the accuracy and efficiency of color correction.

[0022] By employing multiple linear regression to construct independent color correction matrices for each color swatch, and then weighting and aggregating all the independent color correction matrices to obtain a weighted color correction matrix, the standard color correction image is improved, providing a more reliable benchmark for training deep learning models.

[0023] By converting the error terms of each color swatch into the weights of the corresponding independent color correction matrices, the rationality of aggregating the independent color correction matrices into a weighted color correction matrix is ​​improved.

[0024] By omitting all pooling layers before the convolutional layers of a graph-to-graph regression deep convolutional neural network and introducing zero-padding, we ensure that the size of each feature map in the intermediate layers is consistent with the input image.

[0025] The proposed graph-to-graph regression deep convolutional neural network uses a residual learning framework to train residual mappings, which reduces the learning difficulty, avoids excessive modification, improves color correction accuracy, and reduces error accumulation. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a two-stage color correction method for UAV images provided in this application embodiment; Figure 2 The flowcharts for steps one and two provided in the embodiments of this application are shown below; Figure 3 The flowcharts for steps three and four provided in the embodiments of this application are shown below; Figure 4 This is a schematic diagram of the network architecture of a graph-to-graph regression deep convolutional neural network provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] like Figure 1 As shown, this application provides a two-stage color correction method for UAV images, including the following steps: Step 1: Use a drone to take aerial photos of the target area with several color swatches, and obtain several drone-captured images. All drone-captured images contain all the color swatches.

[0030] Step 2: Based on several color swatches, a weighted color correction matrix is ​​constructed using multiple linear regression technology. The weighted color correction matrix is ​​then used to perform standard color correction on the images captured by the UAV to obtain a standard color-corrected image.

[0031] Step 3: Using the standard color-corrected image as a reference, construct and train a deep learning model.

[0032] Step 4: Obtain the color distortion image to be corrected and input it into the trained deep learning model. Output the prediction residual image and subtract the prediction residual image from the color distortion image to obtain the final corrected image.

[0033] Specifically, in this embodiment, the target area, taking an experimental field as an example, is equipped with four color panels.

[0034] For example, in step one, the shooting conditions of several drone-captured images cover different light intensities and different shooting altitudes.

[0035] Specifically, in this embodiment, the drone flies at a constant speed of 2 meters per second, and the camera is always pointing vertically downwards (directly below) at the ground to take pictures. See Table 1 for specific parameters.

[0036] Table 1. Drone shooting parameters

[0037] To ensure comprehensive aerial coverage and seamless image stitching, both forward and backward overlap ratios were set to 75%. To improve color accuracy and consistency, the experimental field was equipped with custom-designed large four-color calibration boards, each measuring 1.2 x 1.8 meters. These calibration boards, inspired by the X-Rite color calibration card, are portable, durable, and offer accurate color reproduction. Their modular four-section design facilitates transportation and assembly, and a specially selected anti-glare matte surface avoids affecting color calibration. The calibration boards are spaced 50 meters apart, serving as a benchmark for color calibration across the entire area. Data acquisition was conducted three times, recording various environmental conditions ranging from sunny to cloudy. These dates were carefully selected to record characteristics of different growth stages of the crops. During each acquisition, the drone performed four flight missions between 9:00 AM and 6:00 PM to capture shadow variations under different lighting conditions. During each flight, the drone's altitude gradually increased from 25 meters to 120 meters, increasing by 5 meters each time. This variation was crucial for assessing the impact of altitude on color accuracy. Meanwhile, a professional light sensor (VBR-200 PAR light meter, VABIRA, China) was used to measure light intensity every 10 minutes. This helped to correlate lighting conditions with image color consistency and understand the impact of environmental factors on the color calibration process during UAV imaging. The original images had a resolution of 5472×3648 pixels and were saved in JPG format. A total of 184 orthophoto mosaics (i.e., images taken by the UAV) were generated in these tasks. For comprehensive analysis, the images were processed using Pix4Dmapper software to generate these orthophoto mosaics, with sizes ranging from 3969×2589 pixels to 15257×15098 pixels. All images contained four color palettes.

[0038] For example, each color swatch consists of several color blocks, and the true color value of each color block is measured in advance by a colorimeter.

[0039] In step two, based on the true color value of each color block in several color swatches and the captured color value of the corresponding color block in the image taken by the drone, an independent color correction matrix for each color swatch is constructed using multiple linear regression technology. After weighting and aggregating all the independent color correction matrices, a weighted color correction matrix is ​​obtained.

[0040] Specifically, in this embodiment, each color swatch consists of 24 color blocks, and the true color value of each color block is measured in advance using a colorimeter and recorded as follows: Let represent the true color value of the i-th color patch. The captured color value of a color patch in an image taken by a drone is denoted as . Let represent the captured color value of the i-th color patch of the j-th color swatch. For each color swatch in the image captured by the drone, an independent color correction matrix is ​​constructed using multiple linear regression. (i.e., the independent color correction matrix of the j-th color swatch), establishing a linear mapping relationship to convert the captured color values ​​into true color values. The linear mapping relationship is shown in the following formula: .

[0041] In the formula, It is an n×m dimensional matrix used to represent the captured color values ​​of images taken by a drone, where n represents the number of color patches and m is the number of color components in the Lab color space. The Lab color space, established by the International Commission on Illumination in 1976, contains three coordinate axes: L represents lightness, and a and b correspond to the color components from green to red and from blue to yellow, respectively. It is an n×k matrix, where k represents the number of color components measured by the colorimeter, and the intercept term is contained by adding a column of all 1s. This is how it is constructed. The matrix (k×m) serves as the coefficient matrix, while the residual matrix... This is an n×m matrix used to describe the residual error between the predicted values ​​and the true color values ​​of the independent color correction matrix. The optimal [value] can be derived through the normal equation. matrix:

[0042] For example, in step two, the error terms of each color swatch are converted into weights of the corresponding independent color correction matrix.

[0043] The error term for each color swatch is defined as the sum of the squared differences between the true color value of each color patch in the swatch and the predicted color value of the independent color correction matrix.

[0044] Specifically, in this embodiment, the error term of the j-th color plate As shown in the following formula: .

[0045] The error terms of each color swatch are converted into weights for a corresponding independent color correction matrix. Then normalize:

[0046] Weighted color correction matrix The calculation is as follows: .

[0047] Then the weighted color correction matrix This is applied to images captured by a drone to obtain a standard color-corrected image. The color values ​​of the standard color-corrected image are denoted as... As shown in the following formula: .

[0048] This weighted aggregation method ensures that the final weighted color correction matrix prioritizes the color palette with the smallest error for adjustment, thereby effectively improving the accuracy of the entire color correction process. Once the weighted color correction matrix is ​​determined, it can be applied to images captured by the drone to obtain standard color-corrected images.

[0049] like Figure 2 The diagram shown is a flowchart of steps one through two. Figure 2 In this diagram, image acquisition corresponds to step one; color distortion corresponds to the captured color values ​​of the drone-captured image; true color corresponds to the true color values ​​measured by the colorimeter; the color calibration matrix corresponds to the derivation and application of the independent color correction matrix and the weighted color correction matrix; the true value corresponds to the color values ​​of the standard color-corrected image; and the residual image is the residual obtained by subtracting the standard color-corrected image from the drone-captured image. The residual image is denoted as... .

[0050] For example, in step three, the deep learning model employs a graph-to-graph regression deep convolutional neural network.

[0051] The graph-to-graph regression deep convolutional neural network comprises five modules: the first module is the input layer; the second module contains a convolutional layer and a ReLU nonlinear layer; the third module contains several repeating structures, each of which includes a convolutional layer, a ReLU nonlinear layer and a batch normalization layer; the fourth module contains a convolutional layer; and the fifth module is the regression layer.

[0052] For example, in step three, all pooling layers are omitted before the convolutional layers of the graph-to-graph regression deep convolutional neural network, and a zero-padding operation is introduced.

[0053] For example, in step three, the graph-to-graph regression deep convolutional neural network uses a residual learning framework to train the residual mapping.

[0054] Specifically, in this embodiment, the input layer of the graph-to-graph regression deep convolutional neural network is used to receive color-distorted images. It is a real image. With residual image The synthesized result. Unlike conventional image-to-image regression tasks, which aim to learn a mapping function... The graph-to-graph regression deep convolutional neural network is used to predict the panchromatic calibration image. It employs a residual learning framework to train the residual mapping, outputting a predicted residual image. The final corrected image is obtained by subtracting the predicted residual image from the color-distorted image. The second module contains a 3×3×3 64-filter convolutional layer (generating 64 feature maps) and a ReLU nonlinear layer. The third module includes 20 repeating structures, each consisting of a 3×3×3 64-filter convolutional layer, a ReLU nonlinear layer, and a batch normalization layer. The fourth module contains a 3×3×64 3-filter convolutional layer for output reconstruction. The training process of the graph-to-graph regression deep convolutional neural network is an end-to-end process, where the loss function is defined as the mean squared error (MSE) to optimize the trainable parameters, representing a function that measures the difference between the target residual image and the image estimated from the color-distorted image. The loss function is shown below: .

[0055] In the formula, This loss function first calculates the difference between the "network prediction residual" and the "target residual" for each of the N color-distorted image samples. It is the network's response to the i-th distorted image. The output prediction residual, This is the target residual corresponding to the sample (from the distorted image). With real images (The difference is obtained by subtraction); then the difference between the two residuals is quantified by the square of the Euclidean norm; finally, the difference results of the N samples are summed and multiplied by... Normalize.

[0056] In this embodiment, the entire training process of the graph-to-graph regression deep convolutional neural network consists of 30 epochs, with each epoch requiring a complete traversal of the entire dataset (i.e., a dataset composed of standard color-corrected images selected according to a preset threshold). To enhance training diversity and reduce biases that may be caused by the order of the data sequence, the image samples are randomly shuffled at the beginning of each epoch. The mini-batch training size is set to 32, which is the maximum achievable under current computing resources. A stochastic gradient descent algorithm with a momentum coefficient of 0.9 is used for parameter optimization, and the learning rate is kept constant at 0.001 throughout the training process.

[0057] like Figure 3 The diagram shown is a flowchart of steps three to four. Figure 3 In this context, "sorted images" refers to images corrected according to standard color, "training dataset" refers to a dataset composed of standard color-corrected images selected according to a preset threshold, "convolutional neural network" refers to a graph-to-graph regression deep convolutional neural network, "estimated residual image" refers to the predicted residual image output by the graph-to-graph regression deep convolutional neural network, and "stitched" refers to stitching together the predicted residual images and then using a formula... Color correction was performed on the rice paddy images, and finally the Lab space and error baseline were determined.

[0058] Figure 4 This is a schematic diagram of the network architecture of a graph-to-graph regression deep convolutional neural network.

[0059] For example, step four is followed by: Step 5: Independently evaluate the color correction effects of the weighted color correction matrix and the deep learning model, and assess the impact of different light intensities and shooting heights on the colors of images captured by the drone.

[0060] Specifically, in this embodiment, to improve color calibration performance, the color calibration effects of the weighted color calibration matrix and the deep learning model are independently evaluated. First, the impact of the benchmark dataset generated by multiple linear regression is studied by comparing the weighted color calibration matrix generated based on a multi-color palette with the traditional monochrome palette calibration matrix. Then, the graph-to-graph regression deep convolutional neural network is evaluated from two dimensions: first, the role of the residual learning model in improving color calibration accuracy and efficiency; second, the output image quality and structural integrity are examined, with a focus on analyzing the advantages of introducing zero-padding technology in the network design. Simultaneously, the influence of flight altitude and day / night lighting conditions on the color of images captured by the UAV is investigated.

[0061] For example, in step five, Euclidean distance is used to quantify the color error between the color correction estimate and the actual color value. The smaller the color error, the more accurate the color correction effect.

[0062] Specifically, in this embodiment, the formula for color error is as follows: .

[0063] In the formula, "Colorerror" represents the color error value. First, for each of the n samples, the Euclidean distance between the "estimated color" and the "actual color" in the Lab color space is calculated. , , These are estimated color values. , , The actual color value is obtained by measuring the true color value with a colorimeter. The square root of the sum of the squares of the differences between the corresponding channels is the color difference of a single sample. Then, the difference values ​​of n samples are summed, and finally, the average value is obtained by dividing by n to get the overall color error.

[0064] Experimental results show that the color error of the two-stage color correction method for UAV images proposed in this application is only 1.52%.

[0065] This application embodiment constructs a weighted color correction matrix based on multiple color palettes, and uses the standard color-corrected image obtained by the weighted color correction matrix as a benchmark to construct and train a deep learning model. This two-stage approach improves color correction accuracy and work efficiency.

[0066] By employing multiple linear regression to construct independent color correction matrices for each color swatch, and then weighting and aggregating all the independent color correction matrices to obtain a weighted color correction matrix, the standard color correction image is improved, providing a more reliable benchmark for training deep learning models.

[0067] By converting the error terms of each color swatch into the weights of the corresponding independent color correction matrices, the rationality of aggregating the independent color correction matrices into a weighted color correction matrix is ​​improved.

[0068] By omitting all pooling layers before the convolutional layers of a graph-to-graph regression deep convolutional neural network and introducing zero-padding, we ensure that the size of each feature map in the intermediate layers is consistent with the input image.

[0069] The proposed graph-to-graph regression deep convolutional neural network uses a residual learning framework to train residual mappings, which reduces the learning difficulty, avoids excessive modification, improves color correction accuracy, and reduces error accumulation.

[0070] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0071] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A two-stage color correction method for UAV images, characterized in that, Includes the following steps: Step 1: Use a drone to take aerial photos of the target area with several color swatches, and obtain several drone-captured images. All drone-captured images contain all the color swatches. Step 2: Based on several color swatches, a weighted color correction matrix is ​​constructed using multiple linear regression technology. The weighted color correction matrix is ​​then used to perform standard color correction on the images captured by the UAV to obtain a standard color-corrected image. Step 3: Using the standard color-corrected image as a reference, construct and train a deep learning model; Step 4: Obtain the color distortion image to be corrected and input it into the trained deep learning model. Output the prediction residual image and subtract the prediction residual image from the color distortion image to obtain the final corrected image.

2. The method for dual-stage color correction of UAV images according to claim 1, characterized in that, In step one, the shooting conditions of several drone images cover different light intensities and different shooting altitudes.

3. The method for dual-stage color correction of UAV images according to claim 1, characterized in that, Each color swatch is composed of several color blocks, and the true color value of each color block is measured in advance using a colorimeter; In step two, based on the true color value of each color block in several color swatches and the captured color value of the corresponding color block in the image taken by the drone, an independent color correction matrix for each color swatch is constructed using multiple linear regression technology. After weighting and aggregating all the independent color correction matrices, a weighted color correction matrix is ​​obtained.

4. The method for dual-stage color correction of UAV images according to claim 3, characterized in that, In step two, the error terms of each color swatch are converted into the weights of the corresponding independent color correction matrix; The error term for each color swatch is defined as the sum of the squared differences between the true color value of each color patch in the swatch and the predicted color value of the independent color correction matrix.

5. The method for dual-stage color correction of UAV images according to claim 1, characterized in that, In step three, the deep learning model employs a graph-to-graph regression deep convolutional neural network; The graph-to-graph regression deep convolutional neural network comprises five modules: the first module is the input layer; The second module contains a convolutional layer and a ReLU nonlinear layer; the third module contains several repeating structures, each of which includes a convolutional layer, a ReLU nonlinear layer and a batch normalization layer; the fourth module contains a convolutional layer; and the fifth module is a regression layer.

6. The method for dual-stage color correction of UAV images according to claim 5, characterized in that, In step three, all pooling layers are omitted before the convolutional layers of the graph-to-graph regression deep convolutional neural network, and zero-padding is introduced.

7. The method for dual-stage color correction of UAV images according to claim 5, characterized in that, In step three, the graph-to-graph regression deep convolutional neural network uses a residual learning framework to train the residual mapping.

8. The method for dual-stage color correction of UAV images according to claim 1, characterized in that, Step four also includes: Step 5: Independently evaluate the color correction effects of the weighted color correction matrix and the deep learning model, and assess the impact of different light intensities and shooting heights on the colors of images captured by the drone.

9. A two-stage color correction method for UAV images according to claim 8, characterized in that, In step five, Euclidean distance is used to quantify the color error between the color correction estimate and the actual color value. The smaller the color error, the more accurate the color correction effect.

Images