Infrared remote sensing image restoration method, device, electronic device and storage medium

Abstract

The present invention provides an infrared remote sensing image restoration method, device, electronic device, and storage medium, relating to the field of image processing technology. The method comprises: obtaining a target observation image; determining a constraint term for removing stripe noise based on the differential curvature value of each pixel in an ideal image, wherein the differential curvature value of the stripe noise contaminated area in the ideal image is greater than the differential curvature value of the non-strip noise contaminated area; constructing a denoising energy functional model based on the constraint term; inputting the target observation image into the denoising energy functional model for iterative processing, and determining the resulting ideal image as the restored image with stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration constraint condition is satisfied. The present invention can effectively remove stripe noise caused by solar pollution, and during the stripe noise removal process, the original image radiation information remains unchanged.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared remote sensing image restoration method, device, electronic equipment and storage medium. Background Art

[0002] The quality of remote sensing images is a key factor affecting their application areas and accuracy. However, image degradation often occurs during the imaging process due to environmental, system, and human factors, limiting their application. The Fengyun-3C (FY-3C) is my country's new-generation polar-orbiting meteorological satellite. Its Visible and Infra-Red Radiometer (VIRR) is one of its primary payloads, comprising 10 spectral channels, three of which are infrared. The FY-3C satellite is primarily used to monitor global cloud cover, identify cloud height, type, and phase, detect ocean surface temperature, monitor vegetation growth and type, monitor high-temperature hotspots, identify surface snow cover, and detect ocean color.

[0003] In the VIRR infrared channel, due to the direct sunlight entering the sensor scanning mirror at the junction of dawn and dusk, and the abnormal clamping voltage caused by stray light pollution of cold air, the third channel (3.74μm) produced obvious stripe noise pollution, affecting the application of subsequent imaging products.

[0004] In the prior art, methods for removing stripe noise mainly include the following three categories:

[0005] (1) Digital filtering: Although this method can remove or reduce correlated noise, it often suffers from problems such as large computational complexity, cumbersome steps, poor processing stability, and low efficiency. Furthermore, for surfaces with complex distribution of objects, filtering can easily lead to loss of image texture, edge details, and other detailed information, resulting in blurred stripe removal results. More importantly, it is difficult to effectively maintain the quantitative radiometric information of the image before and after processing.

[0006] (2) Methods such as histogram matching and moment matching based on the statistical characteristics of image grayscale information. However, this method mainly targets the stripe noise caused by multi-detector parallel scanning and is not applicable to the unit imaging and random stripe problem of VIRR.

[0007] (3) A method based on variational and partial differential equations combines the directional characteristics of stripe noise with a variational algorithm for stripe removal, making stripe noise removal more efficient and accurate. However, while removing stripe noise, this method also blurs the details of the image, affecting the stripe noise removal results. Summary of the Invention

[0008] The present invention provides an infrared remote sensing image restoration method, device, electronic device and storage medium to address the defects in the prior art that are incompatible with removing stripe noise and maintaining detail features, effectively remove stripe noise caused by solar pollution, and keep the original image radiation information unchanged during the stripe noise removal process.

[0009] The present invention provides an infrared remote sensing image restoration method, comprising:

[0010] Acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise;

[0011] determining a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area;

[0012] Based on the constraint term, a denoising energy functional model is constructed, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detail feature information of the target observation image;

[0013] The target observation image is input into the denoising energy functional model for iterative processing, and the ideal image finally obtained is determined as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limitation condition is met.

[0014] According to the infrared remote sensing image restoration method provided by the present invention, determining the constraint item for removing the stripe noise based on the differential curvature value of each pixel in the ideal image includes:

[0015] Determine the regularization parameter based on the ratio of the regularization parameter to the adjustment coefficient in the previous iteration cycle, wherein the initial regularization parameter is the ratio of the L2 norm of the ideal image perpendicular to the stripe noise gradient direction and along the gradient direction;

[0016] constructing a weight matrix based on the differential curvature value of each pixel in the ideal image, wherein the weight matrix is ​​used to adjust the regularization parameter of the stripe noise in different intensity regions;

[0017] Constructing a regularization constraint term based on the gradient of the ideal image perpendicular to the stripe noise gradient direction;

[0018] The constraint term is constructed based on the regularization parameter, the weight matrix and the regular constraint term.

[0019] According to the infrared remote sensing image restoration method provided by the present invention, constructing a weight matrix based on the differential curvature value of each pixel in the ideal image includes:

[0020] In a current iteration cycle, respectively determining a pixel first-order partial derivative, a pixel second-order partial derivative, and a pixel second-order mixed partial derivative of each pixel in the ideal image;

[0021] Based on the pixel first-order partial derivative, the pixel second-order partial derivative, and the pixel second-order mixed partial derivative, respectively determining the horizontal second-order partial derivative of the ideal image along the stripe noise gradient direction and the vertical second-order partial derivative perpendicular to the gradient direction;

[0022] The weight matrix is ​​constructed based on the horizontal second-order partial derivatives and the vertical second-order partial derivatives.

[0023] According to the infrared remote sensing image restoration method provided by the present invention, the target observation image is input into the denoising energy functional model for iterative processing, and the ideal image finally obtained is determined as the restoration image with the stripe noise removed, comprising:

[0024] Constructing the Euler-Lagrange equation corresponding to the denoising energy functional model;

[0025] Determine a discretized Euler-Lagrange equation based on the target observation image, the pixel values ​​of the pixel points corresponding to the ideal image output in the previous iteration cycle, and the Euler-Lagrange equation;

[0026] Determining an ideal image outputted in a current iteration cycle based on the discretized Euler-Lagrange equation;

[0027] Using the ideal image outputted in the current iteration cycle as the input image for the next iteration cycle, and repeatedly outputting the ideal image of the corresponding iteration cycle;

[0028] When the limiting conditions are met, the iteration is terminated, and the ideal image finally outputted is determined as the repaired image.

[0029] According to the infrared remote sensing image restoration method provided by the present invention, the limiting conditions include that the number of iterations is greater than a first threshold or the L2 norm value of the ideal image difference output from two adjacent iteration cycles is less than a second threshold.

[0030] The infrared remote sensing image restoration method provided by the present invention further includes:

[0031] Based on the evaluation index, the stripe noise restoration effect of the target observation image and the corresponding restoration image is quantitatively evaluated, wherein the evaluation index includes: at least one of: a row mean curve, a radiation quality improvement factor and an inverse variation coefficient, the row mean curve is used to characterize the average value of pixels in each row of the image before and after the stripe noise removal, the radiation quality improvement factor is used to characterize the change information of the directional distribution of the stripe noise before and after the stripe noise removal, and the inverse variation coefficient is used to characterize the degree of variation of the image before and after the stripe noise removal.

[0032] The present invention also provides an infrared remote sensing image restoration device, comprising:

[0033] An acquisition module is used to acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise;

[0034] a determination module, configured to determine a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area;

[0035] A model construction module is used to construct a denoising energy functional model based on the constraint term, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain the detail feature information of the target observation image;

[0036] An output module is used to input the target observation image into the denoising energy functional model for iterative processing, and determine the final ideal image as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output by the previous iteration is input into the denoising energy functional model until the iteration limit condition is met.

[0037] The infrared remote sensing image restoration device provided by the present invention further includes:

[0038] An evaluation module is used to quantitatively evaluate the stripe noise restoration effects of the target observation image and the corresponding restoration image based on evaluation indicators.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-described infrared remote sensing image restoration methods is implemented.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the infrared remote sensing image restoration method described above is implemented.

[0041] The infrared remote sensing image restoration method, device, electronic device and storage medium provided by the present invention construct a denoising energy functional model through constraint terms and fidelity terms, and combine the anisotropy of stripe noise and the total variation denoising model. The constraint term is used to remove stripe noise, and the fidelity term is used to maintain detail feature information. In addition, based on the fact that the differential curvature value of the stripe noise contaminated area in the ideal image is greater than the differential curvature value of the non-strip noise contaminated area, the stripe noise contaminated area and the non-strip noise contaminated area in the input image are better distinguished, thereby improving the stripe noise removal effect. The constraint term and the fidelity term are balanced by a regularization parameter, thereby further improving the stripe noise removal accuracy and the restoration image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is one of the flow charts of the infrared remote sensing image restoration method provided by the present invention;

[0044] Figure 2 This is the second flow chart of the infrared remote sensing image restoration method provided by the present invention;

[0045] Figure 3 is a first example original observation image provided by the present invention;

[0046] Figure 4 is a first example repaired image provided by the present invention;

[0047] Figure 5 is a schematic diagram of a row mean curve of a first example restoration image provided by the present invention;

[0048] Figure 6 is a schematic diagram of the change along the gradient direction of the second example original observation image provided by the present invention;

[0049] Figure 7 2 is a schematic diagram of vertical gradient direction changes of a second example original observation image provided by the present invention;

[0050] Figure 8 is a schematic diagram of changes along the gradient direction of the second example restoration image provided by the present invention;

[0051] Figure 9 2 is a schematic diagram of vertical gradient direction changes of a second example restoration image provided by the present invention;

[0052] Figure 10 It is a structural schematic diagram of the infrared remote sensing image restoration device provided by the present invention;

[0053] Figure 11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0055] The quality of remote sensing images is a key factor affecting their application areas and accuracy. However, image degradation often occurs during the imaging process due to environmental, system, and human factors, limiting their application. The Fengyun-3C (FY-3C) is my country's new-generation polar-orbiting meteorological satellite. Its primary mission is to monitor global cloud cover, identify cloud height, type, and phase, detect ocean surface temperature, monitor vegetation growth and type, monitor high-temperature hotspots, identify surface snow cover, and detect ocean color. The Visible and Infra-Red Radiometer (VIRR), one of the FY-3C's primary payloads, utilizes optical scanning imaging and includes 10 spectral channels, three of which are infrared channels: infrared channels 3, 4, and 5. The wavelengths of these three channels are shown in Table 1.

[0056] Table 1 Wavelengths of the FY-3 VIRR infrared channel

[0057]

[0058] In the VIRR infrared channel, due to the direct sunlight entering the sensor scanning mirror at the junction of dawn and dusk, and the abnormal clamping voltage caused by stray light pollution and cold air, the infrared channel produces obvious stripe noise pollution. Among them, the stripe noise in the third channel (3.74μm) is the most serious, affecting the application of subsequent imaging products.

[0059] In the prior art, methods for removing stripe noise mainly include the following three categories:

[0060] (1) Digital filtering: Although this method can remove or reduce correlated noise, it often suffers from problems such as large computational complexity, cumbersome steps, poor processing stability, and low efficiency. Furthermore, for surfaces with complex distribution of objects, filtering can easily lead to loss of image texture, edge details, and other detailed information, resulting in blurred stripe removal results. More importantly, it is difficult to effectively maintain the quantitative radiometric information of the image before and after processing.

[0061] (2) Methods such as histogram matching and moment matching based on the statistical characteristics of image grayscale information. However, this method mainly targets the stripe noise caused by multi-detector parallel scanning and is not applicable to the unit imaging and random stripe problem of VIRR.

[0062] (3) A method based on variational and partial differential equations combines the directional characteristics of stripe noise with a variational algorithm for stripe removal, making stripe noise removal more efficient and accurate. However, while removing stripe noise, this method also blurs the details of the image, affecting the stripe noise removal results.

[0063] In view of the above problems, the present invention provides an infrared remote sensing image restoration method. Figure 1 This is one of the flow charts of the infrared remote sensing image restoration method provided by the present invention, such as Figure 1 As shown, the method includes:

[0064] Step 110: Acquire a target observation image, where the target observation image includes an ideal image and stripe noise;

[0065] Step 120: determining a constraint item for removing the stripe noise based on the differential curvature value of each pixel in the ideal image, wherein the differential curvature value of the stripe noise contaminated area in the ideal image is greater than the differential curvature value of the non-strip noise contaminated area;

[0066] Step 130: constructing a denoising energy functional model based on the constraint term, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, and the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detail feature information of the target observation image;

[0067] Step 140: Input the target observation image into the denoising energy functional model for iterative processing, and determine the final ideal image as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limit condition is met.

[0068] Specifically, because existing methods cannot remove stripe noise while maintaining the detailed feature information of the original image, the embodiments of the present invention remove stripe noise by constructing a constraint term and maintain the detailed feature information in the original image through a fidelity term. The fidelity term and the constraint term are balanced by a regularization parameter to avoid excessive loss of image detail feature information due to an excessively large regularization parameter, or residual noise due to an excessively small regularization parameter, which can affect the image restoration quality. Subsequently, a denoising energy functional model is constructed using the constraint term and the fidelity term. The target observation image is input into the denoising energy functional model for multiple iterative calculations, ultimately outputting a restored image with stripe noise removed. This repairs the stripe noise caused by solar pollution, improves the usability of the restored image, and facilitates subsequent image applications.

[0069] Optionally, the stripe noise can be regarded as additive noise, and the target observation image can be regarded as the sum of the ideal image and the stripe noise, that is, f(x,y)=u(x,y)+s(x,y), where f(x,y) represents the target observation image, u(x,y) represents the ideal image, and s(x,y) represents the stripe noise.

[0070] Optionally, since an iterative solution is required when removing stripe noise from the target observation image and the ideal image is unknown, when the stripe noise is regarded as additive noise, the initial value of the ideal image can be set as the target observation image f(x, y).

[0071] Optionally, the stripe noise has anisotropy, that is, the stripe noise has obvious directional characteristics, and the gradient of the image along the stripe noise direction is much smaller than the gradient of the image in the direction perpendicular to the stripe noise. The direction along the stripe noise is regarded as the x-axis and the direction perpendicular to the stripe noise is regarded as the y-axis. The anisotropy of the stripe noise can be expressed as: Where s represents stripe noise, represents the gradient operator along the direction of the stripe noise, Represents the gradient operator perpendicular to the direction of the stripe noise.

[0072] Optionally, determining a constraint item for removing the stripe noise based on the differential curvature value of each pixel in the ideal image includes:

[0073] Determine the regularization parameter based on the ratio of the regularization parameter to the adjustment coefficient in the previous iteration cycle, wherein the initial regularization parameter is the ratio of the L2 norm of the ideal image perpendicular to the stripe noise gradient direction and along the gradient direction;

[0074] constructing a weight matrix based on the differential curvature value of each pixel in the ideal image, wherein the weight matrix is ​​used to adjust the regularization parameter of the stripe noise in different intensity regions;

[0075] Constructing a regularization constraint term based on the gradient of the ideal image perpendicular to the stripe noise gradient direction;

[0076] The constraint term is constructed based on the regularization parameter, the weight matrix and the regular constraint term.

[0077] Specifically, in order to effectively remove stripe noise, in an embodiment of the present invention, on the basis of considering stripe noise as additive noise, regularization parameters, weight matrices and regularization constraint terms are respectively constructed, and constraint terms are constructed based on the above three terms. While improving the stripe noise removal effect, the constraint terms and fidelity terms are balanced, so that the target observation image still retains the same detailed feature information such as texture and edges as the original image after removing the stripe noise.

[0078] Optionally, the regularization parameter is a dynamically changing parameter, that is, as the number of iterations increases, the stripe noise pollution is gradually repaired, and the regularization parameter in the current iteration cycle needs to be dynamically adjusted based on the ideal image output in the previous iteration cycle to obtain the regularization parameter in the current iteration cycle, so as to avoid noise residue or excessive penalty in the current iteration cycle. Among them, the initial regularization parameter is the L2 norm ratio of the ideal image perpendicular to the stripe noise gradient direction and along the gradient direction. The initial regularization parameter λ0 is shown in formula (1), which is:

[0079]

[0080] Among them, u x represents the gradient of the ideal image along the direction of stripe noise, u y represents the gradient of the ideal image perpendicular to the direction of the stripe noise, ||·||2 represents the L2 norm, and k represents the number of iterations.

[0081] During the iteration process, the regularization parameter is shown in formula (2), which is:

[0082]

[0083] Where M represents the adjustment coefficient.

[0084] Optionally, constructing a weight matrix based on the differential curvature value of each pixel in the ideal image includes:

[0085] Determining the first-order partial derivative, second-order partial derivative, and second-order mixed partial derivative of each pixel in the corresponding ideal image respectively;

[0086] Based on the pixel first-order partial derivative, the pixel second-order partial derivative, and the pixel second-order mixed partial derivative, respectively determine the horizontal second-order partial derivative of the corresponding ideal image along the stripe noise gradient direction and the vertical second-order partial derivative perpendicular to the gradient direction;

[0087] The weight matrix is ​​constructed based on the horizontal second-order partial derivatives and the vertical second-order partial derivatives.

[0088] Specifically, in order to better distinguish the stripe noise pollution area from the non-strip noise pollution area, the weight matrix is ​​constructed by taking advantage of the fact that the differential curvature value in the stripe noise pollution area is larger and the differential curvature value in the non-strip noise pollution area is smaller. The weight matrix is ​​shown in formula (3):

[0089] W=||u ηη |-|u εε ||

[0090] Among them, |·| represents the absolute value operator, u ηη Represents the horizontal second-order partial derivative of the ideal image along the stripe noise gradient direction, u εε represents the vertical second-order partial derivative perpendicular to the gradient direction, u ηη As shown in formula (4), u εε As shown in formula (5), formula (4) is:

[0091]

[0092] Formula (5) is:

[0093]

[0094] Among them, u x and u y Both represent the first-order partial derivatives of pixels in the ideal image, u xx and u yy Both represent the second-order partial derivatives of pixels in the ideal image, u xy Represents the second-order mixed partial derivative of the pixel in the ideal image.

[0095] Optionally, based on the product of the above regularization parameter, weight matrix and regularization constraint term, the constraint term is constructed as While better distinguishing between stripe noise contaminated areas and non-strip noise contaminated areas, the noise removal accuracy is improved and noise residue is avoided.

[0096] Optionally, based on the above constraints and combined with the anisotropy of the stripe noise, a denoising energy functional model is constructed as shown in formula (6), which is:

[0097]

[0098] Where u represents the ideal image, f represents the observed image, It represents the fidelity term, which maintains the detailed feature information such as texture and edge in the process of stripe noise removal. The stripe noise removal problem can be transformed into a problem of solving the extreme value problem in the denoising energy functional model.

[0099] Alternatively, if the denoising energy functional model is E(u)=∫ Ω F(x,y,u,u x ,u y )dΩ, then solving the extreme value problem of E(u) is to solve F(x,y,u,u x ,u y )=0, then, the constructed denoising energy functional model is transformed according to the above steps, and E(u) is shown in formula (7), which is:

[0100]

[0101] When the extreme value of E(u) is 0, that is, when E'(u)=0, equation (7) can be transformed into equation (8), which is:

[0102] F(x,y,u,u x ,u y )=|u x -f x | 2 +λW|u y | 2

[0103] Optionally, inputting the target observation image into the denoising energy functional model for iterative processing, and determining the ultimately obtained ideal image as the repaired image with the stripe noise removed, includes:

[0104] Constructing the Euler-Lagrange equation corresponding to the denoising energy functional model;

[0105] Determine a discretized Euler-Lagrange equation based on the target observation image, the pixel values ​​of the pixel points corresponding to the ideal image output in the previous iteration cycle, and the Euler-Lagrange equation;

[0106] Determining an ideal image output in a current iteration cycle based on the discretized Euler-Lagrange equation;

[0107] Using the ideal image outputted in the current iteration cycle as the input image for the next iteration cycle, and repeatedly outputting the ideal image of the corresponding iteration cycle;

[0108] When the limiting conditions are met, the iteration is terminated, and the ideal image finally outputted is determined as the repaired image.

[0109] Specifically, after constructing the denoising energy functional model, the problem of removing stripe noise can be converted into a problem of solving the extreme value of the denoising energy functional model, and the extreme value problem of the denoising energy functional model can be solved by constructing the Euler-Lagrange equation corresponding to the denoising energy functional model. Specifically, when solving, the denoising energy functional model can be discretized, and the discretized Euler-Lagrange equation is constructed by using the pixel values ​​of the target observation image and the ideal image output in the previous iteration cycle to determine the value of each pixel in the current iteration cycle to remove the stripe noise, and form an ideal image for output. The input image of the next iteration cycle of the ideal image output in the current iteration cycle is continuously input into the denoising energy functional model for iterative calculation until the limited conditions are met to obtain a repaired image that meets the requirements.

[0110] Optionally, according to formula (8), in an embodiment of the present invention, the Euler-Lagrange equation is as shown in formula (9), which is:

[0111]

[0112] Among them, f x Represents the first-order derivative of the target observation image along the direction of the stripe noise.

[0113] Formula (9) can be further transformed into formula (10), which is:

[0114] u xx -f xx +λWu yy =0

[0115] Among them, f xx Represents the second-order gradient of the target observation image along the direction of the stripe noise.

[0116] Optionally, the denoising energy functional model is discretized to construct the relationship between the partial derivative and the pixel value, which is convenient for subsequent iterative solution. The first-order partial derivative of the pixel of the ideal image u x and u y As shown in formula (11), formula (11) is:

[0117]

[0118] Where i represents the number of columns of the ideal image, j represents the number of rows of the ideal image, M represents the total number of columns of pixels in the ideal image, N represents the total number of rows of pixels in the ideal image, and u represents the total number of columns of pixels in the ideal image. i,j represents the pixel value of the i-th column and j-th row in the ideal image, for u x The discretized form of the ideal image pixel first-order partial derivative u xIt can be expressed as the difference between the pixel value of the i+1th column and the jth row in the ideal image and the pixel value of the ith column and the jth row. for u y The discretized form of .

[0119] The second-order partial derivative of the pixel of the ideal image u xx 、u yy and pixel second-order mixed partial derivative u xy As shown in formula (12), formula (12) is:

[0120]

[0121] in, for u xx The discretized form of for u yy The discretized form of for u xy The discretized form of .

[0122] Substituting Equations (11) and (12) into Equation (10), we construct the discretized Euler-Lagrange equation. The discretized form of Equation (10) is shown in Equation (13), which is:

[0123] u i+1,j -2u i,j +u i-1,j -(f i+1,j -2f i,j +f i-1,j )+λW(u i,j+1 -2u i,j +u i,j-1 )=0

[0124] Optionally, based on the discretized Euler-Lagrange equation, during the iteration process, the Gauss-Seidel iteration method is used to solve the pixel values ​​of the ideal image of the current iteration cycle through the pixel values ​​of the ideal image output in the previous iteration cycle and the pixel values ​​of the target observation image. The iteration result is shown in formula (14):

[0125]

[0126] Here, k represents the number of iterations.

[0127] It should be noted that, since there are no four adjacent and symmetrical pixels centered on the pixel point at the ideal image boundary and located in the horizontal and vertical directions of the pixel point, the image gradient calculation must satisfy the Neumann boundary condition at the boundary. The Neumann boundary condition is shown in formula (15), which is:

[0128]

[0129] Optionally, the limiting condition includes that the number of iterations is greater than a first threshold or the L2 norm value of the ideal image difference output from two adjacent iteration cycles is less than a second threshold.

[0130] Optionally, if the specified conditions are met, the stripe noise of the target observation image can be considered to have been removed. The final output restoration image can use the same image format as the target observation image, such as HDF format.

[0131] Optionally, Figure 2 This is the second flow chart of the infrared remote sensing image restoration method provided by the present invention, such as Figure 2 As shown, the present invention can also perform batch restoration of target observation images when there are a large number of target observation images. When performing batch restoration of target observation images, the key is to determine the appropriate initial regularization parameter according to formula (1) based on different target observation images. By determining the initial regularization parameter λ0, the initial ideal image u0, the second threshold ε0 of the iteration limit condition and the number of iterations k for each target observation image, the output result of each iteration can be solved by formula (14), and based on the comparison result with the second threshold ε0, it is determined whether the iterative calculation is terminated. If it is terminated, the final restoration image is output. If the limit condition is not met, the ideal image output by the current iteration cycle is used as the input image of the next iteration cycle, and the regularization parameter is adjusted according to formula (2) before the iterative calculation is performed again.

[0132] Optionally, after outputting the restored image, in addition to subjectively evaluating the restoration effect by visually comparing the target observation image and the restored image, the restoration effect of the restored image can also be quantitatively evaluated by determining evaluation indicators of the restored image, specifically including:

[0133] Based on the evaluation index, the stripe noise restoration effect of the target observation image and the corresponding restoration image is quantitatively evaluated, wherein the evaluation index includes: at least one of: a row mean curve, a radiation quality improvement factor and an inverse variation coefficient, the row mean curve is used to characterize the average value of pixels in each row of the image before and after the stripe noise removal, the radiation quality improvement factor is used to characterize the change information of the directional distribution of the stripe noise before and after the stripe noise removal, and the inverse variation coefficient is used to characterize the degree of variation of the image before and after the stripe noise removal.

[0134] Optionally, a larger value of the Improvement Factors of Radiometric Quality (IF) indicates a better image stripe noise removal effect. Conversely, a smaller value of the Improvement Factors of Radiometric Quality indicates a worse image stripe noise removal effect. The IF value is shown in formula (16), which is:

[0135]

[0136] in, and represents the process parameters, and as shown in formula (17), formula (17) is:

[0137]

[0138] in, and denote the j-th row averages of the original observed image and the repaired image, respectively.

[0139] Alternatively, it is the ratio of the mean value to the standard deviation of the study area. The inverse coefficient of variation (ICV) is usually calculated in a uniform area, and the inverse coefficient of variation (ICV) is shown in formula (18), which is:

[0140]

[0141] Among them, R m represents the mean of the selected area, R s It represents the standard deviation of the selected area. The larger the inverse coefficient of variation value, the better the stripe noise removal effect is. Conversely, the smaller the value, the worse the stripe noise removal effect is.

[0142] For example, Figure 3 is the first example original observation image provided by the present invention, Figure 4 is the first example repaired image provided by the present invention, such as Figure 3-Figure 4 As shown, taking the intercepted image of the 3rd channel of VIRR infrared channel as an example, Figure 3 and Figure 4 They are the original observed image and the repaired image respectively. From the subjective comparison, Figure 4 Obviously repaired Figure 3 There is severe banding noise in Figure 5 is a schematic diagram of a row mean curve of the first example restoration image provided by the present invention, such as Figure 5 As shown in the figure, the row mean curve of the original observed image has sharp burrs and large fluctuations due to the existence of stripe noise, while the row mean curve of the repaired image is relatively smooth, and the sharp burrs are removed, that is, the stripe noise is removed, so that the repaired image has a better visual effect and retains the information of the original observed image to the greatest extent in terms of detail features.

[0143] For example, Figure 6 is a schematic diagram of the change along the gradient direction of the second example original observation image provided by the present invention, Figure 7 : is a schematic diagram of the vertical gradient direction change of the second example original observation image provided by the present invention, such as Figure 6-Figure 7As shown in Figure 2, there is obvious stripe noise in the vertical gradient direction of the original observation image, but there is almost no stripe noise along the gradient direction. Figure 8 is a schematic diagram of the change along the gradient direction of the second example restoration image provided by the present invention, Figure 9 : is a schematic diagram of the vertical gradient direction change of the second example restoration image provided by the present invention, such as Figure 8-Figure 9 As shown in Figure 2, the stripe noise in the vertical gradient direction of the original observation image is effectively removed. At the same time, as shown in Table 2, the gradient characteristics remain basically unchanged during the stripe noise removal process, and the ICV value in the repaired image is greater than the ICV value in the original observation image, indicating that the stripe noise removal effect is good. The repaired image with stripe noise removed can improve the quality of graphic products and the research and application of subsequent quantitative inversion products of remote sensing data, and provide accuracy guarantee for them.

[0144] Table 2 Evaluation indicators of repaired images

[0145]

[0146] The infrared remote sensing image restoration method provided by the present invention constructs a denoising energy functional model through constraint terms and fidelity terms, and combines the anisotropy of stripe noise with a total variation denoising model. The constraint term is used to remove stripe noise, and the fidelity term is used to maintain detail feature information. In addition, based on the fact that the differential curvature value of the stripe noise contaminated area in the ideal image is greater than the differential curvature value of the non-strip noise contaminated area, the stripe noise contaminated area and the non-strip noise contaminated area in the input image are better distinguished, thereby improving the stripe noise removal effect. The constraint term and the fidelity term are balanced by a regularization parameter, thereby further improving the stripe noise removal accuracy in the image and the restoration image quality.

[0147] The infrared remote sensing image restoration device provided by the present invention is described below. The infrared remote sensing image restoration device described below and the infrared remote sensing image restoration method described above can be referenced to each other.

[0148] Figure 10 This is a schematic diagram of the structure of the infrared remote sensing image restoration device provided by the present invention. Figure 10 As shown, the infrared remote sensing image restoration device 1000 includes: an acquisition module 1001, a determination module 1002, a model construction module 1003 and an output module 1004, wherein:

[0149] An acquisition module 1001 is configured to acquire a target observation image, where the target observation image includes an ideal image and stripe noise.

[0150] A determination module 1002 is configured to determine a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of the stripe noise contaminated area in the ideal image is greater than the differential curvature value of the non-strip noise contaminated area;

[0151] A model construction module 1003 is configured to construct a denoising energy functional model based on the constraint term, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, and the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detailed feature information of the target observation image;

[0152] The output module 1004 is used to input the target observation image into the denoising energy functional model for iterative processing, and determine the final ideal image as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limit condition is met.

[0153] The infrared remote sensing image restoration device provided by the present invention constructs a denoising energy functional model through constraint terms and fidelity terms, and combines the anisotropy of stripe noise with a total variation denoising model. The constraint term is used to remove stripe noise, and the fidelity term is used to maintain detail feature information. In addition, based on the fact that the differential curvature value of the stripe noise contaminated area in the ideal image is greater than the differential curvature value of the non-strip noise contaminated area, the stripe noise contaminated area and the non-strip noise contaminated area in the input image are better distinguished, thereby improving the stripe noise removal effect. The constraint term and the fidelity term are balanced by a regularization parameter, thereby further improving the stripe noise removal accuracy in the image and the restoration image quality.

[0154] Optionally, the infrared remote sensing image restoration device 1000 further includes:

[0155] The evaluation module 1005 is configured to quantitatively evaluate the stripe noise restoration effects of the target observation image and the corresponding restoration image based on evaluation indicators.

[0156] Optionally, the evaluation module 1005 is specifically configured to:

[0157] Based on the evaluation index, the stripe noise restoration effect of the target observation image and the corresponding restoration image is quantitatively evaluated, wherein the evaluation index includes: at least one of: a row mean curve, a radiation quality improvement factor and an inverse variation coefficient, the row mean curve is used to characterize the average value of pixels in each row of the image before and after the stripe noise removal, the radiation quality improvement factor is used to characterize the change information of the directional distribution of the stripe noise before and after the stripe noise removal, and the inverse variation coefficient is used to characterize the degree of variation of the image before and after the stripe noise removal.

[0158] Optionally, the determination module 1002 is specifically configured to:

[0159] The determining of the constraint item for removing the stripe noise based on the differential curvature value of each pixel in the ideal image includes:

[0160] Determine the regularization parameter based on the ratio of the regularization parameter to the adjustment coefficient in the previous iteration cycle, wherein the initial regularization parameter is the ratio of the L2 norm of the ideal image perpendicular to the stripe noise gradient direction and along the gradient direction;

[0161] constructing a weight matrix based on the differential curvature value of each pixel in the ideal image, wherein the weight matrix is ​​used to adjust the regularization parameter of the stripe noise in different intensity regions;

[0162] Constructing a regularization constraint term based on the gradient of the ideal image perpendicular to the stripe noise gradient direction;

[0163] The constraint term is constructed based on the regularization parameter, the weight matrix and the regular constraint term.

[0164] Optionally, the determination module 1002 is specifically configured to:

[0165] The step of constructing a weight matrix based on the differential curvature value of each pixel in the ideal image includes:

[0166] In a current iteration cycle, respectively determining a pixel first-order partial derivative, a pixel second-order partial derivative, and a pixel second-order mixed partial derivative of each pixel in the ideal image;

[0167] Based on the pixel first-order partial derivative, the pixel second-order partial derivative, and the pixel second-order mixed partial derivative, respectively determining the horizontal second-order partial derivative of the ideal image along the stripe noise gradient direction and the vertical second-order partial derivative perpendicular to the gradient direction;

[0168] The weight matrix is ​​constructed based on the horizontal second-order partial derivatives and the vertical second-order partial derivatives.

[0169] Optionally, the output module 1004 is specifically configured to:

[0170] Inputting the target observation image into the denoising energy functional model for iterative processing, and determining the ultimately obtained ideal image as the repaired image with the stripe noise removed, comprises:

[0171] Constructing the Euler-Lagrange equation corresponding to the denoising energy functional model;

[0172] Determine a discretized Euler-Lagrange equation based on the target observation image, the pixel values ​​of the pixel points corresponding to the ideal image output in the previous iteration cycle, and the Euler-Lagrange equation;

[0173] Determining an ideal image outputted in a current iteration cycle based on the discretized Euler-Lagrange equation;

[0174] Using the ideal image outputted in the current iteration cycle as the input image for the next iteration cycle, and repeatedly outputting the ideal image of the corresponding iteration cycle;

[0175] When the limiting conditions are met, the iteration is terminated, and the ideal image finally outputted is determined as the repaired image.

[0176] Optionally, the output module 1004 is specifically configured to:

[0177] The limiting conditions include that the number of iterations is greater than a first threshold or the L2 norm value of the ideal image difference output from two adjacent iteration cycles is less than a second threshold.

[0178] Figure 11 An example of a physical structure diagram of an electronic device is shown below. Figure 11 As shown, the electronic device 1100 may include: a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the infrared remote sensing image restoration method, which includes:

[0179] Acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise;

[0180] determining a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area;

[0181] Based on the constraint term, a denoising energy functional model is constructed, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detail feature information of the target observation image;

[0182] The target observation image is input into the denoising energy functional model for iterative processing, and the ideal image finally obtained is determined as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limitation condition is met.

[0183] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the infrared remote sensing image restoration method provided by the above methods, which includes:

[0185] Acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise;

[0186] determining a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area;

[0187] Based on the constraint term, a denoising energy functional model is constructed, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detail feature information of the target observation image;

[0188] The target observation image is input into the denoising energy functional model for iterative processing, and the ideal image finally obtained is determined as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limitation condition is met.

[0189] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the infrared remote sensing image restoration method provided by the above methods, the method comprising:

[0190] Acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise;

[0191] determining a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area;

[0192] Based on the constraint term, a denoising energy functional model is constructed, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detail feature information of the target observation image;

[0193] The target observation image is input into the denoising energy functional model for iterative processing, and the ideal image finally obtained is determined as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limitation condition is met.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0195] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for restoring an infrared remote sensing image, characterized in that: include: Acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise; determining a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area; Based on the constraint term, a denoising energy functional model is constructed, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain detail feature information of the target observation image; The target observation image is input into the denoising energy functional model for iterative processing, and the ideal image finally obtained is determined as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output from the previous iteration is input into the denoising energy functional model until the iteration limitation condition is met.

2. The infrared remote sensing image restoration method according to claim 1, characterized in that: The determining of the constraint item for removing the stripe noise based on the differential curvature value of each pixel in the ideal image includes: Determine the regularization parameter based on the ratio of the regularization parameter to the adjustment coefficient in the previous iteration cycle, wherein the initial regularization parameter is the ratio of the L2 norm of the ideal image perpendicular to the stripe noise gradient direction and along the gradient direction; constructing a weight matrix based on the differential curvature value of each pixel in the ideal image, wherein the weight matrix is ​​used to adjust the regularization parameter of the stripe noise in different intensity regions; Constructing a regularization constraint term based on the gradient of the ideal image perpendicular to the stripe noise gradient direction; The constraint term is constructed based on the regularization parameter, the weight matrix and the regular constraint term.

3. The infrared remote sensing image restoration method according to claim 2, characterized in that: The step of constructing a weight matrix based on the differential curvature value of each pixel in the ideal image includes: In a current iteration cycle, respectively determining a pixel first-order partial derivative, a pixel second-order partial derivative, and a pixel second-order mixed partial derivative of each pixel in the ideal image; Based on the pixel first-order partial derivative, the pixel second-order partial derivative, and the pixel second-order mixed partial derivative, respectively determining the horizontal second-order partial derivative of the ideal image along the stripe noise gradient direction and the vertical second-order partial derivative perpendicular to the gradient direction; The weight matrix is ​​constructed based on the horizontal second-order partial derivatives and the vertical second-order partial derivatives.

4. The infrared remote sensing image restoration method according to any one of claims 1 to 3, characterized in that: Inputting the target observation image into the denoising energy functional model for iterative processing, and determining the ultimately obtained ideal image as the repaired image with the stripe noise removed, comprises: Constructing the Euler-Lagrange equation corresponding to the denoising energy functional model; Determine a discretized Euler-Lagrange equation based on the target observation image, the pixel values ​​of the pixel points corresponding to the ideal image output in the previous iteration cycle, and the Euler-Lagrange equation; Determining an ideal image outputted in a current iteration cycle based on the discretized Euler-Lagrange equation; Using the ideal image outputted in the current iteration cycle as the input image for the next iteration cycle, and repeatedly outputting the ideal image of the corresponding iteration cycle; When the limiting conditions are met, the iteration is terminated, and the ideal image finally outputted is determined as the repaired image.

5. The infrared remote sensing image restoration method according to claim 4, characterized in that: The limiting conditions include that the number of iterations is greater than a first threshold or the L2 norm value of the ideal image difference output from two adjacent iteration cycles is less than a second threshold.

6. The infrared remote sensing image restoration method according to any one of claims 1 to 3, characterized in that: Also includes: Based on the evaluation index, the stripe noise restoration effect of the target observation image and the corresponding restoration image is quantitatively evaluated, wherein the evaluation index includes: at least one of: a row mean curve, a radiation quality improvement factor and an inverse variation coefficient, the row mean curve is used to characterize the average value of pixels in each row of the image before and after the stripe noise removal, the radiation quality improvement factor is used to characterize the change information of the directional distribution of the stripe noise before and after the stripe noise removal, and the inverse variation coefficient is used to characterize the degree of variation of the image before and after the stripe noise removal.

7. An infrared remote sensing image restoration device, characterized in that: include: An acquisition module is used to acquire a target observation image, wherein the target observation image includes an ideal image and stripe noise; a determination module, configured to determine a constraint item for removing the stripe noise based on a differential curvature value of each pixel in the ideal image, wherein the differential curvature value of a stripe noise contaminated area in the ideal image is greater than the differential curvature value of a non-strip noise contaminated area; A model construction module is used to construct a denoising energy functional model based on the constraint term, wherein the denoising energy functional model is obtained based on the anisotropy of the strip noise and the total variation denoising model, the denoising energy functional model includes a fidelity term and the constraint term, and the fidelity term is used to maintain the detail feature information of the target observation image; An output module is used to input the target observation image into the denoising energy functional model for iterative processing, and determine the final ideal image as the repaired image with the stripe noise removed, wherein: during each iteration, the ideal image output by the previous iteration is input into the denoising energy functional model until the iteration limit condition is met.

8. The infrared remote sensing image restoration device according to claim 7, characterized in that: Also includes: An evaluation module is used to quantitatively evaluate the stripe noise restoration effects of the target observation image and the corresponding restoration image based on evaluation indicators.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the infrared remote sensing image restoration method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the infrared remote sensing image restoration method according to any one of claims 1 to 6 is implemented.

Images