Remote sensing image information reconstruction method based on isophot line constraint and color structure control

Abstract

This invention provides a remote sensing image information reconstruction method based on isoluminance constraints and color structure control. The method inputs a registered target image, a reference image, and a mask file marking missing regions. Utilizing spatial information outside the missing regions in the target and reference images, a color structure consistency constraint is constructed to solve for partially missing pixels in the target image's missing regions. The target image and mask file are updated based on the solved partial pixels. Combining the information corresponding to the missing regions in the updated reference image, the isoluminance and gradient weights of the pixels to be identified within that region are calculated. An isoluminance equation is constructed based on the calculated isoluminance and weights, and the reconstructed image is obtained by solving the equation, thus achieving the repair of missing regions in the remote sensing image. This invention is the first to propose calculating isoluminance based on weighted gradient values ​​of each pixel in each direction, forming an image isoluminance network, constructing an isoluminance equation to solve for pixels in missing regions, and improving reconstruction accuracy through color consistency constraints.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image restoration and relates to a method for reconstructing remote sensing image information based on isoluminescence constraints and color structure control. Background Technology

[0002] Optical remote sensing satellites are susceptible to sensor defects and harsh atmospheric conditions during the imaging process, significantly reducing the integrity of remote sensing data. This includes issues such as missing stripes due to sensor malfunctions and missing ground information caused by cloud cover. This not only affects further image processing and restricts the application of remote sensing imagery, but also greatly reduces its utilization rate and accuracy. Researching the reconstruction of missing areas in remote sensing images and restoring missing ground information is of great significance for improving the utilization rate of remote sensing data.

[0003] Reconstructing missing areas in remote sensing images requires utilizing known auxiliary information to restore the missing regions of the target image. Based on the information source used for image reconstruction, existing remote sensing image reconstruction methods can be categorized into spatial information-based methods, spectral information-based methods, and temporal information-based methods. Based on the premise that ground feature information in an image is continuous, spatial information-based methods utilize information from clearly defined areas in the target image to repair missing areas. Generally, this type of method achieves good results when the missing area is small and the ground texture is simple, but its reconstruction accuracy drops significantly when the missing area is large or the ground texture becomes complex. In multispectral and hyperspectral images, because longer wavelength bands have stronger penetrating power, spectral information-based methods utilize undisturbed bands in the image to repair damaged bands. Compared to spatial information-based methods, this type of method makes better use of auxiliary information from different bands, but it is difficult to apply when all bands of the image are disturbed. Temporal information-based methods utilize temporally adjacent multi-temporal images covering the same area to restore missing areas in the target image. This type of method utilizes sufficient auxiliary information to ensure the reliability of the reconstructed image. Generally speaking, this type of method can achieve satisfactory results when there are no significant spectral differences or land cover changes between the target image and the reference image. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies and, considering the significant radiometric differences between images from different time phases, provide a remote sensing image information reconstruction scheme based on isoluminescence constraints and color structure control. This scheme reconstructs the target image using reference images acquired from the same region at different time phases. This method can maximize the preservation of detail information from the reference images and background information from the target image, resulting in reconstruction results with good color consistency.

[0005] The technical solution of this invention includes a remote sensing image information reconstruction method based on isoluminescence constraints and color structure control. The method involves inputting a registered target image, a reference image, and a mask file marking missing regions, and then performing the following processing:

[0006] By utilizing spatial information outside the missing areas of the target image and reference image, a color structure consistency constraint is constructed to solve for some missing pixels in the missing areas of the target image;

[0007] Update the target image and mask file based on the solved partial pixels;

[0008] By combining the updated target image and mask file, and using the information in the reference image corresponding to the missing region, the isoluminance of the pixel to be determined in that region and its gradient weights in each direction are calculated.

[0009] Based on the calculated iso-illuminance and weight information, an iso-illuminance equation is constructed, and the reconstructed image is obtained by solving the equation, thereby realizing the restoration of missing areas in remote sensing images.

[0010] Furthermore, when constructing color structure consistency constraints to solve for some missing pixels in the missing region of the target image, superpixel segmentation is performed on the reference image. Based on the superpixel segmentation results of the reference image, a missing pixel in the target image is solved within each superpixel range.

[0011] Moreover, the method for calculating the isoluminance of the pixel to be solved is to calculate the gradient weight of each pixel in the reference image in each direction within its eight neighborhood range for the pixel to be solved in the mask file, and calculate the isoluminance based on the weight.

[0012] Furthermore, the process of constructing the isoluminance equation based on the calculated isoluminance and weight information uses the isoluminance value of the reference image and the known pixels of the outer boundary of the missing area of ​​the target image as known conditions, and the pixels to be solved in the target image as unknowns, to construct a set of image isoluminance equations.

[0013] Moreover, when obtaining the reconstructed image by solving the equations, the constructed linear equation system is solved iteratively, and the target image is filled according to the solved pixel values ​​to obtain a clear and complete reconstruction result.

[0014] On the other hand, the present invention provides a remote sensing image information reconstruction system based on iso-illuminance constraints and color structure control, for implementing the remote sensing image information reconstruction method based on iso-illuminance constraints and color structure control as described above.

[0015] Moreover, it includes the following modules,

[0016] The first module is used to input the registered target image, reference image, and mask file that marks the missing areas;

[0017] The second module is used to construct color structure consistency constraints to solve for some missing pixels in the missing areas of the target image by utilizing spatial information outside the missing areas of the target image and the reference image.

[0018] The third module is used to update the target image and mask file based on the solved partial pixels;

[0019] The fourth module is used to combine the updated target image and mask file, and use the information in the reference image corresponding to the missing area to calculate the isoluminance of the pixel to be determined in the area and its gradient weights in each direction.

[0020] The fifth module is used to construct the iso-illuminance equation based on the calculated iso-illuminance and weight information, and obtain the reconstructed image by solving the equation, thereby realizing the restoration of missing areas in the remote sensing image.

[0021] Alternatively, it may include a processor and a memory, with the memory used to store program instructions and the processor used to call the stored instructions in the memory to execute a remote sensing image information reconstruction method based on isoluminance constraints and color structure control as described above.

[0022] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements a remote sensing image information reconstruction method based on isoluminescence constraints and color structure control as described above.

[0023] Compared with the prior art, the present invention has the following characteristics:

[0024] This invention is applicable to the restoration of missing regions in remote sensing images. Compared with existing methods, this invention proposes for the first time a method for calculating isoluminescence based on the weighted gradient values ​​of each pixel in each direction, forming an image isoluminescence network. This network constructs an isoluminescence equation to solve for pixels in missing image regions, and further improves reconstruction accuracy through color consistency constraints. This invention does not require a large number of training samples, and can obtain reconstructed images with high color consistency even for reference images with significant radiometric differences. Attached Figure Description

[0025] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] See Figure 1This invention provides a method for reconstructing remote sensing image information based on iso-illuminance constraints and color structure control. The method takes a registered target image, a reference image, and a mask file marking missing regions as input. The target image refers to the cloud-containing image to be restored, and the reference image refers to cloud-free images of the same region acquired at different times. Specifically, it includes the following steps:

[0028] Step 1: Utilize the spatial correlation between the target image and the reference image to construct color structure consistency constraints for pixels in the reference image located within the missing region and search for similar pixels outside the missing region of the target image; assign the pixel in the target image corresponding to the similar pixel to the corresponding pixel in the missing region.

[0029] The present invention further proposes the following implementation method for step 1:

[0030] Perform superpixel segmentation on the reference image, and based on the superpixel segmentation results of the reference image, solve for a missing pixel in the target image within each superpixel range;

[0031] When solving for the current pixel, a color structure consistency constraint is constructed based on the pixel of the corresponding reference image and its neighborhood information, and similar pixels are searched outside the missing area of ​​the reference image.

[0032] The pixel to be solved is assigned a value using the pixel of the target image corresponding to the similar pixel.

[0033] The preferred implementation of this step in the embodiment further includes the following sub-steps:

[0034] Step 1.1: Perform superpixel segmentation and clustering on the reference image to obtain its superpixel segmentation map S. r And classification diagram C rThe preferred implementation suggestions for superpixel segmentation algorithms are found in the following references: Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S. SLIC Superpixels Compared to State-of-the-Art Superpixel Methods[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34, 2274-2282; the preferred implementation suggestions for clustering algorithms are found in the following references: Su T, Dy J. In search of deterministic methods for initializing K-means and Gaussian mixture clustering[J]. Intelligent Data Analysis, 2007, 11, 319-338. These will not be elaborated upon in this invention.

[0035] Step 1.2: Based on the superpixel segmentation results of the reference image, determine pixel p within the range of each superpixel in the missing region. r :

[0036]

[0037] p r ∈P r

[0038] Among them, I r For reference image, P r For I r A superpixel, Ω represents the missing region to be repaired, i.e., the target region, p r For reference image in P r One pixel within.

[0039] Step 1.3, based on the classification result C of the reference image r Determine the region outside the missing region and p r Pixel sets Ω that belong to the same category p :

[0040]

[0041] Among them, C r (q) refers to C r In the category value of pixel q, C r (p r ) is C r At pixel p rThe category value is Ω, where Ω represents the missing region to be repaired.

[0042] Step 1.4, construct color structure consistency constraints, in Ω p Search p in the middle r Similar pixels q r :

[0043]

[0044]

[0045] Where MAE() is the mean absolute error, N p q represents the eight neighbors of pixel p. r p in the reference image r Similar pixels, |I r (q r )-I r (p r )|For reference image I r Chinese q r and p r Pixel value I r () the absolute value of the difference, q t For q r In target image I t The corresponding pixels in For q r The eight neighboring areas and q t The eight neighboring areas The correlation coefficient between them. T is the threshold value of the correlation coefficient, which is preferably set to 0.9 in this embodiment.

[0046] Step 1.5, q t The pixel values ​​are assigned to the target image I. t The middle corresponds to p r The pixel p to be solved t .

[0047] Step 2: Update the target image and mask file based on the solved partial pixels.

[0048] The present invention further proposes that step 2 be implemented by filling and updating the target image with the solved partial pixels, and updating the solved pixel state in the mask file to the known state.

[0049] In the example, it is denoted as:

[0050] I' t (p t ) = I t (q t )

[0051] M′t (p t ) = 0

[0052] Among them, I t () represents the target image, I' t () represents the updated target image, M' t () represents the updated mask file. When M' t A value of 0 for (p) indicates that p is a known pixel; M' t A value of 1 for (p) indicates that p is the pixel to be solved.

[0053] Step 3: Combine the updated target image and mask file to determine the boundary pixels of the target area and the updated known pixels, and use the auxiliary information of the reference image to calculate the isoluminance of the pixels to be solved in the target area and the gradient weights of each direction.

[0054] In this embodiment, the preferred implementation of this step includes the following sub-steps:

[0055] Step 3.1: For the pixel to be solved in the mask file, calculate the gradient weight of each pixel in the reference image in each direction within its eight neighborhood.

[0056] The embodiment calculates the gradient weights of each pixel in each direction for each pixel to be solved within the missing region based on the updated mask file. Taking pixel p as an example, it and its eight neighboring pixels are designated as p1…p9, denoted as the following matrix N. p This section introduces the method for calculating weights:

[0057]

[0058]

[0059] Where, N p - It is the eight neighboring pixels of pixel p, excluding p (p is p5). For p i The gradient weights between i and p, i = 1, 2, ..., 9.

[0060] When p i When -p=0

[0061]

[0062] Where β is a very small constant, preferably 10 in the embodiments. -4 .

[0063] Step 3.2: The isolating intensity of each pixel in the reference image to be solved is further calculated using the weight matrix, i.e.:

[0064]

[0065] in, The isoluminance is the reference image pixel p.

[0066] Step 3.3: Determine the boundary pixels of the target region and the updated set of known pixels. That is, record the set of known pixels of the target image at the boundary of the missing region, and add the updated solved pixels in the target image to this set. Set the center weight of each pixel in this set to 1, and the others to 0.

[0067] Step 4: Construct iso-illuminance equations based on the calculated iso-illuminance and weight information. This includes using the iso-illuminance values ​​of the pixels to be solved based on the reference image and the known pixels of the outer boundary of the missing area of ​​the target image as known conditions, the weight information of the pixels involved as a coefficient matrix, and the pixels to be solved as unknown parameters. Construct a set of image iso-illuminance equations and obtain the reconstructed image by solving the equations.

[0068] The present invention further proposes that,

[0069] After constructing a system of image isoluminance equations, using the isoluminance values ​​of the reference image and the known pixels at the outer boundary of the missing area in the target image as known conditions, and the pixels to be solved in the target image as unknowns;

[0070] The constructed linear equation system is solved iteratively, and the target image is filled with the solved pixel values ​​to obtain a clear and complete reconstruction result.

[0071] In this embodiment, the preferred implementation of this step includes the following sub-steps:

[0072] Step 4.1: Based on the updated mask file, establish equations for all pixels to be solved:

[0073]

[0074]

[0075] in, For the reference image isoluminance, n is the number of pixels within the initial missing region Ω, and m is the number of pixels located at the outer boundary of Ω. The number of pixels on the map, k is the number of known pixels updated in step 1, Ω' is the updated target region, and W Ω' Let be the weight matrix of the pixels to be solved. For the pixel p to be solved i The weight vector, For p i and p j Gradient weights between them. In, only when Only when... otherwise, Here, i = 1, 2, ..., nk, j = 1, 2, ..., n+m.

[0076] Step 4.2: Construct equations for the boundary pixels of the missing region in the target image and the updated known pixels, and combine them with the equations from Step 4.1 to obtain:

[0077]

[0078] Among them, I * To reconstruct the image. Only when i = j, otherwise, Here, i = 1, 2, ..., n + m, j = 1, 2, ..., n + m.

[0079] Step 4.3: Based on the constructed positive definite linear equation system with sparse symmetric coefficient matrix, the initial reconstructed image I can be obtained by iteratively solving the system using the conjugate gradient method. * .

[0080] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0081] In some possible embodiments, a remote sensing image information reconstruction system based on isoluminescence constraints and color structure control is provided, including the following modules:

[0082] The first module is used to input the registered target image, reference image, and mask file that marks the missing areas;

[0083] The second module is used to construct color structure consistency constraints to solve for some missing pixels in the missing areas of the target image by utilizing spatial information outside the missing areas of the target image and the reference image.

[0084] The third module is used to update the target image and mask file based on the solved partial pixels;

[0085] The fourth module is used to combine the updated target image and mask file, and use the information in the reference image corresponding to the missing area to calculate the isoluminance of the pixel to be determined in the area and its gradient weights in each direction.

[0086] The fifth module is used to construct the iso-illuminance equation based on the calculated iso-illuminance and weight information, and obtain the reconstructed image by solving the equation, thereby realizing the restoration of missing areas in the remote sensing image.

[0087] In some possible embodiments, a remote sensing image information reconstruction system based on isoluminance constraints and color structure control is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a remote sensing image information reconstruction method based on isoluminance constraints and color structure control as described above.

[0088] In some possible embodiments, a remote sensing image information reconstruction system based on isoluminescence constraints and color structure control is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the remote sensing image information reconstruction method based on isoluminescence constraints and color structure control as described above.

[0089] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for reconstructing remote sensing image information based on isoluminescence constraints and color structure control, characterized in that: Input the registered target image, reference image, and mask file marking the missing areas, and perform the following processing. By utilizing spatial information outside the missing areas of the target image and reference image, a color structure consistency constraint is constructed to solve for some missing pixels in the missing areas of the target image; Update the target image and mask file based on the solved partial pixels; By combining the updated target image and mask file, and using the information in the reference image corresponding to the missing region, the isoluminance of the pixel to be determined in that region and its gradient weights in each direction are calculated. Based on the calculated iso-illuminance and weight information, an iso-illuminance equation is constructed, and the reconstructed image is obtained by solving the equation, thereby realizing the restoration of missing areas in remote sensing images.

2. The remote sensing image information reconstruction method based on isoluminescence constraints and color structure control according to claim 1, characterized in that: When constructing color structure consistency constraints to solve for some missing pixels in the missing region of the target image, superpixel segmentation is performed on the reference image. Based on the superpixel segmentation results of the reference image, a missing pixel in the target image is solved within each superpixel range.

3. The remote sensing image information reconstruction method based on isoluminescence constraints and color structure control according to claim 1, characterized in that: The method for calculating the isoluminance of the pixel to be solved is as follows: for the pixel to be solved in the mask file, calculate the gradient weight of each pixel in the reference image in each direction within its eight neighborhood, and calculate the isoluminance based on the weight.

4. The remote sensing image information reconstruction method based on isoluminescence constraints and color structure control according to claim 1, characterized in that: The process of constructing the isoluminance equation based on the calculated isoluminance and weight information involves using the isoluminance value of the reference image and the known pixels of the outer boundary of the missing area in the target image as known conditions, and using the pixels in the target image to be solved as unknowns, to construct a set of image isoluminance equations.

5. The remote sensing image information reconstruction method based on isoluminescence constraints and color structure control according to claim 4, characterized in that: When obtaining a reconstructed image by solving equations, the constructed linear equation system is iteratively solved, and the target image is filled according to the solved pixel values ​​to obtain a clear and complete reconstruction result.

6. A remote sensing image information reconstruction system based on isoluminescence constraints and color structure control, characterized in that: This method is used to implement the remote sensing image information reconstruction method based on isoluminescence constraints and color structure control as described in any one of claims 1-5.

7. The remote sensing image information reconstruction system based on isoluminescence constraints and color structure control according to claim 6, characterized in that: Includes the following modules, The first module is used to input the registered target image, reference image, and mask file that marks the missing areas; The second module is used to construct color structure consistency constraints to solve for some missing pixels in the missing areas of the target image by utilizing spatial information outside the missing areas of the target image and the reference image. The third module is used to update the target image and mask file based on the solved partial pixels; The fourth module is used to combine the updated target image and mask file, and use the information in the reference image corresponding to the missing area to calculate the isoluminance of the pixel to be determined in the area and its gradient weights in each direction. The fifth module is used to construct the iso-illuminance equation based on the calculated iso-illuminance and weight information, and obtain the reconstructed image by solving the equation, thereby realizing the restoration of missing areas in the remote sensing image.

8. The remote sensing image information reconstruction system based on isoluminescence constraints and color structure control according to claim 6, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the remote sensing image information reconstruction method based on isoluminance constraints and color structure control as described in any one of claims 1-5.

9. The remote sensing image information reconstruction system based on isoluminescence constraints and color structure control according to claim 6, characterized in that: The method includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements a remote sensing image information reconstruction method based on isoluminance constraints and color structure control as described in any one of claims 1-5.

Images