Remote sensing spliced image rectangularization method and system based on iterative distortion kernel correction

Abstract

The invention provides a remote sensing spliced image rectangularization method and system based on iterative distortion kernel correction. The method comprises the following steps: acquiring a spliced remote sensing image and a mask image thereof; fusing the spliced remote sensing image and the mask image thereof to obtain an input image, and performing feature extraction to obtain potential features related to a distortion kernel; performing distortion kernel prediction on the potential features to obtain distortion kernel parameters; performing distortion operation on the potential features by using the distortion kernel parameters to obtain intermediate distortion potential features; distortion kernel prediction error estimation is carried out on the intermediate distortion potential features; performing iterative compensation on the distortion kernel parameter by using the distortion kernel prediction error to obtain a final distortion kernel parameter; and performing distortion operation on the spliced remote sensing image by using the final distortion kernel parameter to obtain a final rectangular image. According to the invention, the spliced irregular remote sensing image can be distorted into a regular rectangular image without introducing obvious distortion.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image stitching post-processing, and relates to a remote sensing stitching image rectangularization method and system based on iterative distortion kernel correction. Background Art

[0002] Image stitching is a long-standing technology, and its main purpose is to generate an image with a wider field of view (FoV) by integrating multiple input images with narrow fields of view. This technology is widely used in many fields, such as photography, remote sensing, virtual reality, etc. It not only helps people appreciate the scene from a wider perspective, but also provides support for computer vision tasks such as image segmentation and target detection. However, although image stitching technology is quite mature, it still faces some challenges in practical applications, especially when processing remote sensing images.

[0003] When stitching remote sensing images, irregular boundaries often appear in the stitched images due to differences in imaging devices, changes in shooting angles, and the complexity of surface features. These boundaries are usually caused by the fact that the overlapping parts between the images are not perfectly matched, resulting in some areas being unable to find corresponding pixel values, thus forming the so-called "blank" or "invalid" areas. In order to make the final stitched image meet the requirements of most image processing algorithms, these irregular boundaries are usually filled into a standard rectangular shape, but this filling process may introduce additional noise or distortion, which in turn affects the performance of subsequent deep learning-based analysis tasks.

[0004] To address these issues, researchers are exploring various methods to improve the quality of image stitching, including but not limited to improving registration algorithms to achieve higher-precision image alignment, developing smarter filling techniques to reduce or eliminate the impact of irregular boundaries, and designing deep learning models specifically for stitched images so that they can tolerate these defects to a certain extent. In addition, with the development of artificial intelligence technology, especially the progress in the field of deep learning, more innovative solutions are expected to emerge in the future to further improve the effect of image stitching, so that the stitched images can maintain the integrity of the original information and meet the needs of high-quality visual presentation. In order to obtain a complete rectangular image that conforms to visual perception in a general sense, image rectangularization methods have emerged. Although traditional image rectangularization methods have solved the problem of irregular boundaries generated after image stitching to a certain extent, they still have many limitations. According to the different effects of different methods on the original image content, image rectangularization methods can be divided into three categories:

[0005] 1) The rectangularization method based on rectangular frame cropping is the simplest and most direct method, but it will lead to a reduction in the receptive field and a loss of part of the original image content, resulting in partial loss of useful information, which is obviously unacceptable when all details need to be retained.

[0006] 2) The rectangularization method based on image restoration uses image restoration technology to fill the original blank areas with effective textures or features. Although this method can fill these vacant areas, it will generate information that is not included in the original image, which is an increase in the content of the original image. That is, it may introduce content that seems reasonable but does not actually exist, thereby affecting the authenticity and accuracy of the image.

[0007] 3) Based on the content-preserving rectangularization method, the image is locally distorted or geometrically changed under the premise of introducing some acceptable distortion, so as to move the original pixels with practical significance to the original blank area to achieve the purpose of rectangularization. This type of method will not reduce or increase the actual content of the original image. Although it aims to minimize information loss, its implementation process is relatively complex and time-consuming, which brings additional operational difficulty and time cost to users. In addition, its ability to process nonlinear scenes is limited, especially when facing complex backgrounds or large changes in perspective, its performance is unstable, and it is difficult to ensure the consistency and reliability of the output.

[0008] Therefore, how to provide a remote sensing mosaic image rectangularization method and system based on iterative distortion kernel correction that can be efficient and ensure high authenticity of the mosaic image is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0009] In view of this, the present invention proposes a remote sensing stitching image rectangularization method and system based on iterative distortion kernel correction, which can distort the spliced ​​irregular remote sensing image into a regular rectangular image without introducing obvious distortion.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] The present invention discloses a method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction, comprising the following steps:

[0012] S1: Obtain the spliced ​​remote sensing image and the mask image corresponding to the spliced ​​remote sensing image;

[0013] S2: Based on the deep learning model, the spliced ​​remote sensing image and the mask image are fused to obtain an input image, and feature extraction is performed on the input image to obtain potential features related to the distortion kernel;

[0014] S3: Predicting the distortion kernel of the potential features based on the deep learning model to obtain distortion kernel parameters;

[0015] S4: using the distortion kernel parameter to perform a distortion operation on the latent feature to obtain an intermediate distorted latent feature;

[0016] S5: performing a distortion kernel prediction error estimation on the intermediate distortion potential feature based on a deep learning model; compensating the distortion kernel parameters using the distortion kernel prediction error to obtain new distortion kernel parameters; replacing the distortion kernel parameters with the new distortion kernel parameters and repeatedly performing S4-S5 until the estimated distortion kernel prediction error meets the set requirements, obtaining the final distortion kernel parameters, and entering S6;

[0017] S6: Using the final distortion kernel parameters, a distortion operation is performed on the spliced ​​remote sensing image to obtain a final rectangular image.

[0018] Preferably, S2 includes the step of constructing a fusion feature extractor, which is used to extract features from the input image to obtain potential features related to the distortion kernel; the fusion feature extractor includes: an attention module, four convolution modules C1 and three maximum pooling layers connected in sequence.

[0019] Preferably, S3 includes the step of constructing a distortion core predictor, which is used to perform distortion core prediction on the potential features to obtain distortion core parameters; the distortion core predictor includes: four convolution modules C1, three maximum pooling layers and one convolution module C2 connected in sequence.

[0020] Preferably, S5 includes the step of constructing a distortion kernel corrector, wherein the distortion kernel corrector is used to estimate the distortion kernel prediction error of the intermediate distortion potential feature; the distortion kernel corrector has the same network structure as the distortion kernel predictor.

[0021] Preferably, in S4, the distortion operation performed on the potential feature includes: coordinate transformation and / or interpolation operation.

[0022] Preferably, in S5, the step of compensating the distortion kernel parameters by using the distortion kernel prediction error comprises:

[0023] The t-1th distortion kernel parameter compensation result and the tth estimated distortion kernel parameter prediction error are element-wise added to complete the compensation of the tth distortion kernel parameter prediction error, t>1.

[0024] Preferably, before S3, the method further includes: dividing the input image into a plurality of grid units; and distorting the pixels in each grid unit according to the same distortion kernel parameter.

[0025] Preferably, based on the total loss function, the fusion feature extractor, the distortion kernel predictor and the distortion kernel corrector are jointly trained using training data; wherein the fusion feature extractor is used to extract features from the input image to obtain potential features related to the distortion kernel; the distortion kernel predictor is used to predict the distortion kernel on the potential features to obtain distortion kernel parameters; the distortion kernel corrector is used to estimate the distortion kernel prediction error on the intermediate distortion potential features;

[0026] The training data includes: a spliced ​​remote sensing image, a mask image corresponding to the spliced ​​remote sensing image, and a real image;

[0027] The total loss function is:

[0028] L=L d +αL p +βL m +γL s ,

[0029] L d =L d1 +L d2

[0030] =||I RE -W(h0,I ST )||1+||I RE -W(h T ,I ST )||1+||EW(h0,Ma)||1+||EW(h T ,Ma)||1,

[0031] L p =||VGG(I RE )-VGG(W(h0,I ST ))||2+||VGG(I RE )-VGG(W(h T ,I ST ))||2.

[0032] L s =||Sobel(I RE )-Sobel(W(h0,I ST ))||1+||Sobel(I RE )-Sobel(W(h T ,I ST ))||1,

[0033] Where, L d , L p , L m and L sThey are pixel loss, perceptual loss, grid loss and Sobel loss, the grid loss is the grid constraint for the grid unit to be distorted, α, β and γ are the weights of different loss terms; I ST is the spliced ​​remote sensing image, I RE is the real image, Ma is the mask image, h0 is the initial distortion kernel parameter, h T is the final distortion kernel parameter, W is the distortion operator, E is the all-one matrix, and VGG is the deep learning model.

[0034] The present invention also provides a remote sensing mosaic image rectangularization system according to the remote sensing mosaic image rectangularization method based on iterative distortion kernel correction, comprising:

[0035] An image acquisition unit, used for acquiring a spliced ​​remote sensing image and a mask image corresponding to the spliced ​​remote sensing image;

[0036] A fusion feature extractor, used for fusing the spliced ​​remote sensing image and its mask image to obtain an input image, and performing feature extraction on the input image to obtain potential features related to the distortion kernel;

[0037] A distortion kernel predictor, used for performing distortion kernel prediction on the latent features to obtain distortion kernel parameters;

[0038] A distortion unit, configured to perform a distortion operation on the latent feature using the distortion kernel parameter to obtain an intermediate distorted latent feature;

[0039] A distortion kernel corrector is used to estimate a distortion kernel prediction error for the intermediate distortion potential feature; compensate the distortion kernel parameters using the distortion kernel prediction error to obtain new distortion kernel parameters; replace the distortion kernel parameters with the new distortion kernel parameters and repeatedly input them into the distortion unit and the distortion kernel corrector until the estimated distortion kernel prediction error meets the set requirements, thereby obtaining the final distortion kernel parameters;

[0040] The rectangular image output unit is used to use the final distortion kernel parameters to perform a distortion operation on the spliced ​​remote sensing image to obtain a final rectangular image.

[0041] Preferably, the network structure of the distortion kernel corrector is the same as that of the distortion kernel predictor.

[0042] It can be seen from the above technical solution that, compared with the prior art, the beneficial effects of the present invention include:

[0043] The present invention realizes multi-stage feature extraction, distortion kernel prediction and correction by integrating deep learning models, ensuring the boundary regularity and content integrity of the final output image, and effectively reducing the distortion and noise introduced in the image distortion process. It can not only effectively improve the naturalness and realism of the image visual effect, but also maintain high stability and consistency in different scenarios.

[0044] The distortion kernel prediction and correction mechanism of the present invention is based on deep learning, which can maintain high stability and accuracy in different scenarios and is particularly suitable for processing complex image stitching problems.

[0045] The present invention can gradually reduce the prediction error through multi-stage iterative correction, thereby ensuring high quality and high precision of the final output image.

[0046] The present invention adopts a gridding strategy to perform image rectangularization processing, ensuring that no additional distortion or false information is generated during the entire conversion process, thereby maintaining the information integrity and visual quality of the original image to the greatest extent.

[0047] The algorithm design of the present invention is intuitive and concise, which greatly simplifies the processing flow of image rectangularization, reduces the difficulty of operation, improves the processing efficiency, and lowers the threshold for users.

[0048] In summary, the image rectangularization method proposed in the present invention not only overcomes many limitations of traditional methods, but also provides a more efficient and accurate new approach for the field of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work.

[0050] Figure 1 A logic block diagram of a remote sensing mosaic image rectangularization method based on iterative distortion kernel correction provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a distortion operator provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the effects of under-rectangularization and over-rectangularization provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram showing the comparison effect between the rectangularization method provided by an embodiment of the present invention and the rectangularization method using a cropping and filling strategy;

[0054] Figure 5 Schematic diagram of the comparison between the rectangularization method provided by the embodiment of the present invention and the existing rectangularization method on some remote sensing images Figure 1 ;

[0055] Figure 6 Schematic diagram of the comparison between the rectangularization method provided by the embodiment of the present invention and the existing rectangularization method on some remote sensing images Figure 2 ;

[0056] Figure 7 Schematic diagram of the comparison between the rectangularization method provided by the embodiment of the present invention and the existing rectangularization method on natural images Figure 1 ;

[0057] Figure 8 Schematic diagram of the comparison between the rectangularization method provided by the embodiment of the present invention and the existing rectangularization method on natural images Figure 2 ;

[0058] Fig. 9 A schematic diagram of the rectangularization effect of the rectangularization method provided by an embodiment of the present invention under uneven lighting conditions;

[0059] Fig.10 A schematic diagram of the rectangularization effect of the rectangularization method provided in an embodiment of the present invention on a satellite remote sensing image. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] A first aspect of an embodiment of the present invention provides a method for rectangularizing a remote sensing mosaic image based on iterative distortion kernel correction, comprising the following steps:

[0062] S1: Obtain the stitched remote sensing image and the mask image corresponding to the stitched remote sensing image;

[0063] S2: Based on the deep learning model, the spliced ​​remote sensing image and its mask image are fused to obtain the input image, and feature extraction is performed on the input image to obtain the potential features related to the distortion kernel;

[0064] S3: Predict the distortion kernel of the latent features based on the deep learning model and obtain the distortion kernel parameters;

[0065] S4: Use the distortion kernel parameters to distort the latent features to obtain intermediate distorted latent features;

[0066] S5: Estimating the distortion kernel prediction error of the intermediate distortion potential features based on the deep learning model; using the distortion kernel prediction error to compensate the distortion kernel parameters to obtain new distortion kernel parameters; replacing the distortion kernel parameters with the new distortion kernel parameters and repeatedly executing S4-S5 until the estimated distortion kernel prediction error meets the set requirements, obtaining the final distortion kernel parameters, and entering S6;

[0067] S6: Using the final distortion kernel parameters, the stitched remote sensing image is distorted to obtain a final rectangular image.

[0068] like Figure 1 (a) shows the overall flow chart of the rectangularization method of the present invention. The distortion kernel in the embodiment step refers to a distortion operator mathematical model used to guide the distortion deformation of the image, the purpose of which is to adjust the irregular boundary after splicing into a regular rectangular boundary. The distortion kernel parameters can indicate how each pixel should be moved so as to achieve the desired rectangularization effect while maintaining the integrity of the image content.

[0069] In one embodiment, Figure 2 As shown, the distortion operations performed on the potential features include: coordinate transformation and / or interpolation operations:

[0070] Coordinate transformation: Determine the new position of each pixel based on the distortion kernel parameters. This step requires mapping the original coordinates of the pixel to the new coordinate system.

[0071] Interpolation operation: Since the coordinate transformation may result in the new position not being an integer coordinate, an interpolation algorithm (such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation) is needed to estimate the pixel value at the new position.

[0072] In one embodiment, the warp kernel parameters include:

[0073] Grid parameters: Grid parameters define the number, size, and shape of grid cells into which the image is divided. The pixels in each grid cell will be distorted according to the same distortion kernel parameters.

[0074] Displacement vector: The pixels in each grid cell will move to a new position according to certain rules. The displacement vector specifies the direction and distance of each pixel movement. These displacement vectors are calculated based on the extracted latent features.

[0075] Deformation coefficient: The deformation coefficient controls the degree of image deformation and determines the degree of stretching or compression of the image during the distortion process. Different deformation coefficients can produce different degrees of distortion effects.

[0076] Boundary conditions: Boundary conditions define the image edge processing conditions, such as whether pixels are allowed to exceed the image boundary, or how to handle pixels that exceed the boundary. This helps to maintain the integrity of the image after rectangularization.

[0077] Weight matrix: The weight matrix is ​​used to adjust the influence between different grid cells to ensure that the deformation of adjacent grid cells is coordinated and consistent to avoid obvious boundaries or cracks.

[0078] It can be understood that in this embodiment, the warp kernel prediction error is iteratively estimated using the latent features after being distorted based on the warp kernel parameters of the previous step.

[0079] In one embodiment, S2 includes the step of constructing a fusion feature extractor, which is used to extract features from the input image to obtain potential features related to the distortion kernel. Figure 1 (b) shows the specific structure of the fusion feature extractor, which includes: an attention module, four convolution modules C1 and three maximum pooling layers connected in sequence.

[0080] The specific implementation process of this embodiment is as follows:

[0081] Given the spliced ​​image to be rectangularized I ST And its corresponding mask image Ma identifying the valid area, a fusion feature extractor F is used to extract the potential feature F0 related to the distortion kernel, that is:

[0082] F0=F ω (I ST ,Ma),

[0083] Where ω is the network parameter of the fusion feature extractor F. The fusion feature extractor F consists of an attention module A, four convolution modules C1, and three maximum pooling layers Maxpool. The attention module is responsible for fusing the image to be rectangularized I ST And its mask image Ma, the implementation process is:

[0084]

[0085] Among them, Sigmoid refers to the Sigmoid activation function, Conv 1×1 Refers to the convolution layer with a convolution kernel of 1×1, and Cat refers to the joint operation of tensors in the channel dimension. Refers to element-based multiplication. The convolution module C1 consists of two stacked convolution layers with a convolution kernel of 3×3 and a rectified linear unit (ReLU). The maximum pooling layer Maxpool is adjacent to the convolution module C1 and is used to reduce the size of the feature map and alleviate position sensitivity.

[0086] In one embodiment, S3 includes the step of constructing a distortion core predictor, which is used to perform distortion core prediction on potential features to obtain distortion core parameters; Figure 1 (c) shows the specific structure of the distortion kernel predictor, which includes four convolution modules C1, three maximum pooling layers and one convolution module C2 connected in sequence.

[0087] The specific implementation process of this embodiment is as follows:

[0088] After obtaining the potential feature F0 related to the distortion kernel, the distortion kernel predictor P is used to predict the potential feature F0 input in the previous step, and the preliminary predicted distortion kernel parameter h0 is output, that is:

[0089] h0=P θ (F0),

[0090] Where θ is the network parameter of the distortion kernel predictor P. The distortion kernel predictor P consists of four convolution modules C1, three maximum pooling layers Maxpool, and one convolution module C2. The maximum pooling layer Maxpool is adjacent to the first three convolution modules C1, and the fourth convolution module C1 is followed by the convolution module C2. The convolution module C2 consists of a convolution layer with a convolution kernel of 3×4 and two convolution layers with a convolution kernel of 1×1, and a rectified linear unit (ReLU) is inserted between each convolution layer.

[0091] In one embodiment, S5 includes the step of constructing a distortion kernel corrector, the distortion kernel corrector being used to perform distortion kernel prediction error estimation on the intermediate distortion potential features; Figure 1 (c) is the specific structure of the distortion kernel corrector. The distortion kernel corrector has the same network structure as the distortion kernel predictor. Due to the limitations of the network design, the initially predicted distortion kernel parameters may have errors with the true values ​​(unknown). Therefore, in S5, the distortion kernel corrector C uses the distorted latent features to estimate the error of the distortion kernel parameter prediction, thereby effectively compensating for it and obtaining a better and more accurate distortion kernel parameter estimate. In addition, due to the sensitivity of the distortion kernel, only one error compensation may result in insufficient or excessive error compensation (over-rectangularization or under-rectangularization), so S5 adopts an iterative correction strategy.

[0092] In this embodiment, the step of compensating the distortion kernel parameters by using the distortion kernel prediction error includes:

[0093] The t-1th distortion kernel parameter compensation result and the tth estimated distortion kernel parameter prediction error are element-wise added to complete the compensation of the tth distortion kernel parameter prediction error, t>1.

[0094] The specific implementation process of this embodiment is as follows:

[0095] In the tth iteration compensation, the estimated distortion kernel parameter prediction error Δh t Obtained by the following formula:

[0096] Δh t =C φ (F' t-1 ),t=1,L,T,

[0097] Among them, φ is the network parameter of the distortion kernel corrector C, and T is the total number of iterations. t-1 The intermediate potential features are obtained by distorting the potential features of the same input image based on the distortion kernel parameters after the t-1th distortion kernel parameter prediction error estimation compensation, that is:

[0098] F' t-1 =W(h t-1 ,F0),

[0099] Where W represents the distortion kernel h obtained after the t-1th compensation. t-1 The distortion operator performs a distortion operation on the potential feature F0, which includes coordinate transformation and interpolation. The compensation operation based on the prediction error estimation of the distortion kernel parameters is as follows:

[0100] h t =h t-1 +Δh t .

[0101] As shown in the above formula, the t-1th distortion kernel parameter compensation result h t-1 The estimated distortion kernel parameter prediction error Δh t By adding the elements, the prediction error of the distortion kernel parameters can be compensated, and the entire error estimation and compensation process can be completed. Through such iterative error estimation and compensation, iterative correction of the distortion kernel can be achieved.

[0102] In one embodiment, the specific execution process of S6 is:

[0103] Through the previous iterative steps, a more accurate distortion kernel parameter h can be obtained. T The last step is to use the input image to be rectangularized based on the final distortion kernel parameter h T Distort and get the final rectangular result I OUT :

[0104] I OUT =W(h T ,I ST ).

[0105] In one embodiment, the fusion feature extractor, the distortion kernel predictor, and the distortion kernel corrector need to be jointly trained on a public dataset using a specially designed loss function to obtain an algorithm instance that can be deployed.

[0106] In the process of iteratively estimating the distortion kernel prediction error, since the distortion operator W contains a large number of matrix inversion calculations, it is troublesome and time-consuming to optimize after each iteration. Therefore, it is chosen to limit the initial distortion kernel estimation and the final distortion kernel output.

[0107] The present invention adds a grid constraint as regularization in the total loss function and designs a Sobel loss to maintain the edge of the image.

[0108] The training data includes: the spliced ​​remote sensing image, the mask image corresponding to the spliced ​​remote sensing image, and the real image;

[0109] The total loss function is:

[0110] L=L d +αL p +βL m +γL s ,

[0111] For the pixel-wise loss, it includes the content of the input image and its corresponding mask image, namely:

[0112] L d =L d1 +L d2

[0113] =||I RE -W(h0,I ST )1+||I RE -W(h T ,I ST )||1+||EW(h0,Ma)||1+||EW(h T ,Ma)||1,

[0114] The perceptual loss is used to compare the differences between deep features extracted by the same pre-trained VGG network (denoted as VGG). The perceptual loss is often used in tasks such as super-resolution:

[0115] L p =||VGG(I RE )-VGG(W(h0,I ST ))||2+||VGG(I RE )-VGG(W(h T ,I ST ))||2.

[0116] Sobel loss helps preserve edge information during the distortion process. The specific form is:

[0117] L s =||Sobel(I RE )-Sobel(W(h0,I ST ))||1+||Sobel(I RE )-Sobel(W(h T ,I ST ))||1,

[0118] Where, L d , L p , L m and L s They are pixel loss, perceptual loss, grid loss and Sobel loss, α, β and γ are weights of different loss items. In this embodiment, α=5e-6, β=1 and γ=1e-4 are set; I ST is the spliced ​​remote sensing image, I RE is the real image, Ma is the mask image, h0 is the initial distortion kernel parameter, h T is the final distortion kernel parameter, W is the distortion operator, E is the all-one matrix, VGG is the deep learning model, and Sobel is the Sobel operator on the horizontal and vertical axes.

[0119] Grid item loss L m The grid constraint is used to distort the grid unit. The grid constraint means that during the distortion process, the grid of the image is constrained to prevent excessive distortion and distortion. Specifically, the grid constraint can limit the movement range of the grid nodes to ensure that the image maintains a certain smoothness and continuity during the distortion process. The grid item loss includes intra-grid constraints and inter-grid constraints, which imposes an appropriate amount of penalty on the grid movement.

[0120] The second aspect of the embodiment of the present invention also provides a remote sensing stitching image rectangularization system according to the first aspect of the embodiment, including: an image acquisition unit, used to acquire a stitched remote sensing image and a mask image corresponding to the stitched remote sensing image; a fusion feature extractor, used to fuse the stitched remote sensing image and its mask image to obtain an input image, and perform feature extraction on the input image to obtain potential features related to a distortion kernel; a distortion kernel predictor, used to perform distortion kernel prediction on the potential features to obtain distortion kernel parameters; a distortion unit, used to perform a distortion operation on the potential features using the distortion kernel parameters to obtain an intermediate distortion potential feature; a distortion kernel corrector, used to perform a distortion kernel prediction error estimation on the intermediate distortion potential feature; compensate the distortion kernel parameters using the distortion kernel prediction error to obtain new distortion kernel parameters; replace the distortion kernel parameters with the new distortion kernel parameters and repeatedly input them into the distortion unit and the distortion kernel corrector until the estimated distortion kernel prediction error meets the set requirements to obtain the final distortion kernel parameters; and a rectangularized image output unit, used to perform a distortion operation on the stitched remote sensing image using the final distortion kernel parameters to obtain a final rectangularized image.

[0121] In one embodiment, the distortion kernel corrector and the distortion kernel predictor have the same network structure.

[0122] The second aspect of the embodiment of the present invention is used to execute all the steps of the first aspect of the embodiment.

[0123] Combine the following Figure 3-Figure 10 The technical effect of the method of the present invention is illustrated by examples:

[0124] like Figure 3 The following are examples of under-rectangularization and over-rectangularization. Inaccurate prediction of the distortion kernel parameters may cause under-rectangularization or over-rectangularization.

[0125] like Figure 4 As shown in the figure, the difference in effect between the rectangularization method proposed by the present invention and other traditional methods (such as cropping and filling) in processing remote sensing images is compared. The figure can intuitively show the results of rectangularization of the same image by different methods, highlighting the advantages of the present invention in maintaining the authenticity of image content and reducing distortion.

[0126] like Figures 5 to 8As shown in the figure, the results of rectangularization of the same remote sensing image using different methods are shown. Among them, the traditional rectangularization method based on content preservation cannot perceive the image content and has poor adaptability, while other conventional deep learning rectangularization methods only use simple stacked convolutional layers to make a single direct prediction of the grid displacement and cannot accurately estimate the parameters of the distortion kernel. Therefore, both types of rectangularization methods are prone to incorrect estimation of the distortion kernel, resulting in "under-rectangularization" and "over-rectangularization" phenomena in the rectangularization results. Specifically, Figure 5 and Figure 7 As shown in the figure, the results of rectangularization of the same remote sensing image using different methods are shown. It can be seen that the traditional content-preserving method and the conventional depth method fail to fully stretch the image boundary, resulting in the phenomenon of "under-rectangularization", that is, the image fails to completely fill the entire rectangular area, leaving blank or invalid areas. Figure 6 and Figure 8 As shown, it is shown that the traditional method based on content preservation and the conventional depth method have the problem of "over-rectangularization" when processing remote sensing images, that is, in order to fit the image into the rectangular frame, the image is over-stretched, resulting in image deformation and even introducing false information. However, the present invention can achieve perfect rectangularization without losing image quality when processing remote sensing images.

[0127] like Fig. 9 As shown, it can be seen that even under complex lighting conditions, the method of the present invention can still generate high-quality rectangular images and maintain good visual effects. Fig.10 As shown, the method of the present invention is applied to satellite remote sensing images. By comparing with the original image, it can be clearly seen that the image processed by the present invention not only has regular boundaries, but also retains the original texture details without introducing obvious distortion or noise. All of these indicate the effectiveness and practicality of the present invention in processing specific types of images, and it has strong robustness, is applicable to various complex practical scenes, and improves its application value.

[0128] The above is a detailed introduction to the remote sensing stitching image rectangularization method and system based on iterative distortion kernel correction provided by the present invention. In this embodiment, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0129] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present embodiments may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present embodiment, but will conform to the widest range consistent with the principles and novel features disclosed in the present embodiment.

Claims

1. A method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction, characterized in that: The steps include: S1: Obtain the stitched remote sensing image and the mask image corresponding to the stitched remote sensing image; S2: Based on the deep learning model, the spliced ​​remote sensing image and the mask image are fused to obtain an input image, and feature extraction is performed on the input image to obtain potential features related to the distortion kernel; S3: Predicting the distortion kernel of the potential features based on the deep learning model to obtain distortion kernel parameters; S4: using the distortion kernel parameter to perform a distortion operation on the latent feature to obtain an intermediate distorted latent feature; S5: performing a distortion kernel prediction error estimation on the intermediate distortion potential feature based on a deep learning model; compensating the distortion kernel parameters using the distortion kernel prediction error to obtain new distortion kernel parameters; replacing the distortion kernel parameters with the new distortion kernel parameters and repeatedly performing S4-S5 until the estimated distortion kernel prediction error meets the set requirements, obtaining the final distortion kernel parameters, and entering S6; S6: Using the final distortion kernel parameters, a distortion operation is performed on the spliced ​​remote sensing image to obtain a final rectangular image.

2. The method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 1, characterized in that: The S2 includes the step of constructing a fusion feature extractor, which is used to extract features from the input image to obtain potential features related to the distortion kernel; the fusion feature extractor includes: an attention module, four convolution modules C1 and three maximum pooling layers connected in sequence.

3. The method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 1, characterized in that: The S3 includes the step of constructing a distortion core predictor, which is used to perform distortion core prediction on the potential features to obtain distortion core parameters; the distortion core predictor includes four convolution modules C1, three maximum pooling layers and one convolution module C2 connected in sequence.

4. A method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 1 or 3, characterized in that: The S5 includes the step of constructing a distortion kernel corrector, which is used to estimate the distortion kernel prediction error of the intermediate distortion potential feature; the distortion kernel corrector has the same network structure as the distortion kernel predictor.

5. The method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 1, characterized in that: In S4, the distortion operation performed on the potential features includes: coordinate transformation and / or interpolation operation.

6. The method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 1, characterized in that: In S5, the step of compensating the distortion kernel parameters by using the distortion kernel prediction error includes: The t-1th distortion kernel parameter compensation result and the tth estimated distortion kernel parameter prediction error are element-wise added to complete the compensation of the tth distortion kernel parameter prediction error, t>1.

7. The method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 1, characterized in that: Before S3, the method further includes: dividing the input image into a plurality of grid units; and distorting the pixels in each grid unit according to the same distortion kernel parameter.

8. The method for rectangularizing remote sensing mosaic images based on iterative distortion kernel correction according to claim 7, characterized in that: Based on the total loss function, the fusion feature extractor, the distortion kernel predictor and the distortion kernel corrector are jointly trained using the training data; wherein the fusion feature extractor is used to extract features from the input image to obtain potential features related to the distortion kernel; the distortion kernel predictor is used to predict the distortion kernel on the potential features to obtain distortion kernel parameters; the distortion kernel corrector is used to estimate the distortion kernel prediction error on the intermediate distortion potential features; The training data includes: a spliced ​​remote sensing image, a mask image corresponding to the spliced ​​remote sensing image, and a real image; The total loss function is: L=L d +αL p +βL m +γL s , L d =L d1 +L d2 =||I RE -W(h0,I ST )||1+||I RE -W(h T ,AND ST )||1+||EW(h0,Ma)||1+||EW(h T ,Has)||1, L p =||VGG(I RE )-VGG(W(h0,I ST ))||2+||VGG(I RE )-VGG(W(h T ,I ST ))||2. L s =||Sobel(I RE )-Sobel(W(h0,I ST ))||1+||Sobel(I RE )-Sobel(W(h T ,I ST ))||1, Where, L d , L p , L m and L s They are pixel loss, perceptual loss, grid loss and Sobel loss, the grid loss is the grid constraint for the grid unit to be distorted, α, β and γ are the weights of different loss terms; I ST is the spliced ​​remote sensing image, I RE is the real image, Ma is the mask image, h0 is the initial distortion kernel parameter, h T is the final distortion kernel parameter, W is the distortion operator, E is the all-one matrix, and VGG is the deep learning model.

9. A remote sensing image rectangularization system according to any one of claims 1 to 8, characterized in that: include: An image acquisition unit, used for acquiring a spliced ​​remote sensing image and a mask image corresponding to the spliced ​​remote sensing image; A fusion feature extractor, used for fusing the spliced ​​remote sensing image and its mask image to obtain an input image, and performing feature extraction on the input image to obtain potential features related to the distortion kernel; A distortion kernel predictor, used for performing distortion kernel prediction on the latent features to obtain distortion kernel parameters; A distortion unit, configured to perform a distortion operation on the latent feature using the distortion kernel parameter to obtain an intermediate distorted latent feature; A distortion kernel corrector is used to estimate a distortion kernel prediction error for the intermediate distortion potential feature; compensate the distortion kernel parameters using the distortion kernel prediction error to obtain new distortion kernel parameters; replace the distortion kernel parameters with the new distortion kernel parameters and repeatedly input them into the distortion unit and the distortion kernel corrector until the estimated distortion kernel prediction error meets the set requirements, thereby obtaining the final distortion kernel parameters; The rectangular image output unit is used to use the final distortion kernel parameters to perform a distortion operation on the spliced ​​remote sensing image to obtain a final rectangular image.

10. The remote sensing mosaic image rectangularization system according to claim 9, characterized in that: The network structure of the distortion kernel corrector is the same as that of the distortion kernel predictor.

Images