A method, device and electronic equipment for repairing distortion and deformation of remote sensing images
By generating an adversarial network model and pixel-level supervision loss function, the distortion and deformation of remote sensing images are automatically repaired, and the problems of high cost and low efficiency of manual repair in the prior art are solved, and the rapid and accurate repair of remote sensing images is achieved.
Patent Information
- Application Number
- CN202411195733.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-29
AI Technical Summary
In the prior art, the distortion deformation repair of remote sensing images requires a lot of manual intervention, and the time and labor cost are high, and not all images have distortion deformation, resulting in cumbersome repair process.
Generative adversarial network model is adopted, and the distortion deformation repair model is trained based on training data. Real repair samples and simulation synthetic samples are used, pixel-level supervision loss function is set, model parameters are optimized through error punishment, and remote sensing images are automatically repaired.
It realizes rapid and accurate repair of remote sensing images, reduces the cost of manual intervention and improves the repair efficiency.
Smart Images

Figure CN118710556B_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present specification relate to image processing technology, and more particularly to a method, device, and electronic device for repairing distortion and deformation of remote sensing images. Background Art
[0002] When remote sensing images are generated, some images will be distorted and deformed, affecting the overall data quality of the remote sensing images. In order to solve this problem, the operators generally perform photo editing and adjustment on the remote sensing images in Photoshop software, which increases huge time and labor costs. In addition, there is also a method in the prior art to repair the distorted images according to the affine transformation matrix after feature point detection and matching, but since not all images are distorted, this method requires manual confirmation of the distorted area each time before repairing, which still requires a lot of time and human resources. Summary of the invention
[0003] In order to solve the above problems, one or more embodiments of this specification describe a method, device and electronic device for repairing distortion and deformation of remote sensing images.
[0004] According to a first aspect, a method for repairing distortion and deformation of a remote sensing image is provided, the method comprising:
[0005] A generative adversarial network model is used as a basic model, and the basic model is trained based on training data to obtain a distortion and deformation repair model, wherein the training data includes real repair samples and simulated synthetic samples, wherein the real repair samples are obtained through manual annotation in an actual operation process, and the simulated synthetic samples are obtained after random deformation operations are performed on real non-distorted and deformed images, and the constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples. A pixel-level supervised loss function is set in the basic model, and the pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and an error penalty is generated according to the size of the pixel error, and the error penalty is used to guide the adjustment of model parameters to minimize the pixel error;
[0006] Acquire a remote sensing image to be restored, divide the remote sensing image into at least one first image based on a first preset size, and divide each of the first images into a plurality of second images based on a second preset size and a preset overlap ratio;
[0007] For any of the first images, processing each of the second images based on the distortion and deformation restoration model to obtain each first restoration image, and generating a second restoration image of the first image based on each of the first restoration images;
[0008] After all the second restored images are obtained, the second restored images are integrated to obtain a restored remote sensing image corresponding to the remote sensing image.
[0009] Preferably, the basic model is a pix2pixHD model, and the pixel-level supervised loss function is used to determine the pixel mean square error of each pixel between the model output result and the actual result, and generate an error penalty based on the size of the pixel mean square error. The error penalty is used to guide the adjustment of model parameters to minimize the pixel mean square error, and the pixel mean square error is positively correlated with the size of the error penalty.
[0010] Preferably, the training of the basic model based on the training data comprises:
[0011] Dividing the training data into training samples and test samples according to a preset ratio, and processing the training data based on a preset data enhancement strategy, wherein the proportion of the number of real restoration samples in the training data is less than the proportion of the number of simulated synthetic samples, and the data enhancement strategy includes random scaling, random flipping, and random parameter transformation, and the parameters include color, brightness, contrast, and saturation;
[0012] The basic model is trained based on the training samples and the test samples, and the model is back-propagated and updated according to the stochastic gradient descent method during the training process, and the learning rate of the model decays as the number of training rounds increases.
[0013] Preferably, the training data further includes non-repaired samples, each of the real repaired samples and the simulated synthetic samples is an image pair consisting of a distorted image and a repaired image, and the non-repaired samples are an image pair consisting of two identical non-distorted images;
[0014] After integrating the second restored images to obtain the restored remote sensing image corresponding to the remote sensing image, the method further includes:
[0015] When there is a pixel difference between the remote sensing image and the second restoration image, annotating a first restoration area of the second restoration image based on a binary mask, performing contour extraction on the first restoration area, generating a distortion detection result, and sending the distortion detection result to a preset terminal;
[0016] When a correct sample sent by the preset terminal based on the distortion detection result is detected, the correct sample is added to the real repair sample, and the distortion repair model is optimized based on the training data.
[0017] Preferably, the repaired image in the real repair sample is marked with at least one target box of a preset size, and the framed areas corresponding to all the target boxes cover the second repair area of the repaired image.
[0018] Preferably, the types of the random deformation operations for generating the simulated synthetic samples include wave distortion and extrusion deformation, and the selection probability of each random deformation operation is the same and independent.
[0019] Preferably, the step of generating a second restored image of the first image based on each of the first restored images comprises:
[0020] Dividing the overlapping area of the first restored image into a plurality of overlapping sub-areas, and setting a weight for each overlapping sub-area according to the distance from the image center of the first restored image, wherein the weight of the overlapping sub-area that is farther from the image center is greater;
[0021] A weighted calculation is performed on the overlapping areas of the first restored images to obtain a second restored image.
[0022] According to a second aspect, a device for repairing distortion and deformation of a remote sensing image is provided, the device comprising:
[0023] A training module is used to train the basic model based on training data using a generative adversarial network model as a basic model to obtain a distortion and deformation repair model, wherein the training data includes real repair samples and simulated synthetic samples, wherein the real repair samples are obtained through manual annotation in an actual operation process, and the simulated synthetic samples are obtained after random deformation operations are performed on real non-distorted and deformed images, wherein the constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples, and a pixel-level supervised loss function is set in the basic model, wherein the pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and an error penalty is generated according to the size of the pixel error, and the error penalty is used to guide the adjustment of model parameters to minimize the pixel error;
[0024] an acquisition module, used for acquiring a remote sensing image to be restored, dividing the remote sensing image into at least one first image based on a first preset size, and dividing each of the first images into a plurality of second images based on a second preset size and a preset overlap rate;
[0025] A first restoration module, configured to process each of the second images based on the distortion and deformation restoration model for any of the first images to obtain each first restoration image, and generate a second restoration image of the first image based on each of the first restoration images;
[0026] The second restoration module is used to integrate the second restoration images after obtaining all the second restoration images to obtain a restored remote sensing image corresponding to the remote sensing image.
[0027] According to a third aspect, there is provided an electronic device, comprising a processor and a memory;
[0028] The processor is connected to the memory;
[0029] The memory is used to store executable program code;
[0030] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the steps of the method provided in the first aspect or any possible implementation manner of the first aspect.
[0031] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer-readable storage medium stores instructions, which, when the instructions are executed on a computer or a processor, cause the computer or the processor to execute the method provided in the first aspect or any possible implementation of the first aspect.
[0032] The method and device provided in the embodiments of this specification train a distortion repair model based on a generative adversarial network model, and generate a large number of simulated synthetic samples in the training data based on random deformation operations of real non-distorted images to meet the requirements of the generative adversarial network model for training samples. In addition to the conventional constraint loss function, an additional pixel-level supervised loss function is set to determine the pixel error of the training results, and the model parameters are optimized by minimizing the pixel error. Finally, the remote sensing image is repaired by the trained distortion repair model, and the distortion of the remote sensing image can be quickly and accurately repaired without manual preprocessing. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0034] Figure 1 It is a flowchart of a method for repairing distortion and deformation of a remote sensing image in one embodiment of the present specification.
[0035] Figure 2 It is a structural schematic diagram of a device for repairing distortion and deformation of a remote sensing image in one embodiment of the present specification.
[0036] Figure 3 It is a structural diagram of an electronic device in one embodiment of this specification. DETAILED DESCRIPTION
[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0038] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application, and different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments recorded. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, although the embodiment may not be clearly recorded in the following text.
[0039] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present application. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted or combined. In addition, features described in some examples may be combined in other examples.
[0040] See also Figure 1 , Figure 1 : is a flow chart of a method for repairing distortion and deformation of a remote sensing image provided in an embodiment of the present application. In the embodiment of the present application, the method includes:
[0041] S101, taking a generative adversarial network model as a basic model, training the basic model based on training data to obtain a distortion and deformation repair model.
[0042] Among them, the training data includes real repair samples and simulated synthetic samples. The real repair samples are obtained through manual annotation during the actual operation process, and the simulated synthetic samples are obtained after random deformation operations based on real non-distorted deformed images. The constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples. A pixel-level supervised loss function is set in the basic model. The pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and generate an error penalty according to the size of the pixel error. The error penalty is used to guide the adjustment of model parameters to minimize the pixel error.
[0043] The execution entity of this application can be a cloud server.
[0044] In the embodiments of this specification, the Generative Adversarial Networks (GAN) model is used as the basic model to train the distortion repair model. Considering that the training of the Generative Adversarial Network model requires a large number of training samples, and the remote sensing image itself is relatively large, it is necessary to manually annotate the distorted and deformed parts in the remote sensing image as training samples, which makes it difficult to obtain enough actual training samples to train the model. Therefore, in the training data of this application, in addition to some real repair samples, that is, samples obtained after artificial repair and annotation based on real distorted and deformed images, simulated synthetic samples will also be provided. The simulated synthetic samples are samples obtained after random deformation operations based on real non-distorted and deformed images, that is, real images that do not have distortion and deformation problems themselves, wherein the random deformation operations may include wave distortion (wave), squeeze deformation (pinch), etc. Each real repair sample and simulated synthetic sample is actually a group of images formed by the image before and after the distortion repair that are compared with each other, so as to train the model as input and output data respectively. The constrained loss function set in the model assigns different constrained loss function weights to real repair samples and simulated synthetic samples, so that the weight of real repair samples is greater than that of simulated synthetic samples, so that the model can obtain more supervision and guidance from real situations, and finally the trained model has better effect.
[0045] In addition, in addition to generating the constraint loss function of the adversarial network model itself, in the basic model, in order to align the model output results with the actual results at the pixel level, an additional pixel-level supervision loss function is added to calculate the pixel error of each pixel in the model output results and the actual results, and generate an error penalty based on this. The model parameters are optimized according to the error penalty to minimize the pixel error and guide the model to output the same result as the actual result. Among them, the pixel-level supervision loss function can be selected from cross entropy loss, Dice loss, mean square error loss, etc. The error penalty can essentially be understood as the quantized value of the pixel error. The larger the error penalty, the more inaccurate the result. Under the guidance of the error penalty, the model will determine the minimum value of the pixel error according to the pixel-level supervision loss function, and optimize the model with the parameters at the minimum value.
[0046] In one possible implementation, the base model is a pix2pixHD model, and the pixel-level supervised loss function is used to determine the pixel mean square error of each pixel between the model output result and the actual result, and to generate an error penalty based on the size of the pixel mean square error. The error penalty is used to guide the adjustment of model parameters to minimize the pixel mean square error, and the pixel mean square error is positively correlated with the size of the error penalty.
[0047] In the embodiments of this specification, the basic model can specifically use the pix2pixHD model, which is an advanced version of the pix2pix model, which is mainly designed to generate high-resolution images and allows semantic editing of the generated images. The pixel-level supervision loss function can specifically use the mean square error loss function, through which the mean square error loss function is used to calculate the pixel mean square error of each pixel between the output result of the model and the actual result, so as to obtain the corresponding error penalty, and use this to guide the adjustment of the model parameters to minimize the pixel mean square error.
[0048] In one possible implementation, the training of the basic model based on training data includes:
[0049] Dividing the training data into training samples and test samples according to a preset ratio, and processing the training data based on a preset data enhancement strategy, wherein the proportion of the number of real restoration samples in the training data is less than the proportion of the number of simulated synthetic samples, and the data enhancement strategy includes random scaling, random flipping, and random parameter transformation, and the parameters include color, brightness, contrast, and saturation;
[0050] The basic model is trained based on the training samples and the test samples, and the model is back-propagated and updated according to the stochastic gradient descent method during the training process, and the learning rate of the model decays as the number of training rounds increases.
[0051] In an embodiment of the present specification, the cloud server will first divide the data according to a preset ratio (for example, 8:2), and divide the training data into training samples and test samples. The training samples are used to train the model, and the test samples are used to detect the effectiveness of the model. Then the training samples are set to a fixed size and data enhancement is performed. The data enhancement strategy can be set according to the specific situation. For example, the training samples can be randomly enlarged or reduced (randomly selected between 0.5-2 times) through multi-scale training to adapt to images of different sizes. The training samples can also be randomly flipped horizontally, vertically, etc. The training samples can also be randomly color-changed to change the brightness, contrast, saturation, etc. of the data (randomly select transformation parameters between 0-0.5) to avoid overfitting of the model and guide the model to focus on more robust features. When performing the above data enhancement, it is necessary to apply exactly the same operation to the paired pre- and post-repair data. In the early stage of training, the proportion of real repair samples is small. For example, the ratio of real repair samples to simulated synthetic samples can be 1:50. However, in the weight coefficient of the constrained loss function, the weight of the real repair samples needs to be 10 times the weight of the simulated synthetic samples, so as to guide the model to focus on the real situation.
[0052] In addition, during the training process, the stochastic gradient descent method is also used to back-propagate the model. A total of 150 rounds of training are performed. The initial learning rate can be set to 0.0001, and the adam optimizer is used. As the number of training rounds increases, the learning rate can be decayed. For example, it can be decayed a little bit each time the number of training rounds increases, or it can be decayed once when a specified number of training rounds (such as 100 rounds and 130 rounds) is reached. The learning rate reduction rate for each decay can be 10 times. The specific process of the stochastic gradient descent method can include initializing model parameters; shuffling the training data set before each round of training; traversing each sample in the training data in each round; calculating the gradient of the loss function with respect to the model parameters, and updating the model parameters in the direction of the negative gradient according to the principle of gradient descent; and repeating the iteration.
[0053] In one possible implementation, the repaired image in the real repair sample is marked with at least one target box of a preset size, and the framed areas corresponding to all the target boxes cover the second repair area of the repaired image.
[0054] In the embodiments of the present specification, the performance of the generative adversarial network model is heavily dependent on the quality of the training samples, so the production of training data is crucial. The cloud server will obtain some images before and after manual restoration from the image data production process, and use them as images of two periods of the change detection task, and then manually mark the areas where changes have been detected, and perform sample annotation. In this way, the repaired restoration image will be annotated with a target frame of uniform size. The target frame is generally a square frame of 512*512 in size. The target frame will cover the changed area as much as possible, that is, the second restoration area where the distortion and deformation have changed. When a target frame cannot cover the second restoration area, multiple target frames will be used for annotation, and there may be overlaps between different target frames. At the same time, when annotating with a target frame, the restoration area is not always necessarily in the middle of the target frame, so as to avoid the mistaken belief that the restoration distortion and deformation always occur in the central area during the model training process.
[0055] In one possible implementation, the types of the random deformation operations for generating the simulated synthetic sample include wave distortion and extrusion deformation, and the selection probability of each random deformation operation is the same and independent.
[0056] In the embodiment of the present specification, it takes a lot of time to manually annotate the real distorted repair data, and the training of the generative adversarial network model requires a large number of samples. Therefore, the present application will also use simulated synthetic samples as training data, and use the simulated synthetic data and the real repair data as samples to train the model to alleviate the problem of insufficient samples. The specific generation process of simulated synthetic data can be, first, obtain a real non-distorted deformed image, cut the entire image into multiple small blocks with 512*512 as the setting value, and save it as a repaired sample. Then, for each small sample image, a random deformation operation is performed on it, that is, a region is randomly selected within the image range, and the width and height of the region are set to random values between a preset range (such as 100-200), and the region center, width and height parameters are all randomly generated to ensure the randomness of the deformation. The amplitude and frequency in the wave distortion are both random within the set range, so that the distortion effect is random. The deformation radius and extrusion strength of the extrusion deformation are also randomly generated, where the extrusion deformation can be inward extrusion and outward extrusion to form a shrinking and expanding effect. In addition, when performing random deformation operations on each image, the probability of each random deformation operation being selected is the same, and there is a possibility that multiple random deformation operations are selected at the same time. If there is an image that has not undergone any deformation, it is used as a negative sample to help the model learn to understand the characteristics of distorted images and avoid the model from generating incorrect outputs for areas that do not need to be repaired.
[0057] S102: Acquire a remote sensing image to be restored, divide the remote sensing image into at least one first image based on a first preset size, and divide each of the first images into a plurality of second images based on a second preset size and a preset overlap rate.
[0058] In the embodiment of the present specification, since the remote sensing image is usually very large, in order to ensure the detection and repair effect and speed of the super large image, the remote sensing image will be processed in blocks. Specifically, the cloud server will first obtain the metadata of the remote sensing image, which includes image resolution, size, number of bands, and the effective value range of each band. At the same time, the cloud server will construct a tif to save the image based on the metadata, and the geographic information of the saved image is consistent with the original image of the remote sensing image. Then, the cloud server will divide the remote sensing image according to the first preset size to obtain at least one first image, that is, to obtain at least one block, and calculate and obtain the coordinate range of each block, and then take the first image as a whole, and process each first image in turn in a loop, that is, after completing the repair process of the shadow area of a certain first image, continue to process the next first image. For each first image, the cloud server will continue to divide the first image according to the second preset size and the preset overlap rate to obtain multiple second images, that is, multiple patches, wherein the preset overlap rate is set to reduce the inaccuracy of the model prediction at the edge of each patch image, so that the overall accuracy of a block area can be improved by fusion area later.
[0059] S103: for any of the first images, process each of the second images based on the distortion and deformation restoration model to obtain each first restoration image, and generate a second restoration image of the first image based on each of the first restoration images.
[0060] In the embodiments of the present specification, for each first image, the cloud server will process each second image in the first image in turn according to the trained distortion and deformation repair model to obtain each first repaired image, and then splice and synthesize these first repaired images according to the coordinate range and overlap rate corresponding to each first repaired image to generate a second repaired image corresponding to the first image as a whole.
[0061] In one possible implementation, generating a second restored image of the first image based on each of the first restored images includes:
[0062] Dividing the overlapping area of the first restored image into a plurality of overlapping sub-areas, and setting a weight for each overlapping sub-area according to the distance from the image center of the first restored image, wherein the weight of the overlapping sub-area that is farther from the image center is greater;
[0063] A weighted calculation is performed on the overlapping areas of the first restored images to obtain a second restored image.
[0064] In the embodiments of this specification, the model has different repair results for different images. If different patches are simply spliced together, there will be obvious boundary differences, resulting in obvious discontinuity in the overall image. Therefore, in the process of splicing to generate the second repaired image, a feathering transition will be performed for the overlapping areas. Specifically, the overlapping area of the image will be divided into multiple sub-overlapping areas according to the distance from the center of the image, and the weight of each sub-overlapping area will be incrementally set according to the distance of each sub-overlapping area from the center of the image. Ultimately, the image of the overlapping area will determine the pixel value of each pixel in the overlapping area by weighted calculation, and finally splicing to generate the second repaired image.
[0065] S104: After obtaining all the second restored images, integrate the second restored images to obtain a restored remote sensing image corresponding to the remote sensing image.
[0066] In the embodiment of the present specification, after obtaining the second restored images of all the first images, all the second restored images are integrated and spliced according to the coordinate range of each first image, so as to finally obtain the restored remote sensing image corresponding to the remote sensing image.
[0067] In one possible implementation, the training data further includes non-repaired samples, each of the real repaired samples and the simulated synthetic samples is an image pair consisting of a distorted image and a repaired image, and the non-repaired samples are an image pair consisting of two identical non-distorted images;
[0068] After integrating the second restored images to obtain the restored remote sensing image corresponding to the remote sensing image, the method further includes:
[0069] When there is a pixel difference between the remote sensing image and the second restoration image, annotating a first restoration area of the second restoration image based on a binary mask, performing contour extraction on the first restoration area, generating a distortion detection result, and sending the distortion detection result to a preset terminal;
[0070] When a correct sample sent by the preset terminal based on the distortion detection result is detected, the correct sample is added to the real repair sample, and the distortion repair model is optimized based on the training data.
[0071] In the embodiment of the present specification, the training data may also include non-repaired samples, that is, samples without distorted images and no need to be repaired. These samples will be used as negative samples to help the model learn and understand the characteristics of distorted images, and avoid the model from generating erroneous outputs for areas that do not need to be repaired. In addition, the model trained using negative samples may also have the function of detecting distorted areas, that is, it may be able to determine whether there is a pixel difference between the remote sensing image and the second repaired image. If there is a pixel difference, it means that there is a distorted repaired area. The cloud server will mark the first repaired area in the second repaired image by a binary mask annotation method. For example, 0 indicates the same position as the input and output, 1 indicates the position repaired by the model, and the area composed of all 1s is the first repaired area. Then, the contour of the repaired area can be extracted according to the mask, and a distortion deformation detection result is generated. The distortion deformation detection result may include a remote sensing image and a second repaired image with the contour of the repaired area marked. The distortion deformation detection result will be sent to a preset terminal used by the verification personnel. The verification personnel will verify the identified deformed area to determine whether the area here is indeed distorted and deformed, and whether the model repair result is correct. If there are errors, they will be manually modified and adjusted, and then added to the real repair samples as new correct samples to further fine-tune the optimization model. If the model result is correct, it will also be directly used as a new correct sample and added to the real repair samples. In addition, as the number of added correct samples increases, the number of simulated synthetic samples can be gradually reduced to further improve the effect of model optimization.
[0072] In addition, since the restored remote sensing image is actually obtained by splicing the patches in each block, the binary mask annotation and contour extraction process can also be performed after all patches in a block are processed in a loop. After the binary mask is annotated, morphological filtering can be used on the mask to filter out small holes, and then the contour of the mask is extracted by the contour extraction algorithm, so that the contour effect is better.
[0073] The following will be combined with the attached Figure 2 , the distortion and deformation repair device of remote sensing images provided in the embodiment of the present application is introduced in detail. It should be noted that the attached Figure 2 The remote sensing image distortion and deformation repair device shown is used to implement the present application Figure 1 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 1 The embodiment shown.
[0074] See also Figure 2 , Figure 2Schematic diagram of a device for repairing distortion and deformation of remote sensing images provided in an embodiment of the present application. Figure 2 As shown, the device comprises:
[0075] A training module 201 is used to train the basic model based on training data using a generative adversarial network model as a basic model to obtain a distortion and deformation repair model, wherein the training data includes real repair samples and simulated synthetic samples, wherein the real repair samples are obtained through manual annotation in an actual operation process, and the simulated synthetic samples are obtained after random deformation operations are performed on real non-distorted and deformed images, wherein the constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples, and a pixel-level supervised loss function is set in the basic model, wherein the pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and an error penalty is generated according to the size of the pixel error, and the error penalty is used to guide the adjustment of model parameters to minimize the pixel error;
[0076] An acquisition module 202 is used to acquire a remote sensing image to be restored, divide the remote sensing image into at least one first image based on a first preset size, and divide each of the first images into a plurality of second images based on a second preset size and a preset overlap rate;
[0077] A first restoration module 203 is used for processing each of the second images based on the distortion and deformation restoration model for any of the first images to obtain each first restoration image, and generating a second restoration image of the first image based on each of the first restoration images;
[0078] The second restoration module 204 is used to integrate the second restoration images after obtaining all the second restoration images to obtain a restoration remote sensing image corresponding to the remote sensing image.
[0079] In one possible implementation, the base model is a pix2pixHD model, and the pixel-level supervised loss function is used to determine the pixel mean square error of each pixel between the model output result and the actual result, and to generate an error penalty based on the size of the pixel mean square error. The error penalty is used to guide the adjustment of model parameters to minimize the pixel mean square error, and the pixel mean square error is positively correlated with the size of the error penalty.
[0080] In one embodiment, the training module 201 is specifically used for:
[0081] Dividing the training data into training samples and test samples according to a preset ratio, and processing the training data based on a preset data enhancement strategy, wherein the proportion of the number of real restoration samples in the training data is less than the proportion of the number of simulated synthetic samples, and the data enhancement strategy includes random scaling, random flipping, and random parameter transformation, and the parameters include color, brightness, contrast, and saturation;
[0082] The basic model is trained based on the training samples and the test samples, and the model is back-propagated and updated according to the stochastic gradient descent method during the training process, and the learning rate of the model decays as the number of training rounds increases.
[0083] In one possible implementation, the training data further includes non-repaired samples, each of the real repaired samples and the simulated synthetic samples is an image pair consisting of a distorted image and a repaired image, and the non-repaired samples are an image pair consisting of two identical non-distorted images;
[0084] The second repair module 204 is specifically used for:
[0085] When there is a pixel difference between the remote sensing image and the second restoration image, annotating a first restoration area of the second restoration image based on a binary mask, performing contour extraction on the first restoration area, generating a distortion detection result, and sending the distortion detection result to a preset terminal;
[0086] When a correct sample sent by the preset terminal based on the distortion detection result is detected, the correct sample is added to the real repair sample, and the distortion repair model is optimized based on the training data.
[0087] In one possible implementation, the repaired image in the real repair sample is marked with at least one target box of a preset size, and the framed areas corresponding to all the target boxes cover the second repair area of the repaired image.
[0088] In one possible implementation, the types of the random deformation operations for generating the simulated synthetic sample include wave distortion and extrusion deformation, and the selection probability of each random deformation operation is the same and independent.
[0089] In one possible implementation, the first repair module 203 is specifically used for:
[0090] Dividing the overlapping area of the first restored image into a plurality of overlapping sub-areas, and setting a weight for each overlapping sub-area according to the distance from the image center of the first restored image, wherein the weight of the overlapping sub-area that is farther from the image center is greater;
[0091] A weighted calculation is performed on the overlapping areas of the first restored images to obtain a second restored image.
[0092] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented with the help of software and / or hardware. The "unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.
[0093] Each processing unit and / or module of the embodiments of the present application may be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or may be implemented by software that executes the functions described in the embodiments of the present application.
[0094] See also Figure 3 , which shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, the electronic device can be used to implement Figure 1 The method in the embodiment shown. Figure 3 As shown, the electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .
[0095] The communication bus 302 is used to realize the connection and communication between these components.
[0096] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.
[0097] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0098] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts of the entire electronic device 300, and executes various functions and processes data of the electronic device 300 by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics central processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.
[0099] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally be at least one storage device located away from the aforementioned processor 301. As Figure 3 As shown, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.
[0100] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the remote sensing image distortion and deformation repair application stored in the memory 305, and specifically perform the following operations:
[0101] A generative adversarial network model is used as a basic model, and the basic model is trained based on training data to obtain a distortion and deformation repair model, wherein the training data includes real repair samples and simulated synthetic samples, wherein the real repair samples are obtained through manual annotation in an actual operation process, and the simulated synthetic samples are obtained after random deformation operations are performed on real non-distorted and deformed images, and the constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples. A pixel-level supervised loss function is set in the basic model, and the pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and an error penalty is generated according to the size of the pixel error, and the error penalty is used to guide the adjustment of model parameters to minimize the pixel error;
[0102] Acquire a remote sensing image to be restored, divide the remote sensing image into at least one first image based on a first preset size, and divide each of the first images into a plurality of second images based on a second preset size and a preset overlap ratio;
[0103] For any of the first images, processing each of the second images based on the distortion and deformation restoration model to obtain each first restoration image, and generating a second restoration image of the first image based on each of the first restoration images;
[0104] After all the second restored images are obtained, the second restored images are integrated to obtain a restored remote sensing image corresponding to the remote sensing image.
[0105] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.
[0106] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0107] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0108] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0110] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.
[0112] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct related hardware, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0113] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A method for repairing distortion of remote sensing images, characterized in that: The method comprises: A generative adversarial network model is used as a basic model, and the basic model is trained based on training data to obtain a distortion and deformation repair model, wherein the training data includes real repair samples and simulated synthetic samples, wherein the real repair samples are obtained through manual annotation in an actual operation process, and the simulated synthetic samples are obtained after random deformation operations are performed on real non-distorted and deformed images, and the constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples. A pixel-level supervised loss function is set in the basic model, and the pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and an error penalty is generated according to the size of the pixel error, and the error penalty is used to guide the adjustment of model parameters to minimize the pixel error; Acquire a remote sensing image to be restored, divide the remote sensing image into at least one first image based on a first preset size, and divide each of the first images into a plurality of second images based on a second preset size and a preset overlap ratio; For any of the first images, processing each of the second images based on the distortion and deformation restoration model to obtain each first restoration image, and generating a second restoration image of the first image based on each of the first restoration images; After obtaining all the second restored images, integrating the second restored images to obtain a restored remote sensing image corresponding to the remote sensing image; Wherein, the training data also includes non-repaired samples, each of the real repaired samples and the simulated synthetic samples is an image pair consisting of a distorted image and a repaired image, and the non-repaired samples are an image pair consisting of two identical non-distorted images; After integrating the second restored images to obtain the restored remote sensing image corresponding to the remote sensing image, the method further includes: When there is a pixel difference between the remote sensing image and the second restoration image, annotating a first restoration area of the second restoration image based on a binary mask, performing contour extraction on the first restoration area, generating a distortion detection result, and sending the distortion detection result to a preset terminal; When a correct sample sent by the preset terminal based on the distortion detection result is detected, the correct sample is added to the real repair sample, and the distortion repair model is optimized based on the training data.
2. The method according to claim 1, characterized in that The basic model is the pix2pixHD model. The pixel-level supervised loss function is used to determine the pixel mean square error of each pixel between the model output result and the actual result, and to generate an error penalty based on the size of the pixel mean square error. The error penalty is used to guide the adjustment of model parameters to minimize the pixel mean square error. The pixel mean square error is positively correlated with the size of the error penalty.
3. The method according to claim 1, characterized in that The step of training the basic model based on the training data comprises: Dividing the training data into training samples and test samples according to a preset ratio, and processing the training data based on a preset data enhancement strategy, wherein the proportion of the number of real restoration samples in the training data is less than the proportion of the number of simulated synthetic samples, and the data enhancement strategy includes random scaling, random flipping, and random parameter transformation, and the parameters include color, brightness, contrast, and saturation; The basic model is trained based on the training samples and the test samples, and the model is back-propagated and updated according to the stochastic gradient descent method during the training process, and the learning rate of the model decays as the number of training rounds increases.
4. The method according to claim 1, characterized in that: The repaired image in the real repair sample is marked with at least one target box of a preset size, and the framed areas corresponding to all the target boxes cover the second repair area of the repaired image.
5. The method according to claim 1, characterized in that The types of the random deformation operations for generating the simulated synthetic samples include wave distortion and extrusion deformation, and the selection probability of each random deformation operation is the same and independent.
6. The method according to claim 1, characterized in that The step of generating a second restored image of the first image based on each of the first restored images comprises: Dividing the overlapping area of the first restored image into a plurality of overlapping sub-areas, and setting a weight for each overlapping sub-area according to the distance from the image center of the first restored image, wherein the weight of the overlapping sub-area that is farther from the image center is greater; A weighted calculation is performed on the overlapping areas of the first restored images to obtain a second restored image.
7. A device for repairing distortion and deformation of remote sensing images, characterized in that: The device comprises: A training module is used to train the basic model based on training data using a generative adversarial network model as a basic model to obtain a distortion and deformation repair model, wherein the training data includes real repair samples and simulated synthetic samples, wherein the real repair samples are obtained through manual annotation in an actual operation process, and the simulated synthetic samples are obtained after random deformation operations are performed on real non-distorted and deformed images, wherein the constraint loss function weight of the real repair samples is greater than the constraint loss function weight of the simulated synthetic samples, and a pixel-level supervised loss function is set in the basic model, wherein the pixel-level supervised loss function is used to determine the pixel error of each pixel point between the model output result and the actual result, and an error penalty is generated according to the size of the pixel error, and the error penalty is used to guide the adjustment of model parameters to minimize the pixel error; an acquisition module, used for acquiring a remote sensing image to be restored, dividing the remote sensing image into at least one first image based on a first preset size, and dividing each of the first images into a plurality of second images based on a second preset size and a preset overlap rate; A first restoration module, configured to process each of the second images based on the distortion and deformation restoration model for any of the first images to obtain each first restoration image, and generate a second restoration image of the first image based on each of the first restoration images; A second restoration module is used for, after obtaining all the second restoration images, integrating the second restoration images to obtain a restoration remote sensing image corresponding to the remote sensing image; Wherein, the training data also includes non-repaired samples, each of the real repaired samples and the simulated synthetic samples is an image pair consisting of a distorted image and a repaired image, and the non-repaired samples are an image pair consisting of two identical non-distorted images; The second repair module is specifically used for: When there is a pixel difference between the remote sensing image and the second restoration image, annotating a first restoration area of the second restoration image based on a binary mask, performing contour extraction on the first restoration area, generating a distortion detection result, and sending the distortion detection result to a preset terminal; When a correct sample sent by the preset terminal based on the distortion detection result is detected, the correct sample is added to the real repair sample, and the distortion repair model is optimized based on the training data.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, wherein the computer-readable storage medium has instructions stored therein, and when the instructions are executed on a computer or a processor, the computer or the processor executes the steps of the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Image restoration method of multi-residual-block conditional generative adversarial network
CN112419171A
Image deblurring algorithm based on knowledge distillation and deep neural network
CN114677304A
Remote sensing image conversion map migration method based on semi-supervised generative adversarial network
CN114792349A
Atmospheric turbulence image restoration method and system based on multi-scale generative adversarial network
CN117611456A
Image restoration method and device based on SRGAN-GAN, equipment and storage medium
CN117830148A
