Training method of image restoration model, image restoration method, equipment and product

Abstract

The embodiment of the invention provides a training method of an image restoration model, an image restoration method, equipment and a product, which are applied to the technical field of smart power grids. The training method comprises the following steps: acquiring training sample data, wherein the training sample data comprises a sample shielding image and a sample real image corresponding to the sample shielding image; based on the training sample data, training the initial image restoration model so as to alternately update parameters corresponding to a discriminator, a generator and a PPO strategy network in the initial image restoration model in different training rounds to obtain a to-be-determined image restoration model; the initial image restoration model is obtained after training of a preset training round based on training sample data; and when the to-be-determined image restoration model satisfies a preset condition, determining the to-be-determined image restoration model as a trained target image restoration model. The method achieves the technical effect of improving the image restoration precision.

Description

Technical Field

[0001] The present application relates to the field of smart grid technology, and in particular to a training method for an image restoration model, an image restoration method, a device, and a product. Background Art

[0002] In scenarios like drone inspections and maintenance of power equipment, drone images of the monitored area are often lost due to factors like occlusion and noise. These issues severely impact the ability to control potentially unauthorized drones in the monitored area, and in turn, the efficiency and safety of power grid monitoring.

[0003] In the prior art, the method for repairing occluded images mainly utilizes a convolutional neural network to learn the local features of the image and implements the repair of the occluded image based on the image classification method of the convolutional neural network.

[0004] Since the existing technology may cause fuzzy restoration when processing complex background images, the existing technology has a technical problem of low image restoration accuracy. Summary of the Invention

[0005] The embodiments of the present application provide a training method for an image restoration model, an image restoration method, an apparatus, and a product to achieve the technical effect of improving the accuracy of image restoration.

[0006] In a first aspect, an embodiment of the present application provides a method for training an image restoration model, comprising:

[0007] Acquire training sample data, where the training sample data includes a sample occluded image and a sample real image corresponding to the sample occluded image;

[0008] Based on the training sample data, the initial image restoration model is trained to alternately update the parameters corresponding to the discriminator, generator, and PPO strategy network in the initial image restoration model in different training rounds to obtain the image restoration model to be determined; the initial image restoration model is obtained after training for a preset number of training rounds based on the training sample data;

[0009] When the image restoration model to be determined meets a preset condition, the image restoration model to be determined is determined as the trained target image restoration model.

[0010] In one possible implementation, the initial image restoration model is trained based on the training sample data, so that the parameters corresponding to the discriminator, the generator, and the PPO strategy network in the initial image restoration model are alternately updated in different training rounds, including:

[0011] Encode the sample occluded image through the encoder of the initial image restoration model to obtain a latent vector corresponding to the sample occluded image;

[0012] The latent vector is reconstructed and repaired through the action instructions of the generator and the PPO strategy network to obtain the repaired image. The action instructions include the learning rate of the generator and the switching instructions of the generator's repair phase.

[0013] Based on the first prediction parameter update strategy, update the parameters of the generator;

[0014] Through the discriminator, a first score corresponding to the repaired image and a second score of the sample real image corresponding to the sample occluded image are calculated respectively;

[0015] Based on the second prediction parameter update strategy, update the parameters of the discriminator;

[0016] Based on the restored image and the sample real image, a first evaluation index and a second evaluation index are calculated; wherein the first evaluation index is used to measure the similarity between the restored image and the sample real image, and the second evaluation index is used to measure the error between the restored image and the sample real image;

[0017] The action instructions are updated through the PPO policy network and the state information sequence corresponding to the PPO policy network, and the parameters of the PPO policy network are updated based on the third prediction parameter update strategy; wherein the state information sequence includes state information of multiple image restoration stages, and each state information includes a first evaluation indicator, a second evaluation indicator, a first score, a second score, a potential vector and a stage identifier of the image restoration stage.

[0018] In one possible implementation, the latent vector is reconstructed and repaired using the generator and the action instructions of the PPO strategy network to obtain a repaired image, including:

[0019] Based on the learning rate in the action instruction and the restoration phase switching instruction, update the generator's learning rate and the phase identifier of the image restoration phase;

[0020] Based on the updated generator and the image restoration stage corresponding to the stage identifier, the latent vector is reconstructed and restored to obtain the restored image of the image restoration stage.

[0021] In a possible implementation, updating the action instruction using the PPO policy network and the state information sequence corresponding to the PPO policy network includes:

[0022] When the generator parameter update is detected and the cumulative update times of the discriminator parameters reach the preset cumulative times, the state information sequence corresponding to the PPO strategy network is obtained;

[0023] Based on the state information sequence and the length of the repair time of the generator's image repair phase, the update amplitude of the PPO strategy network is calculated;

[0024] Update the parameters of the PPO policy network according to the update amplitude of the PPO policy network and the preset learning rate of the PPO policy network;

[0025] Based on the PPO strategy network with updated parameters and the state information sequence, the action instructions are updated.

[0026] In one possible implementation, the update amplitude of the PPO strategy network is calculated based on the state information sequence and the length of the repair time of the image repair phase of the generator, including:

[0027] Based on the first evaluation index, the second evaluation index, the first score, the second score and the repair time length in the state information sequence, the reward value of the PPO strategy network after the current batch of training sample data is input;

[0028] Based on the reward value and state information sequence, the advantage value of the PPO strategy network is calculated;

[0029] Based on the advantage value and reward value, the update amplitude of the PPO strategy network is calculated.

[0030] In one possible implementation, updating the parameters of the generator based on the first prediction parameter update strategy includes:

[0031] When the cumulative update times of the discriminator parameters reach the preset cumulative times, the total loss value of the generator is calculated based on the repaired image and the sample real image; the total loss value includes the reconstruction loss value and the adversarial loss value;

[0032] Update the parameters of the generator based on the total loss value and the generator's learning rate;

[0033] Set the cumulative update times of the discriminator parameters to the initial value.

[0034] In one possible implementation, updating the parameters of the discriminator based on the second prediction parameter updating strategy includes:

[0035] When the cumulative update number of the discriminator parameters is lower than the preset cumulative number, the discriminator calculates the first score of the repaired image and the discriminant loss value corresponding to the second score of the sample real image; the discriminant loss value is the weighted value of the gradient penalty value of the discriminator and the distance loss value;

[0036] Update the discriminator parameters based on the loss value and the discriminator's preset learning rate.

[0037] In one possible embodiment, the preset conditions include at least any one of the following: the cumulative rounds of iterative training are greater than or equal to the preset rounds; the loss value of the image restoration model to be determined is less than or equal to the preset loss value; the first evaluation index of the image restoration model to be determined is greater than the first preset value, and the second evaluation index of the image restoration model to be determined is greater than the second preset value.

[0038] In a second aspect, an embodiment of the present application provides an image restoration method, comprising:

[0039] Obtain the image to be repaired;

[0040] The image to be repaired is input into the image repair model to obtain a repaired image corresponding to the image to be repaired; wherein the image repair model is obtained by training according to the training method of the image repair model of any one of the first aspects.

[0041] In a third aspect, an embodiment of the present application provides a training device for an image restoration model, comprising:

[0042] An acquisition module is used to acquire training sample data, where the training sample data includes a sample occluded image and a sample real image corresponding to the sample occluded image;

[0043] The first processing module is configured to train an initial image restoration model based on the training sample data, so as to alternately update the parameters corresponding to the discriminator, the generator, and the PPO strategy network in the initial image restoration model in different training rounds to obtain a to-be-determined image restoration model; the initial image restoration model is obtained after training for a preset number of training rounds based on the training sample data;

[0044] The second processing module is used to determine the image restoration model to be determined as the trained target image restoration model when the image restoration model to be determined meets a preset condition.

[0045] In a possible implementation, the first processing module is further configured to:

[0046] Encode the sample occluded image through the encoder of the initial image restoration model to obtain a latent vector corresponding to the sample occluded image;

[0047] The latent vector is reconstructed and repaired through the action instructions of the generator and the PPO strategy network to obtain the repaired image. The action instructions include the learning rate of the generator and the switching instructions of the generator's repair phase.

[0048] Based on the first prediction parameter update strategy, update the parameters of the generator;

[0049] Through the discriminator, a first score corresponding to the repaired image and a second score of the sample real image corresponding to the sample occluded image are calculated respectively;

[0050] Based on the second prediction parameter update strategy, update the parameters of the discriminator;

[0051] Based on the restored image and the sample real image, a first evaluation index and a second evaluation index are calculated; wherein the first evaluation index is used to measure the similarity between the restored image and the sample real image, and the second evaluation index is used to measure the error between the restored image and the sample real image;

[0052] The action instructions are updated through the PPO policy network and the state information sequence corresponding to the PPO policy network, and the parameters of the PPO policy network are updated based on the third prediction parameter update strategy; wherein the state information sequence includes state information of multiple image restoration stages, and each state information includes a first evaluation indicator, a second evaluation indicator, a first score, a second score, a potential vector and a stage identifier of the image restoration stage.

[0053] In a possible implementation, the first processing module is further configured to:

[0054] Based on the learning rate in the action instruction and the restoration phase switching instruction, update the generator's learning rate and the phase identifier of the image restoration phase;

[0055] Based on the updated generator and the image restoration stage corresponding to the stage identifier, the latent vector is reconstructed and restored to obtain the restored image of the image restoration stage.

[0056] In a possible implementation, the first processing module is further configured to:

[0057] When the generator parameter update is detected and the cumulative update times of the discriminator parameters reach the preset cumulative times, the state information sequence corresponding to the PPO strategy network is obtained;

[0058] Based on the state information sequence and the length of the repair time of the generator's image repair phase, the update amplitude of the PPO strategy network is calculated;

[0059] Update the parameters of the PPO policy network according to the update amplitude of the PPO policy network and the preset learning rate of the PPO policy network;

[0060] Based on the PPO strategy network with updated parameters and the state information sequence, the action instructions are updated.

[0061] In a possible implementation, the first processing module is further configured to:

[0062] Based on the first evaluation index, the second evaluation index, the first score, the second score and the repair time length in the state information sequence, the reward value of the PPO strategy network after the current batch of training sample data is input;

[0063] Based on the reward value and state information sequence, the advantage value of the PPO strategy network is calculated;

[0064] Based on the advantage value and reward value, the update amplitude of the PPO strategy network is calculated.

[0065] In a possible implementation, the first processing module is further configured to:

[0066] When the cumulative update times of the discriminator parameters reach the preset cumulative times, the total loss value of the generator is calculated based on the repaired image and the sample real image; the total loss value includes the reconstruction loss value and the adversarial loss value;

[0067] Update the parameters of the generator based on the total loss value and the generator's learning rate;

[0068] Set the cumulative update times of the discriminator parameters to the initial value.

[0069] In a possible implementation, the first processing module is further configured to:

[0070] When the cumulative update number of the discriminator parameters is lower than the preset cumulative number, the discriminator calculates the first score of the repaired image and the discriminant loss value corresponding to the second score of the sample real image; the discriminant loss value is the weighted value of the gradient penalty value of the discriminator and the distance loss value;

[0071] Update the discriminator parameters based on the loss value and the discriminator's preset learning rate.

[0072] In one possible embodiment, the preset conditions include at least any one of the following: the cumulative rounds of iterative training are greater than or equal to the preset rounds; the loss value of the image restoration model to be determined is less than or equal to the preset loss value; the first evaluation index of the image restoration model to be determined is greater than the first preset value, and the second evaluation index of the image restoration model to be determined is greater than the second preset value.

[0073] In a fourth aspect, an embodiment of the present application provides an image restoration device, comprising:

[0074] An acquisition module, used to acquire the image to be repaired;

[0075] A processing module is used to input the image to be repaired into the image repair model to obtain a repaired image corresponding to the image to be repaired; wherein the image repair model is obtained by training according to the training method of the image repair model of any one of the first aspects.

[0076] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0077] Memory stores computer-executable instructions;

[0078] The processor executes the computer-executable instructions stored in the memory, so that the processor executes possible implementations of the first aspect or the second aspect as described above.

[0079] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement possible implementation methods as described in the first or second aspect above.

[0080] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements possible implementation methods in the first or second aspect above.

[0081] The training method, image restoration method, device and product of the image restoration model provided by the embodiments of the present application. The training method obtains training sample data, trains the initial image restoration model based on the training sample data, and alternately updates the parameters of the discriminator, generator and PPO strategy network in the initial image restoration model in different training rounds, thereby obtaining the image restoration model to be determined; determines whether the image restoration model to be determined meets the preset conditions, and when it is determined that it meets the conditions, determines the image restoration model to be determined as the target image restoration model. Compared with the prior art, the present application adds a PPO strategy network on the basis of the conventional encoder-generator-discriminator image reconstruction framework, alternately updates the parameters of the generator, discriminator and PPO strategy network in different training rounds, and achieves the purpose of model training optimization using the PPO strategy network. By continuously optimizing the image restoration effect of the generator, the accuracy of the initial image restoration model in image restoration is further improved, thereby achieving the technical effect of improving the accuracy of image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0083] Figure 1 Schematic diagram of the training process of the image restoration model provided in this application Figure 1 ;

[0084] Figure 2 Schematic diagram of the training process of the image restoration model provided in this application Figure 2 ;

[0085] Figure 3 A schematic diagram of the structure of an image restoration model provided in this application;

[0086] Figure 4A flowchart of a UAV image restoration model training method provided in this application;

[0087] Figure 5 Schematic diagram showing the results of drone image restoration provided for this application;

[0088] Figure 6 Schematic diagram of the curve changes of the structural similarity index and peak signal-to-noise ratio provided in this application;

[0089] Figure 7 A flowchart of the image restoration method provided in this application;

[0090] Figure 8 A schematic diagram of the structure of the training device for the image restoration model provided in this application;

[0091] Figure 9 A schematic diagram of the structure of the image restoration device provided in this application;

[0092] Figure 10 This is a schematic diagram of the structure of the electronic device provided in this application.

[0093] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0094] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0095] First, let’s explain the relevant terms of this application design:

[0096] Proximal Policy Optimization (PPO): The PPO policy network is a policy optimization method based on reinforcement learning that ensures training stability by limiting the step size of policy updates.

[0097] Kullback-Leibler (KL) divergence: KL divergence is an indicator of the asymmetry of the difference between two probability distributions.

[0098] Structural Similarity Index Measure (SSIM): SSIM is an indicator used to measure the similarity between two images in terms of brightness, contrast, and structure. The range is between [-1, 1], where 1 means they are exactly the same.

[0099] Peak Signal-to-Noise Ratio (PSNR): PSNR is an image quality assessment metric based on mean square error, which measures the ratio of the maximum possible power of the signal or original image to the noise power.

[0100] In the existing technology, the main method for repairing occluded images is: collecting a large number of occluded images and the real images corresponding to the occluded images to form training sample data; using convolutional neural networks to extract image features of the occluded images, and training the image repair model in combination with the real images corresponding to the occluded images; when performing image repair, using the convolutional neural network model you obtained to process the input occluded image to obtain the real image prediction result corresponding to the image.

[0101] However, the image restoration method in the existing technology mainly uses deep learning models to predict the classification results. When the occluded image being processed has a complex image background, the features extracted by the feature extraction method are highly chaotic, and the final restored image is blurred, making it impossible to perform high-precision image restoration. Therefore, the technical problem of low image restoration accuracy exists in the existing technology.

[0102] In response to the above technical problems, the present application proposes the following technical concept: combining the PPO strategy network to train the image restoration model. Specifically: adding the PPO strategy network to the common image reconstruction framework, through the combination of the PPO strategy network, the discriminator and the generator, the parameters are updated alternately in different training rounds, thereby realizing the dynamic adjustment of the parameters of the PPO strategy network, the discriminator and the generator; after different rounds of training, the image restoration model to be determined is obtained, and the preset conditions are used to realize the determination of the target image restoration model. Compared with the existing technology, the image restoration model training method of the present application optimizes the model training process by introducing the PPO strategy network, thereby achieving the technical effect of improving the accuracy of image restoration.

[0103] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0104] Figure 1 Schematic diagram of the training process of the image restoration model provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0105] S101 : Acquire training sample data, where the training sample data includes a sample occluded image and a sample real image corresponding to the sample occluded image.

[0106] In this step, the methods for obtaining training sample data include: adding artificial occlusions to clear sample real images to generate paired data; taking two sets of images of the same scene through physical occlusion to obtain paired data; collecting existing public data sets to obtain multiple paired data; a pair of paired data includes a sample occluded image and the sample real image corresponding to the sample occluded image.

[0107] For example, training sample data is collected from drone flight images in a power grid scenario, and training sample data is generated by simulating occlusion. The specific method is as follows:

[0108] S1011. Collect real images from multiple angles generated by the drone flying in the power grid scene.

[0109] S1012: Perform occlusion simulation on each real image to obtain multiple occlusion images.

[0110] S1013: Combine each occluded image with its corresponding real image to obtain training sample data corresponding to the UAV.

[0111] S102. Based on the training sample data, the initial image restoration model is trained to alternately update the corresponding parameters of the discriminator, generator, and PPO strategy network in the initial image restoration model in different training rounds to obtain the image restoration model to be determined.

[0112] In this step, the initial image restoration model is obtained based on the training sample data after a preset number of training rounds. The purpose of obtaining the initial image restoration model through the preset training rounds is to preliminarily determine the parameters of the initial image restoration model to facilitate subsequent model training.

[0113] Optionally, a possible implementation method of obtaining an initial image restoration model based on the training sample data after a preset training round is:

[0114] S1021. Divide the training sample data into multiple batches.

[0115] S1022. In each training round, perform model training for each batch.

[0116] S1023. For each batch, when the cumulative number of updates of the discriminator parameters is less than 5 times, freeze the parameters of the generator and encoder, enable the discriminator to perform gradient calculation, and update the discriminator parameters by backpropagation.

[0117] S1024. When the cumulative number of parameter updates of the discriminator reaches 5, freeze the parameters of the discriminator, enable the generator and encoder to perform gradient calculation, use backpropagation to update the parameters of the generator and encoder, and reset the cumulative number of parameter updates of the discriminator.

[0118] For example, the current training sample data size is 6000, the number of divided batches is 600, the preset training rounds is 50, and the ratio of parameter updates between the discriminator and the generator is 5 to 1. Then the total number of parameter updates of the discriminator is 2500 times, and the total number of parameter updates of the generator is 500 times.

[0119] In this step, the specific implementation method of alternately updating the corresponding parameters of the discriminator, generator and PPO strategy network in the initial image restoration model in different training rounds is as follows: Figure 2 Further explanation is given in the embodiment shown and no redundant description is given here.

[0120] S103. When the image restoration model to be determined meets a preset condition, the image restoration model to be determined is determined as a trained target image restoration model.

[0121] In this step, the preset conditions include at least any one of the following: the cumulative rounds of iterative training are greater than or equal to the preset rounds; the loss value of the image restoration model to be determined is less than or equal to the preset loss value; the first evaluation index of the image restoration model to be determined is greater than the first preset value, and the second evaluation index of the image restoration model to be determined is greater than the second preset value.

[0122] Among them, the first evaluation indicator refers to the structural similarity index, which is used to measure the similarity between the sample real image and the repaired image in brightness, contrast and structure; the second evaluation indicator refers to the peak signal-to-noise ratio, which is used to measure the pixel-level error between the sample real image and the repaired image, indicating the quality of image reconstruction.

[0123] The training method of the image restoration model provided in the embodiment of the present application obtains training sample data, trains the initial image restoration model based on the training sample data, and alternately updates the parameters of the discriminator, generator, and PPO strategy network in the initial image restoration model in different training rounds, thereby obtaining the image restoration model to be determined; determines whether the image restoration model to be determined meets the preset conditions, and when it is determined that it meets the conditions, determines the image restoration model to be determined as the target image restoration model. Compared with the prior art, the present application adds a PPO strategy network on the basis of the conventional encoder-generator-discriminator image reconstruction framework, alternately updates the parameters of the generator, discriminator, and PPO strategy network in different training rounds, and achieves the purpose of optimizing model training using the PPO strategy network. By continuously optimizing the image restoration effect of the generator, the accuracy of the initial image restoration model in image restoration is further improved, thereby achieving the technical effect of improving the accuracy of image restoration.

[0124] Figure 2 Schematic diagram of the training process of the image restoration model provided in this application Figure 2 Based on the above embodiment, this embodiment further explains the training process of the image restoration model to be determined in step 102, as shown in FIG. Figure 2 As shown, the method includes:

[0125] S201. Encode the sample occluded image through the encoder of the initial image restoration model to obtain a latent vector corresponding to the sample occluded image.

[0126] In this step, the method of obtaining the latent vector through the encoder of the initial image restoration model is mainly as follows: using the encoder of the initial image model to process the sample occluded image to obtain the mean and logarithmic variance of the latent vector corresponding to the sample occluded image; reparameterizing based on the mean and logarithmic variance of the latent vector to obtain the latent vector of the sample occluded image.

[0127] For example, the method of generating a latent vector for an occluded image of a drone is as follows:

[0128] S2011. Input a sample occluded image of a drone into the encoder of the initial image restoration model, map the sample occluded image into statistical parameters of the latent space, and obtain the mean and logarithmic variance of the latent vector.

[0129] In this step, the mean of the latent vector refers to the center position of the image feature of the sample occluded image, encoding the global structure as well as the local features; the logarithmic variance refers to the logarithm of the variance of the image feature, reflecting the uncertainty of the latent space.

[0130] S2012. Using the reparameterization technique, we sample from the standard normal distribution based on the mean and log variance to obtain a latent vector, providing the generator with an input that is both random and contains global structural information.

[0131] S202: Perform image reconstruction and restoration on the latent vector through the action instructions of the generator and the PPO strategy network to obtain a restored image.

[0132] In this step, the action instructions include the learning rate of the generator and the repair phase switching instructions of the generator.

[0133] Optionally, through the action instructions of the generator and the PPO strategy network, the latent vector is reconstructed and repaired to obtain a repaired image. One possible implementation method is:

[0134] S2021. Based on the learning rate in the action instruction and the restoration phase switching instruction, update the learning rate of the generator and the phase identifier of the image restoration phase.

[0135] In this step, the image restoration stage of the generator includes three restoration stages, namely the global structure restoration stage, the local detail enhancement stage, and the post-processing optimization stage.

[0136] Exemplarily, the learning rate in the action instruction is Δη, which is used to indicate the magnitude by which the learning rate of the current generator needs to be adjusted; the action switching instruction in the action instruction indicates that the image restoration stage needs to switch to the next stage. The learning rate of the current generator is η, and the image restoration stage of the current generator is identified as 0, which refers to the global structure restoration stage. The method for updating the learning rate and stage identification of the generator based on the action instruction is: updating the learning rate of the generator, and obtaining the updated learning rate is: η+Δη; updating the image restoration stage of the generator, and obtaining the updated image restoration stage identification as 1, which refers to the local detail enhancement stage.

[0137] In this step, the purpose of adjusting the learning rate based on action instructions is to increase the learning rate through action instructions to accelerate the convergence speed of the generator when it is detected that the occlusion of the sample occluded image is serious; when it is detected that the sample occluded image is in the detail enhancement stage, it is necessary to reduce the learning rate based on the action instructions to ensure the stability of the generator.

[0138] S2022. Based on the updated generator and the image restoration stage corresponding to the stage identifier, perform image reconstruction and restoration on the latent vector to obtain a restored image of the image restoration stage.

[0139] In this step, the global structure restoration stage is used to reconstruct the overall outline of the sample occluded image and generate a low-resolution feature map; the local detail enhancement stage is used to refine the local details in the low-resolution feature map to obtain a high-resolution image; the post-processing optimization stage is used to reduce noise and sharpen edges on the high-resolution image to optimize the overall visual effect and obtain a restored image corresponding to the sample occluded image.

[0140] Exemplarily, the method of reconstructing and repairing the image of the full vector using the updated generator and the image restoration stage corresponding to the stage identifier is: when the stage identifier is 0, the image restoration stage is determined to be the global structure restoration stage, and the obtained restoration image is a low-resolution feature map; when the stage identifier is 1, the image restoration stage is determined to be the local detail enhancement stage, and the obtained restoration image is a high-resolution image; when the stage identifier is 2, the image restoration stage is determined to be the post-processing optimization stage, and the obtained restoration image is an optimized high-resolution image.

[0141] S203. Update the parameters of the generator based on the first prediction parameter update strategy.

[0142] Optionally, based on the first prediction parameter updating strategy, a possible implementation method for updating the parameters of the generator is:

[0143] S2031. When the cumulative update times of the discriminator parameters reach the preset cumulative times, the total loss value of the generator is calculated based on the repaired image and the sample real image.

[0144] In this step, the total loss value includes the reconstruction loss value and the adversarial loss value. The total loss value is used to measure the image restoration effect of the generator.

[0145] Exemplarily, the preset cumulative number is set to 10. When the cumulative number of updates of the discriminator parameters reaches 10, the total loss value of the generator is calculated based on the repaired image of the current generator and the real image of the sample.

[0146] The total loss value is calculated as shown in Formula 1:

[0147]

[0148] Among them, L Total Refers to the total loss value, L rec Refers to the reconstruction loss value, L KL Refers to the KL divergence loss value, L GRefers to the adversarial loss; α, β, and γ are the weight coefficients corresponding to each loss value. The total loss value includes not only the reconstruction loss and the adversarial loss, but also the KL divergence loss. The reconstruction loss is used to enforce pixel-level or feature-level alignment between the inpainted image and the real sample image, ensuring that the inpainted image is structurally consistent with the real sample image. The KL divergence loss constrains the distribution of the latent vector output by the encoder to be close to the standard normal distribution, preventing the collapse of the latent space and ensuring the diversity of the generated results. The adversarial loss is used to improve the realism of the inpainted image through feedback from the discriminator, compensating for the blurring caused by pure reconstruction loss.

[0149] In this step, the weight coefficients corresponding to each loss value in the total loss value can be dynamically adjusted through the PPO strategy network. For example, in the global structure restoration stage, it is necessary to prioritize improving the accuracy of pixel-level reconstruction, so the weight coefficient α of the reconstruction loss value can be set to a larger value; in the local detail enhancement stage, the weight coefficient γ of the adversarial loss value can be appropriately increased to achieve the purpose of strengthening the adversarial training effect.

[0150] S2032. Update the parameters of the generator based on the total loss value and the learning rate of the generator.

[0151] In this step, the parameters of the generator can be updated based on the total loss value and the learning rate by: calculating the gradient of the generator parameters based on the total loss value; scaling the gradient based on the learning rate of the generator; and adjusting the parameters of the generator based on the calculated gradient and learning rate.

[0152] S2033. Set the cumulative update times of the discriminator parameters to the initial value.

[0153] In this step, the parameter update of the discriminator and the parameter update of the generator are performed alternately. When the cumulative number of parameter updates of the discriminator reaches the preset cumulative number, the parameter update of the discriminator is suspended and the parameter update of the generator is selected; when the parameter update of the generator is completed, the cumulative number of parameter updates of the discriminator is set to the initial value, prompting the discriminator to restart the parameter update.

[0154] For example, the number of training batches for the model in the current training round is 100. A single discriminator parameter update or a single generator parameter update refers to the process of inputting one batch of training sample data into the initial image restoration model for training. The preset cumulative number of iterations is set to 9. In the model training batches 1-9, the discriminator parameters are updated 9 times, and in the model training batch 10, the generator parameters are updated once. In 100 training batches, the discriminator parameters are updated 90 times, and the generator parameters are updated 10 times.

[0155] S204 , using the discriminator, respectively calculating a first score corresponding to the restored image and a second score of the sample real image corresponding to the sample occluded image.

[0156] In this step, the first score and the second score can be used to calculate the discriminator loss value. The larger the value of the first score, the closer the discriminator task repair image is to the real sample image. It is used to measure the authenticity of the repaired image and drive the generator optimization. Ideally, the second score should be significantly higher than the first score. The second score is used to anchor the real data distribution and constrain the discriminator from being overconfident.

[0157] For example, if the first score calculated by the discriminator is -2.5 and the second score is 2.1, it indicates that the discriminator determines that the current restored image is false and the restoration purpose is not achieved.

[0158] S205: Update the parameters of the discriminator based on the second prediction parameter updating strategy.

[0159] Optionally, based on the second prediction parameter updating strategy, a possible implementation method for updating the parameters of the discriminator is:

[0160] S2051. When the cumulative update times of the discriminator parameters are lower than the preset cumulative times, the discriminator calculates a first score of the restored image and a discrimination loss value corresponding to a second score of the sample real image.

[0161] In this step, the discriminant loss value is the weighted value of the discriminator's gradient penalty value and the distance loss value.

[0162] For example, the gradient penalty value is calculated as shown in Formula 2:

[0163]

[0164] Among them, x real Refers to the sample real image, x fake refers to the repaired image, Refers to the random linear interpolation of the sample real image and the repaired image, Refers to the gradient of the discriminator's output of the interpolated image with respect to the input, and ||·||2 refers to the L2 norm of the gradient, which is used to measure the gradient strength.

[0165] The distance loss value is calculated as shown in Formula 3:

[0166]

[0167] Among them, D(x real ) refers to the second score corresponding to the sample real image, D(x fake ) refers to the first score corresponding to the repaired image.

[0168] The calculated gradient penalty value and distance loss value are weighted to obtain the discriminator loss value.

[0169] S2052: Update the parameters of the discriminator based on the loss value and the preset learning rate of the discriminator.

[0170] In this step, the discriminator parameters can be updated by performing backpropagation based on the discriminator loss value to calculate the gradient of the discriminator parameters; and updating the discriminator parameters based on the optimizer and a preset learning rate. The optimizer can be either Adam or SGD.

[0171] For example, the loss value of the discriminator is L D , the discriminator parameter is F, the preset learning rate is n, and the calculated gradient of the discriminator parameter is ▽ F L D , then the method to update the parameters of the discriminator is: Fn·▽ F L D →F.

[0172] S206: Calculate a first evaluation index and a second evaluation index based on the restored image and the sample real image.

[0173] In this step, the first evaluation metric is used to measure the similarity between the inpainted image and the real sample image, and the second evaluation metric is used to measure the error between the inpainted image and the real sample image. The first evaluation metric is the structural similarity index, which is used to compare the similarity between the inpainted image and the real sample image in terms of brightness, contrast, and structure. The second evaluation metric is the peak signal-to-noise ratio, which is used to measure the pixel-level difference between the inpainted image and the real sample image through the mean squared error.

[0174] Exemplarily, the method for calculating the first evaluation index is shown in Formula 4:

[0175]

[0176] Among them, x refers to the sample real image, y refers to the repaired image, μ x ,μ y They refer to the brightness mean of the sample real image and the restored image respectively; σ x ,σ y They refer to the contrast standard deviation of the two images; σ xy Refers to the covariance of the structural similarity of two images; C1 and C2 refer to stability constants; SSIM(x,y) is the first evaluation indicator.

[0177] The method for calculating the second evaluation index is shown in Formula 5:

[0178]

[0179] Among them, PSNR refers to the second evaluation index, MAX I is the maximum value of the pixel; MSE refers to the mean square error between the two images.

[0180] S207 , updating the action instructions through the PPO policy network and the state information sequence corresponding to the PPO policy network, and updating the parameters of the PPO policy network based on the third prediction parameter update strategy.

[0181] In this step, the state information sequence includes state information of multiple image restoration stages, and each state information includes a first evaluation index, a second evaluation index, a first score, a second score, a potential vector, and a stage identifier of the image restoration stage.

[0182] Optionally, a possible implementation method of updating the action instruction with the state information sequence based on the PPO policy network set is:

[0183] S2071. When the parameter update of the generator is monitored and the cumulative update times of the parameters of the discriminator reaches a preset cumulative number, a state information sequence corresponding to the PPO strategy network is obtained.

[0184] In this step, the parameter update frequency of the PPO strategy network is consistent with the parameter update frequency of the generator. When the generator is detected to be updating its parameters, the parameter update of the PPO strategy network continues synchronously.

[0185] S2072. Calculate the update amplitude of the PPO strategy network based on the state information sequence and the length of the repair time of the image repair phase of the generator.

[0186] Alternatively, a possible implementation method of calculating the update amplitude of the PPO strategy network is:

[0187] a1. Based on the first evaluation indicator, the second evaluation indicator, the first score, the second score, and the repair time in the state information sequence, calculate the reward value of the PPO strategy network after the current batch of training sample data is input.

[0188] In this step, the reward value is calculated as shown in Formula 6:

[0189]

[0190] Among them, w1, w2, w3, and w4 refer to the weights of various parameters respectively; ΔSSIM refers to the change value of the first evaluation index, ΔPSNR refers to the change value of the second evaluation index, T stageRefers to the length of the restoration time of the current image restoration stage; D(x) refers to the difference between the first score and the second score generated by the discriminator, which is used to reflect the authenticity of the restored image; R t Refers to the calculated reward value.

[0191] a2. Based on the reward value and state information sequence, the advantage value of the PPO strategy network is calculated.

[0192] In this step, the advantage value is calculated as shown in Formula 7:

[0193]

[0194] in, Refers to the time difference error; V(s) refers to the state value function, which is used to indicate the expected return of the evaluator of the PPO strategy network in state s; γ refers to the discount factor, and λ refers to the smoothing coefficient; A t refers to the advantage value; R t Refers to the calculated reward value; t refers to the current moment, k refers to the number of steps forward from the current moment; s refers to the state observed by the decision maker of the PPO strategy network at moment t.

[0195] a3. Based on the advantage value and reward value, calculate the update amplitude of the PPO strategy network.

[0196] In this step, the PPO policy network limits the update amplitude of the policy network through the clipping mechanism, as shown in Formula 8:

[0197]

[0198] Among them, L(θ) refers to the loss function of the PPO strategy network, that is, the update amplitude; E t Refers to the expectation of all time steps t; R t (θ) refers to the probability ratio of the new strategy to the old strategy of the PPO strategy network; R t (θ)A t , clip refers to clipping the probability ratio to the interval , Refers to the clipping threshold; A t Refers to the advantage value.

[0199] S2073. Update the parameters of the PPO policy network according to the update amplitude of the PPO policy network and the preset learning rate of the PPO policy network.

[0200] In this step, the method for updating the parameters of the PPO strategy network can be: θ←θ+η·▽θL(θ), based on the preset learning rate η, the gradient ▽θL(θ) corresponding to the update amplitude is calculated to obtain the updated parameter θ.

[0201] S2074: Update the action instructions based on the PPO strategy network with updated parameters and the state information sequence.

[0202] In this step, the action instructions can be generated by combining the current state information in the state information sequence, using the PPO strategy network after parameter update to predict the action switching and learning rate adjustment, and updating the action instructions based on the prediction results to instruct the generator to adjust the learning rate and action switching.

[0203] In this embodiment, the training process of the generator is optimized through the action instructions of the PPO strategy network, and the parameters of the discriminator, generator and PPO strategy network are updated by alternating parameter updates, thereby improving the image restoration accuracy of the model.

[0204] Figure 3 A schematic diagram of the structure of an image restoration model provided in this application is shown as follows: Figure 3 As shown, the model includes:

[0205] The encoder 301 is configured to perform encoding and re-parameterization processing on the input sample occlusion image to obtain a latent vector.

[0206] The generator 302 is used to generate the repaired image in stages based on the latent vector and the action instructions of the PPO strategy network 303.

[0207] The PPO strategy network 303 is used to update the action instructions based on the latent vector, the repaired image, the real image and the true or false judgment results of the discriminator 304, and control the switching of the image repair stage of the generator and the adjustment of the learning rate.

[0208] The discriminator 304 is used to perform adversarial training and true / false judgment based on the restored image of the generator.

[0209] Figure 4 A flowchart of a UAV image restoration model training method provided in this application is shown in Figure 3 Based on the image restoration model shown in FIG, this embodiment explains the training method of the drone image restoration model, such as Figure 4 As shown, the method includes:

[0210] A1. Initialize the model parameters of the drone image restoration model.

[0211] In this step, the model parameters refer to the parameters of the discriminator, generator, and PPO policy network. The discriminator and generator are respectively set to the initial learning rate and , and the Adam optimizer is used for parameter update.

[0212] A2. Input the drone’s sample training data into the drone image restoration model.

[0213] In this step, the sample training data includes: occluded images of the drone and real images.

[0214] A3. The encoder extracts the mean and log-variance of the latent vector of the drone occluded image.

[0215] A4. The encoder performs reparameterization based on the latent vector mean and log variance to obtain the latent vector.

[0216] A5. The generator generates the repaired image of the current stage based on the latent vector and the action instructions of the PPO strategy network.

[0217] A6. When the cumulative number of discriminator parameter updates reaches N times, the total loss value of the generator is calculated based on the repaired image and the real image, the generator parameters are updated based on the total loss value and the generator's learning rate, and the number of discriminator parameter updates is set to the initial value.

[0218] A7. The discriminator calculates the inpainted image score and the real image score based on the inpainted image and the real image.

[0219] A8. When the cumulative number of updates to the discriminator parameters does not reach N, a loss value is calculated based on the scores of the two images, and the discriminator parameters are updated based on the loss value and the preset learning rate of the discriminator.

[0220] A9. When the generator parameter update is detected, the structural similarity index and peak signal-to-noise ratio are calculated based on the repaired image and the real image.

[0221] A10. Update the parameters of the PPO policy network and the action instructions of the PPO policy network based on the peak signal-to-noise ratio, structural similarity index, repaired image score, real image score, latent vector, and generator repair time.

[0222] A11. When the drone image restoration model meets the preset conditions, it is determined as the target drone image restoration model.

[0223] In this step, the preset conditions include at least one of the following: the cumulative rounds of iterative training are greater than or equal to the preset rounds; the loss value of the drone image restoration model is less than or equal to the preset loss value; the structural similarity index of the drone image restoration model is greater than the first preset value, and the peak signal-to-noise ratio of the image restoration model to be determined is greater than the second preset value.

[0224] For example, in Figure 4 Based on the embodiment shown, the embodiment of the present application also provides a plurality of pictures showing the results of drone image restoration. Figure 5The schematic diagram of the drone image restoration results provided for this application is as follows: Figure 5 As shown in the figure, after 100 iterations of model training, a drone image restoration model for image restoration is obtained; 5 drone occluded images to be restored are input into the model to obtain the restoration image corresponding to each drone occluded image; among them, the first image in each row from left to right is the real image, the second image is the drone occluded image, and the third image is the drone restoration image.

[0225] In this step, after each iteration of training, the structural similarity index and peak signal-to-noise ratio of the current UAV image restoration model are recorded.

[0226] For example, the structural similarity index and peak signal-to-noise ratio obtained after 100 training cycles are shown in the figure below. Figure 6 As shown, Figure 6 Schematic diagram of the curve change of the structural similarity index and peak signal-to-noise ratio provided in this application; Figure 6 As shown, the left picture is a schematic diagram of the curve change of the structural similarity index, which is represented by Validation SSIM, and the right picture is a schematic diagram of the curve change of the structural similarity index and the peak signal-to-noise ratio, which is represented by Validation PSNR. Among them, the horizontal axis represents the number of training times, and the vertical axis represents the values ​​of the structural similarity index and the peak signal-to-noise ratio, which are represented by SSIM and PSNR respectively. Figure 6 The curve diagram shows that during 100 training cycles of the sample training data, the loss functions of the generator and discriminator of the drone image restoration model converged significantly, with only slight fluctuations in the loss curves. Similarly, the changes in the discriminator and generator loss function values ​​were recorded during the 100 training cycles. The parameters of the discriminator and generator were continuously updated and optimized as the number of training rounds increased. The resulting loss value trend shows that the loss function value component decreases with the increase in training times and eventually stabilizes.

[0227] Figure 7 A flow chart of the image restoration method provided in this application is shown in FIG. Figure 7 As shown, the method includes:

[0228] S701: Acquire an image to be repaired.

[0229] In this step, the method of obtaining the image to be repaired needs to be determined based on the specific scenario of image repair.

[0230] For example, if the current scene is a power grid scene, a method for obtaining the image to be repaired may be: collecting the image to be repaired of the drone in the power grid scene based on a real-time monitoring device.

[0231] S702: Input the image to be repaired into the image repair model to obtain a repaired image corresponding to the image to be repaired.

[0232] In this step, the image restoration model is based on Figures 1 to 2 The image restoration model is obtained by training using any of the training methods.

[0233] Figure 8 This is a schematic diagram of the structure of the training device for the image restoration model provided in this application, such as Figure 8 As shown, the training device of the image restoration model provided in this embodiment includes:

[0234] The acquisition module 801 is used to acquire training sample data, where the training sample data includes a sample occlusion image and a sample real image corresponding to the sample occlusion image.

[0235] The first processing module 802 is used to train the initial image restoration model based on the training sample data, so as to alternately update the corresponding parameters of the discriminator, generator and PPO strategy network in the initial image restoration model in different training rounds to obtain the image restoration model to be determined; the initial image restoration model is obtained after training for preset training rounds based on the training sample data.

[0236] The second processing module 803 is configured to determine the image restoration model to be determined as the trained target image restoration model when the image restoration model to be determined meets a preset condition.

[0237] In a possible implementation, the first processing module 802 is further configured to:

[0238] The sample occluded image is encoded through the encoder of the initial image restoration model to obtain a latent vector corresponding to the sample occluded image.

[0239] The latent vector is reconstructed and repaired through the action instructions of the generator and the PPO strategy network to obtain a repaired image; wherein the action instructions include the learning rate of the generator and the switching instructions of the generator's repair phase.

[0240] Based on the first prediction parameter update strategy, the parameters of the generator are updated.

[0241] The discriminator calculates the first score corresponding to the restored image and the second score of the sample real image corresponding to the sample occluded image.

[0242] Based on the second prediction parameter updating strategy, the parameters of the discriminator are updated.

[0243] Based on the repaired image and the sample real image, a first evaluation index and a second evaluation index are calculated; wherein the first evaluation index is used to measure the similarity between the repaired image and the sample real image, and the second evaluation index is used to measure the error between the repaired image and the sample real image.

[0244] The action instructions are updated through the PPO policy network and the state information sequence corresponding to the PPO policy network, and the parameters of the PPO policy network are updated based on the third prediction parameter update strategy; wherein the state information sequence includes state information of multiple image restoration stages, and each state information includes a first evaluation indicator, a second evaluation indicator, a first score, a second score, a potential vector and a stage identifier of the image restoration stage.

[0245] In a possible implementation, the first processing module 802 is further configured to:

[0246] Based on the learning rate in the action instruction and the restoration phase switching instruction, the learning rate of the generator and the phase identifier of the image restoration phase are updated.

[0247] Based on the updated generator and the image restoration stage corresponding to the stage identifier, the latent vector is reconstructed and restored to obtain the restored image of the image restoration stage.

[0248] In a possible implementation, the first processing module 802 is further configured to:

[0249] When the parameter update of the generator is monitored and the cumulative update times of the parameters of the discriminator reaches the preset cumulative times, the state information sequence corresponding to the PPO strategy network is obtained.

[0250] According to the state information sequence and the length of the repair time of the generator's image repair phase, the update amplitude of the PPO strategy network is calculated.

[0251] Update the parameters of the PPO policy network according to the update amplitude of the PPO policy network and the preset learning rate of the PPO policy network.

[0252] Based on the PPO strategy network with updated parameters and the state information sequence, the action instructions are updated.

[0253] In a possible implementation, the first processing module 802 is further configured to:

[0254] Based on the first evaluation indicator, the second evaluation indicator, the first score, the second score and the repair time length in the state information sequence, the reward value of the PPO strategy network after the current batch of training sample data is input is calculated.

[0255] Based on the reward value and state information sequence, the advantage value of the PPO strategy network is calculated.

[0256] Based on the advantage value and reward value, the update amplitude of the PPO strategy network is calculated.

[0257] In a possible implementation, the first processing module 802 is further configured to:

[0258] When the cumulative update times of the discriminator parameters reach the preset cumulative times, the total loss value of the generator is calculated based on the repaired image and the real sample image; the total loss value includes the reconstruction loss value and the adversarial loss value.

[0259] Based on the total loss value and the generator's learning rate, update the generator's parameters.

[0260] Set the cumulative update times of the discriminator parameters to the initial value.

[0261] In a possible implementation, the first processing module 802 is further configured to:

[0262] When the cumulative update times of the discriminator parameters are lower than the preset cumulative times, the discriminator calculates the first score of the repaired image and the discriminant loss value corresponding to the second score of the sample real image; the discriminant loss value is the weighted value of the gradient penalty value of the discriminator and the distance loss value.

[0263] Update the discriminator parameters based on the loss value and the discriminator's preset learning rate.

[0264] In one possible implementation, the preset conditions include at least any one of the following: the cumulative rounds of iterative training are greater than or equal to the preset rounds; the loss value of the image restoration model to be determined is less than or equal to the preset loss value; the first evaluation index of the image restoration model to be determined is greater than the first preset value, and the second evaluation index of the image restoration model to be determined is greater than the second preset value.

[0265] Figure 9 This is a schematic diagram of the structure of the image restoration device provided in this application, as shown in FIG. Figure 9 As shown, the image restoration device provided in this embodiment includes:

[0266] An acquisition module 901 is used to acquire an image to be restored;

[0267] The processing module 902 is used to input the image to be repaired into the image repair model to obtain a repaired image corresponding to the image to be repaired; wherein the image repair model is based on Figure 8 The image restoration model is obtained by training using any of the training methods.

[0268] The image restoration model training device and the image restoration device provided in this embodiment can execute the method provided in the above method embodiment. The implementation principles and technical effects are similar and will not be described in detail in this embodiment.

[0269] Figure 10 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 10 As shown, the electronic device provided by this embodiment includes: at least one processor 1001 and a memory 1002. Optionally, the device further includes a communication component 1003. The processor 1001, the memory 1002, and the communication component 1003 are connected via a bus 1004.

[0270] During the specific implementation process, at least one processor 1001 executes the computer-executable instructions stored in the memory 1002, so that the at least one processor 1001 performs the above method.

[0271] The specific implementation process of the processor 1001 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0272] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0273] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0274] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0275] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0276] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0277] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0278] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may reside in an application-specific integrated circuit (ASIC). The processor and the readable storage medium may also reside in a device as discrete components.

[0279] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, whether electrical, mechanical, or otherwise, through some interface.

[0280] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0282] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0283] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0284] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A training method for an image restoration model, characterized in that: The method comprises: Acquire training sample data, where the training sample data includes a sample occluded image and a sample real image corresponding to the sample occluded image; Based on the training sample data, an initial image restoration model is trained to alternately update corresponding parameters of a discriminator, a generator, and a proximity optimization (PPO) strategy network in the initial image restoration model in different training rounds to obtain an image restoration model to be determined; the initial image restoration model is obtained after training for a preset number of training rounds based on the training sample data; When the image restoration model to be determined meets a preset condition, the image restoration model to be determined is determined as a trained target image restoration model.

2. The method according to claim 1, characterized in that The initial image restoration model is trained based on the training sample data to alternately update the parameters corresponding to the discriminator, the generator, and the PPO strategy network in the initial image restoration model in different training rounds, including: Encoding the sample occluded image through the encoder of the initial image restoration model to obtain a latent vector corresponding to the sample occluded image; Performing image reconstruction and restoration on the latent vector through the action instructions of the generator and the PPO strategy network to obtain a restored image; wherein the action instructions include a learning rate of the generator and a restoration phase switching instruction of the generator; Based on a first prediction parameter update strategy, updating parameters of the generator; Calculating, by the discriminator, a first score corresponding to the restored image and a second score of the sample real image corresponding to the sample occluded image; Based on a second prediction parameter updating strategy, updating the parameters of the discriminator; Based on the inpainted image and the sample real image, a first evaluation index and a second evaluation index are calculated; wherein the first evaluation index is used to measure the similarity between the inpainted image and the sample real image, and the second evaluation index is used to measure the error between the inpainted image and the sample real image; The action instruction is updated through the PPO policy network and the state information sequence corresponding to the PPO policy network, and the parameters of the PPO policy network are updated based on a third prediction parameter update strategy; wherein the state information sequence includes state information of multiple image restoration stages, and each state information includes the first evaluation indicator, the second evaluation indicator, the first score, the second score, the latent vector and the stage identifier of the image restoration stage.

3. The method according to claim 2, characterized in that The step of reconstructing and repairing the latent vector through the generator and the action instructions of the PPO strategy network to obtain a repaired image includes: Based on the learning rate in the action instruction and the restoration phase switching instruction, updating the learning rate of the generator and the phase identifier of the image restoration phase; Based on the updated generator and the image restoration stage corresponding to the stage identifier, image reconstruction and restoration are performed on the latent vector to obtain a restored image of the image restoration stage.

4. The method according to claim 3, characterized in that The updating of the action instruction by using the PPO strategy network and the state information sequence corresponding to the PPO strategy network includes: When the parameter update of the generator is monitored and the cumulative number of updates of the parameters of the discriminator reaches a preset cumulative number, obtaining the state information sequence corresponding to the PPO strategy network; Calculating an update amplitude of the PPO strategy network according to the state information sequence and the length of the repair time of the image repair phase of the generator; Updating parameters of the PPO strategy network according to the update amplitude of the PPO strategy network and the preset learning rate of the PPO strategy network; The action instruction is updated based on the PPO strategy network after parameter update and the state information sequence.

5. The method according to claim 4, characterized in that The updating amplitude of the PPO strategy network is calculated based on the state information sequence and the repair time length of the image repair phase of the generator, including: Calculate the reward value of the PPO strategy network after inputting the training sample data of the current batch based on the first evaluation indicator, the second evaluation indicator, the first score, the second score, and the repair time length in the state information sequence; Calculating the advantage value of the PPO strategy network based on the reward value and the state information sequence; Based on the advantage value and the reward value, an update amplitude of the PPO strategy network is calculated.

6. The method according to claim 2, characterized in that The updating of the parameters of the generator based on the first prediction parameter updating strategy includes: When the cumulative update times of the parameters of the discriminator reaches a preset cumulative number, a total loss value of the generator is calculated based on the repaired image and the sample real image; the total loss value includes a reconstruction loss value and an adversarial loss value; Based on the total loss value and the learning rate of the generator, update the parameters of the generator; The cumulative update times of the parameters of the discriminator are set to initial values.

7. The method according to claim 2, characterized in that The updating of the parameters of the discriminator based on the second prediction parameter updating strategy includes: When the cumulative update number of the parameters of the discriminator is lower than a preset cumulative number, the discriminator calculates the first score of the restored image and a discrimination loss value corresponding to the second score of the sample real image; the discrimination loss value is a weighted value of the gradient penalty value and the distance loss value of the discriminator; Based on the loss value and a preset learning rate of the discriminator, the parameters of the discriminator are updated.

8. The method according to claim 1, characterized in that The preset conditions include at least any one of the following: the cumulative number of iterative training rounds is greater than or equal to the preset number of rounds; the loss value of the image restoration model to be determined is less than or equal to the preset loss value; The first evaluation index of the image restoration model to be determined is greater than a first preset value, and the second evaluation index of the image restoration model to be determined is greater than a second preset value.

9. An image restoration method, characterized in that: The method comprises: Obtain the image to be repaired; The image to be repaired is input into an image repair model to obtain a repaired image corresponding to the image to be repaired; wherein the image repair model is obtained by training according to the image repair model training method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8 or 9.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 or 9 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 or 9 when executed by a processor.

Images