An image inpainting method, device, apparatus and storage medium

Abstract

The application discloses an image repairing method, device and equipment and a storage medium. The method comprises the following steps: acquiring a training set for generating a blurred image, and training a generative adversarial network by using the training set; acquiring a training result of the generative adversarial network at a target training node in the training process, and determining an error parameter of the generative adversarial network at present according to the training result; adjusting a control parameter of a PID controller by using a blurring algorithm based on the error parameter; adjusting a training parameter of the generative adversarial network by using the PID controller after adjustment until the training of the generative adversarial network is stopped when the training result of the generative adversarial network reaches an expected precision to obtain an image repairing model, so as to perform image repairing on a to-be-repaired image by using the image repairing model. The controllable precision loss can shorten the time required for training the neural network and reduce the computing power required for training, thereby improving the efficiency of image repairing.

Description

Technical Field

[0001] This invention relates to the field of image restoration, and in particular to an image restoration method, apparatus, device, and storage medium. Background Technology

[0002] Image inpainting refers to the process of reconstructing lost or damaged parts of images and videos. Because the targets of image inpainting technology are highly complex, each type of image requires a large number of samples and computational power, but current methods for acquiring such computational power are prohibitively expensive. Existing technologies typically use Generative Adversarial Networks (GANs) for image inpainting. GANs are deep learning models and are among the most promising unsupervised learning methods on complex distributions in recent years. However, GANs suffer from convergence issues. Theoretically, GANs should perform exceptionally well in achieving Nash equilibrium, but gradient descent only guarantees Nash equilibrium under convex functions. When the generator and discriminator in a GAN are both represented by neural networks, it is possible for them to perpetually adjust their strategies without actually reaching equilibrium. In addition, Generative Adversarial Networks (GANs) are difficult to train and prone to collapse. GAN models are defined as minimax problems and have no loss function. During training, it is difficult to distinguish whether progress is being made. Therefore, the learning process of GANs may collapse. The generator begins to degenerate, always generating the same sample points and unable to continue learning. When the generative model collapses, the discriminative model will also point similar sample points in similar directions, making training impossible and reducing the efficiency of image inpainting. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an image restoration method, apparatus, device, and medium that can shorten the time required to train a neural network with controllable accuracy loss and reduce the computing power required for training, thereby improving the efficiency of image restoration. The specific solution is as follows:

[0004] In a first aspect, this application discloses an image restoration method, comprising:

[0005] Obtain blurred images to generate a training set, and use the training set to train a generative adversarial network;

[0006] The training results of the generative adversarial network are obtained at the target training node during the training process, and the current error parameters of the generative adversarial network are determined based on the training results.

[0007] Based on the error parameters, the control parameters of the PID controller are adjusted using a fuzzy algorithm.

[0008] The training parameters of the generative adversarial network are adjusted using the modified PID controller until the training result of the generative adversarial network reaches the expected accuracy. Then, the training of the generative adversarial network is stopped to obtain an image restoration model, which can then be used to restore the image to be restored.

[0009] Optionally, the process of training the generative adversarial network using the training set further includes:

[0010] A joint loss function is constructed based on the reconstruction loss function and the adversarial loss function; the reconstruction loss function is a function constructed based on the Euclidean distance under the prerequisite mask layer;

[0011] The joint loss function is used as the loss function during the training process of the generative adversarial network.

[0012] Optionally, determining whether the training result of the generative adversarial network has achieved the expected accuracy includes:

[0013] The training results of the generative adversarial network are evaluated using GAN evaluation metrics to determine whether the training results of the generative adversarial network have achieved the expected accuracy.

[0014] Optionally, the training parameters of the generative adversarial network include any one or more of the following: the number of training iterations per unit cycle of the generator, the number of training iterations per unit cycle of the discriminator, the Adam optimizer parameters, the generator noise dimension, the generator learning rate, the discriminator learning rate, and the total number of training iterations.

[0015] Optionally, determining the current error parameters of the generative adversarial network based on the training results includes:

[0016] Based on the training results, the first learning rate corresponding to the generator in the current generative adversarial network and the second learning rate corresponding to the discriminator in the generative adversarial network are determined.

[0017] Based on the first learning rate and the second learning rate, as well as the configured generator target learning rate and discriminator target learning rate, the current error and the current error rate of the generative adversarial network are determined to obtain the error parameters.

[0018] Optionally, adjusting the control parameters of the PID controller using a fuzzy algorithm based on the error parameters includes:

[0019] Based on the current error and the current error rate, a pre-created fuzzy rule table is consulted to determine the parameter variable values ​​corresponding to each control term of the PID controller; the control term includes proportional, integral, and derivative;

[0020] Based on the parameter variable values, the control parameters of the PID controller are adjusted according to the most recent historical control parameters of each control item to obtain the latest proportional control parameters, integral control parameters, and derivative control parameters.

[0021] Optionally, adjusting the training parameters of the generative adversarial network using the adjusted PID controller includes:

[0022] Using the adjusted PID controller, the target learning rate of the generator and the target learning rate of the discriminator corresponding to the generative adversarial network are dynamically adjusted so that the generator of the generative adversarial network is trained according to the adjusted generator learning rate, and the discriminator of the generative adversarial network is trained according to the adjusted discriminator learning rate.

[0023] Secondly, this application discloses an image restoration apparatus, comprising:

[0024] A blurred image acquisition module is used to acquire blurred images to generate a training set, and to use the training set to train a generative adversarial network.

[0025] The error parameter determination module is used to obtain the training results of the generative adversarial network at the target training node during the training process, and determine the current error parameters of the generative adversarial network based on the training results.

[0026] A control parameter adjustment module is used to adjust the control parameters of the PID controller based on the error parameters using a fuzzy algorithm.

[0027] The network training parameter adjustment module is used to adjust the training parameters of the generative adversarial network using the adjusted PID controller until the training result of the generative adversarial network reaches the expected accuracy, at which point the training of the generative adversarial network is stopped to obtain an image restoration model, so that the image restoration model can be used to restore the image to be restored.

[0028] Thirdly, this application discloses an electronic device, including:

[0029] Memory, used to store computer programs;

[0030] A processor is used to execute the computer program to implement the aforementioned image restoration method.

[0031] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned image restoration method.

[0032] In this application, a blurred image generation training set is obtained, and a generative adversarial network (GAN) is trained using the training set. The training results of the GAN are obtained at the target training node during the training process, and the current error parameters of the GAN are determined based on the training results. The control parameters of a PID controller are adjusted using a fuzzy algorithm based on the error parameters. The training parameters of the GAN are then adjusted using the adjusted PID controller until the training results of the GAN reach the expected accuracy, at which point the training of the GAN is stopped to obtain an image restoration model. This model is then used to perform image restoration on the image to be restored. As can be seen from the above, in this embodiment, during the training of the generative adversarial network, the control parameters of the PID controller are adjusted using a fuzzy algorithm based on the error between the network training result and the expected result at the target training node, i.e., at different stages of the training process. The PID controller then controls the training of the generative adversarial network by controlling the network training parameters. By adjusting the control parameters of the PID controller in real time according to the network error, the PID controller dynamically adjusts the training parameters during the network training process, enabling more precise and reasonable control of the network training. This allows for rapid training with the expected accuracy as the target, and stops training when the accuracy is reached, stabilizing the network near that accuracy. This achieves rapid convergence while meeting the accuracy requirements, avoiding the problem of wasting a lot of time and computing power by manually adjusting network training parameters in existing technologies. It significantly optimizes the time required for network training and greatly reduces the computing power required to train the neural network with controllable accuracy loss. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 A flowchart of an image restoration method provided in this application;

[0035] Figure 2 This application provides a specific fuzzy PID control principle diagram;

[0036] Figure 3 A flowchart of a specific image restoration method provided in this application;

[0037] Figure 4 A schematic diagram of an image restoration device provided in this application;

[0038] Figure 5 This application provides a structural diagram of an electronic device. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] In existing technologies, generative adversarial networks (GANs) are commonly used for image inpainting. However, GANs suffer from convergence issues and are difficult to train and prone to crashing. GAN models are defined as minimax problems and lack a loss function, making it difficult to distinguish whether progress is being made during training. Therefore, the learning process of GANs may collapse, preventing further learning and reducing the efficiency of image inpainting. To overcome these technical problems, this application proposes an image inpainting method that can shorten the training time of the neural network with controllable accuracy loss and reduce the computational power required for training, thereby improving the efficiency of image inpainting.

[0041] This application discloses an image restoration method, see [link to relevant documentation]. Figure 1 As shown, the method may include the following steps:

[0042] Step S11: Obtain a training set of blurred images and use the training set to train the generative adversarial network.

[0043] In this embodiment, a training set for generating blurred images is first obtained, and the generative adversarial network (GAN) is trained using this training set. This embodiment can use either the aforementioned GAN network or a Deep Convolutional Generative Adversarial Network (DCGAN). DCGAN has excellent image feature recognition capabilities, and its context-based encoding and decoding structure enables the trained generator model (image inpainting network) to reasonably repair images based on contextual information. The context-encoder-based image inpainting model mainly consists of two parts: an image inpainting network and a discriminator network based on contextual rationality. The image inpainting network is abstracted as the generator network in a generative adversarial network. It is responsible for repairing missing images and is the most important restoration network after deep learning training. The image inpainting network and the context-based discriminator network are trained adversarially, improving each other, and ultimately a relatively good image inpainting model is obtained. The reason why the image inpainting network is abstracted as a generator network is that it mainly consists of an encoder and a decoder. The input of the image inpainting network is the image to be repaired, and the output is the repaired image. The encoder encodes the input data into the image to be repaired through multiple convolutional layers. The decoder restores the encoded data to the original image size through deconvolution operations. The training process of the aforementioned generative adversarial network specifically involves using a training set containing blurred images as input, and deconvolving, for example, a 2x2 matrix to a 4x4 matrix, to extract the pixel matrix for adversarial training between the discriminator and generator within the DCGAN adversarial network.

[0044] In this embodiment, the process of training the generative adversarial network using the training set may further include: constructing a joint loss function based on the reconstruction loss function and the adversarial loss function; the reconstruction loss function is a function constructed based on the Euclidean distance under the prerequisite mask layer; and the joint loss function is used as the loss function in the training process of the generative adversarial network. It is understood that due to internal changes in the generator model, model training cannot be performed using the traditional GAN ​​cross-entropy loss function. In the context-encoding-based image restoration algorithm, a joint loss function is used for training. The joint loss function consists of two parts: the reconstruction loss function and the adversarial loss function. The reconstruction loss function is the square of the L2 norm of the difference between the missing region completion result and the true result, specifically a function constructed based on the Euclidean distance under the prerequisite mask layer, expressed as follows:

[0045]

[0046] in, This is a pre-condition mask, i.e., a prerequisite mask layer. In the matrix, missing regions are marked as 1, and background images are marked as 0. Soon The matrix changes 0 to 1 and 1 to 0; x represents real image data; This is the output of the context encoder; ⊙ represents the Hadamard product (multiplication of corresponding matrix elements); It is a 2-norm;

[0047] The expression for the adversarial loss function is as follows:

[0048]

[0049] Here, D() represents the discriminator within the generative adversarial network, and the adversarial loss function is the condition of substituting the discriminator D into the adversarial loss function.

[0050] The expression for the joint loss function is as follows:

[0051]

[0052] Where, λ rec Specifically, it can be 0.999, λ adv Specifically, it can be 0.001. It's clear that the effectiveness of generative adversarial networks depends on the size of the loss function, which means that... Minimize the time as much as possible while keeping the time as small as possible.

[0053] Step S12: Obtain the training results of the generative adversarial network at the target training node during the training process, and determine the current error parameters of the generative adversarial network based on the training results.

[0054] In this embodiment, the training results of the generative adversarial network are obtained at the target training nodes during the training process, and the current error parameters of the generative adversarial network are determined based on the training results. The target training nodes are multiple time points at different stages in the training process, and the time intervals between different target training nodes can be configured according to the actual operation. The error parameters represent the error between the network training results and the expected results.

[0055] In this embodiment, determining the current error parameters of the generative adversarial network (GAN) based on the training results may include: determining a first learning rate corresponding to the generator and a second learning rate corresponding to the discriminator within the GAN based on the training results; and determining the current error and current error rate of the GAN based on the first and second learning rates, as well as the configured target learning rates for the generator and discriminator, to obtain the error parameters. Specifically, in this embodiment, the difference between the target learning rate and the actual learning rates corresponding to the generator and discriminator within the network are used as the error parameters. That is, the generator's error parameters are obtained based on the first learning rate corresponding to the generator and the configured target learning rate for the generator; and the discriminator's error parameters are obtained based on the second learning rate corresponding to the discriminator and the configured target learning rate for the discriminator. The error parameters specifically include the current error and the current error rate.

[0056] Step S13: Adjust the control parameters of the PID controller using a fuzzy algorithm based on the error parameters.

[0057] In this embodiment, after obtaining the error parameters corresponding to the generative adversarial network, the control parameters of the PID controller are adjusted using a fuzzy algorithm based on these error parameters. The fuzzy algorithm is similar to a successive refinement process; its principle is essentially mimicking the human process of adjusting a system. Taking a temperature control system as an example, when the deviation between the preset temperature and the current temperature (i.e., the temperature error) is large, the controller will increase the power output; when the temperature error is small, the controller will decrease the power output. In other words, this embodiment uses a fuzzy algorithm to dynamically change the three important control parameters of the PID controller during the control process, and the control strategy of the PID controller is determined based on the error parameters corresponding to the generative adversarial network, thereby making the control capability of the PID controller more accurate and stable.

[0058] In this embodiment, adjusting the control parameters of the PID controller using a fuzzy algorithm based on the error parameters may include: determining the parameter variable values ​​corresponding to each control term of the PID controller by consulting a pre-created fuzzy rule table based on the current error and the current error rate; the control terms include proportional, integral, and derivative; and adjusting the control parameters of the PID controller based on the most recent historical control parameters of each control term of the PID controller according to the parameter variable values ​​to obtain the latest proportional control parameters, integral control parameters, and derivative control parameters. PID is an abbreviation for Proportional, Integral, and Differential. The fuzzy rule table is a pre-created rule table, specifically including K... P Fuzzy rule table, K IFuzzy rule table and K D The fuzzy rule table allows you to determine the parameter variable value corresponding to each control parameter based on the error parameter by querying the rule table.

[0059] The data expression for the PID controller is as follows:

[0060]

[0061] Where e(t) is the aforementioned error parameter; K P K is the proportional control parameter. I K is the integral control parameter. D These are the differential control parameters. It's understandable that the three control parameters of a traditional PID controller are fixed after being set and lack adaptive characteristics. In this embodiment, a fuzzy algorithm is used to adjust the three control parameters of the PID controller in real time based on the error parameter. That is, the three important parameters K of the PID controller are dynamically adjusted by using the two input variables, error and the rate of change of error. P K I K D This allows the controller to achieve optimal performance. In this embodiment, the PID control parameters can be tuned incrementally, which avoids excessive errors and improves tuning accuracy. Specifically, for example... Figure 2 As shown, the input to the fuzzy controller is two-dimensional, but it can also be designed to be multi-dimensional when there are multiple types of input data. In this embodiment, the error and the rate of change of the error are used as the input to the fuzzy controller, and the output of the fuzzy controller is the variable value of the PID parameters, namely Δkp, Δki, and Δkd; then the latest proportional control parameter, integral control parameter, and derivative control parameter of the PID are respectively: Kp(n) = Kp(n-1) + Δkp; Ki(n) = Ki(n-1) + Δki; Kd(n) = Kd(n-1) + Δkd.

[0062] Step S14: Adjust the training parameters of the generative adversarial network using the adjusted PID controller until the training result of the generative adversarial network reaches the expected accuracy, then stop training the generative adversarial network to obtain an image restoration model, so as to use the image restoration model to perform image restoration on the image to be restored.

[0063] In this embodiment, the training parameters of the generative adversarial network (GAN) are adjusted using a PID controller with dynamically adjusted parameters until the training result of the GAN reaches the expected accuracy. Training of the GAN is then stopped to obtain an image restoration model, which is then used to restore the image to be restored. For example... Figure 3As shown, by introducing fuzzy PID control—that is, combining fuzzy algorithms with PID—the learning parameters in the image information matrix are fuzzified, and the parameter set is continuously and dynamically adjusted. This dynamically adjusts the training parameters of the generative adversarial network, ensuring that the current training parameters best match the current state of the network, thereby optimizing training efficiency. In this embodiment, the process ends if the network reaches the expected accuracy; if it does not reach the expected accuracy, the input parameters are modified, specifically the target learning rate of the generator and the target learning rate of the discriminator. The input parameters have upper and lower limits, with the non-collapse limit as the upper limit.

[0064] In this embodiment, determining whether the training result of the generative adversarial network (GAN) has reached the expected accuracy may include: evaluating the accuracy of the training result of the GAN using GAN evaluation metrics to determine whether the training result of the GAN has reached the expected accuracy. The GAN evaluation metrics include, but are not limited to, InceptionScore (IS), Frechet Inception Distance (FID), Mode Score, Kernel MMD (Maximum Mean Discrepancy), Wasserstein distance, etc.

[0065] In this embodiment, adjusting the training parameters of the generative adversarial network (GAN) using the adjusted PID controller may include: dynamically adjusting the generator target learning rate and the discriminator target learning rate corresponding to the GAN using the adjusted PID controller, so that the generator of the GAN is trained according to the adjusted generator learning rate, and the discriminator of the GAN is trained according to the adjusted discriminator learning rate. In this embodiment, the error corresponding to the network's learning rate is used as the control condition for controlling the PID controller. Therefore, in this embodiment, the PID controller is used to dynamically adjust the aforementioned generator target learning rate and the aforementioned discriminator target learning rate corresponding to the GAN, so that the generator of the GAN is trained according to the adjusted generator learning rate, and the discriminator of the GAN is trained according to the adjusted discriminator learning rate. That is, the generator target learning rate and the discriminator target learning rate are manually set inputs. Figure 2 The input in the training process is adjusted by a PID controller to make it conform to the current state of the network. The controlled object is the generative adversarial network or the deep convolutional adversarial network. Training stops after the expected accuracy is reached, thereby reducing training computing power and parameter tuning time.

[0066] In this embodiment, the training parameters of the generative adversarial network may include, but are not limited to, the number of training iterations per unit cycle for the generator, the number of training iterations per unit cycle for the discriminator, the Adam optimizer parameters, the generator noise dimension, the generator learning rate, the discriminator learning rate, and the total number of training iterations. That is, in addition to adjusting the generator learning rate and the discriminator learning rate as exemplified in this embodiment, any one or more of the above training parameters can be used as training parameters that need to be adjusted during the training process.

[0067] As can be seen from the above, in this embodiment, a training set is generated by acquiring blurred images, and the generative adversarial network (GAN) is trained using the training set. The training results of the GAN are obtained at the target training node during the training process, and the current error parameters of the GAN are determined based on the training results. The control parameters of the PID controller are adjusted using a fuzzy algorithm based on the error parameters. The training parameters of the GAN are then adjusted using the adjusted PID controller until the training results of the GAN reach the expected accuracy, at which point the training of the GAN is stopped to obtain an image restoration model. This model is then used to perform image restoration on the image to be restored. As can be seen from the above, in this embodiment, during the training of the generative adversarial network, the control parameters of the PID controller are adjusted using a fuzzy algorithm based on the error between the network training result and the expected value at the target training node, i.e., at different stages of the training process. The PID controller then controls the training of the generative adversarial network by controlling the network training parameters. By adjusting the control parameters of the PID controller in real time according to the network error, the PID controller dynamically adjusts the training parameters during the network training process, enabling more precise and reasonable control of the network training. This allows for rapid training with the expected accuracy as the target, and stops training when the accuracy is reached, stabilizing the network near that accuracy. This achieves rapid convergence while meeting the accuracy requirements, avoiding the problem of wasting a lot of time and computing power by manually adjusting network training parameters in existing technologies. This significantly optimizes the time required for network training and greatly reduces the computing power required to train the neural network with controllable accuracy loss.

[0068] Accordingly, this application also discloses an image restoration apparatus, see [link to relevant documentation]. Figure 4 As shown, the device includes:

[0069] The blurred image acquisition module 11 is used to acquire a blurred image to generate a training set, and to use the training set to train the generative adversarial network.

[0070] Error parameter determination module 12 is used to obtain the training results of the generative adversarial network at the target training node during the training process, and determine the current error parameters of the generative adversarial network based on the training results.

[0071] Control parameter adjustment module 13 is used to adjust the control parameters of the PID controller based on the error parameters using a fuzzy algorithm;

[0072] The network training parameter adjustment module 14 is used to adjust the training parameters of the generative adversarial network using the adjusted PID controller until the training result of the generative adversarial network reaches the expected accuracy, at which point the training of the generative adversarial network is stopped to obtain an image restoration model, so that the image restoration model can be used to restore the image to be restored.

[0073] As can be seen from the above, in this embodiment, a training set is generated by acquiring blurred images, and the generative adversarial network (GAN) is trained using the training set. The training results of the GAN are obtained at the target training node during the training process, and the current error parameters of the GAN are determined based on the training results. The control parameters of the PID controller are adjusted using a fuzzy algorithm based on the error parameters. The training parameters of the GAN are then adjusted using the adjusted PID controller until the training results of the GAN reach the expected accuracy, at which point the training of the GAN is stopped to obtain an image restoration model. This model is then used to perform image restoration on the image to be restored. As can be seen from the above, in this embodiment, during the training of the generative adversarial network, the control parameters of the PID controller are adjusted using a fuzzy algorithm based on the error between the network training result and the expected value at the target training node, i.e., at different stages of the training process. The PID controller then controls the training of the generative adversarial network by controlling the network training parameters. By adjusting the control parameters of the PID controller in real time according to the network error, the PID controller dynamically adjusts the training parameters during the network training process, enabling more precise and reasonable control of the network training. This allows for rapid training with the expected accuracy as the target, and stops training when the accuracy is reached, stabilizing the network near that accuracy. This achieves rapid convergence while meeting the accuracy requirements, avoiding the problem of wasting a lot of time and computing power by manually adjusting network training parameters in existing technologies. This significantly optimizes the time required for network training and greatly reduces the computing power required to train the neural network with controllable accuracy loss.

[0074] In some specific embodiments, the image restoration device may specifically include:

[0075] The loss function construction unit is used to construct a joint loss function based on the reconstruction loss function and the adversarial loss function; the reconstruction loss function is a function constructed based on the Euclidean distance under the prerequisite mask layer;

[0076] The training unit is used to use the joint loss function as the loss function in the training process of the generative adversarial network.

[0077] In some specific embodiments, the network training parameter adjustment module 14 may specifically include:

[0078] The accuracy judgment unit is used to evaluate the accuracy of the training results of the generative adversarial network using GAN evaluation metrics, so as to determine whether the training results of the generative adversarial network have achieved the expected accuracy.

[0079] In some specific embodiments, the training parameters of the generative adversarial network may include any one or more of the following: the number of training iterations per unit cycle of the generator, the number of training iterations per unit cycle of the discriminator, the Adam optimizer parameters, the generator noise dimension, the generator learning rate, the discriminator learning rate, and the total number of training iterations.

[0080] In some specific embodiments, the error parameter determination module 12 may specifically include:

[0081] The learning rate determination unit is used to determine, based on the training results, the first learning rate corresponding to the generator in the current generative adversarial network and the second learning rate corresponding to the discriminator in the generative adversarial network.

[0082] An error and error rate determination unit is used to determine the current error and current error rate of the generative adversarial network based on the first learning rate and the second learning rate, as well as the configured generator target learning rate and discriminator target learning rate, so as to obtain the error parameters.

[0083] In some specific embodiments, the control parameter adjustment module 13 may specifically include:

[0084] The parameter variable value determination unit is used to determine the parameter variable value corresponding to each control term of the PID controller by referring to a pre-created fuzzy rule table based on the current error and the current error rate; the control term includes proportional, integral and derivative.

[0085] The control parameter adjustment unit is used to adjust the control parameters of the PID controller based on the most recent historical control parameters of each control item of the PID controller according to the parameter variable values, so as to obtain the latest proportional control parameters, integral control parameters and derivative control parameters.

[0086] In some specific embodiments, the network training parameter adjustment module 14 can be used to dynamically adjust the generator target learning rate and the discriminator target learning rate corresponding to the generative adversarial network using the adjusted PID controller, so as to train the generator of the generative adversarial network according to the adjusted generator learning rate and train the discriminator of the generative adversarial network according to the adjusted discriminator learning rate.

[0087] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 5 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0088] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the image restoration method disclosed in any of the foregoing embodiments.

[0089] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0090] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including blurred images, etc. The storage method can be temporary storage or permanent storage.

[0091] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the image restoration method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0092] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the image restoration method steps disclosed in any of the foregoing embodiments.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0094] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] The above provides a detailed description of the image restoration method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image inpainting method characterized by, The method comprises the following steps: obtaining a training set of blurred images and training a generative adversarial network using the training set; obtaining a training result of the generative adversarial network at a target training node in the training process, and determining a current error parameter of the generative adversarial network according to the training result; the error parameter comprises a current error and a current error rate; adjusting the control parameters of a PID controller using a fuzzy algorithm based on the error parameter; adjusting the training parameters of the generative adversarial network using the adjusted PID controller until the training result of the generative adversarial network reaches the expected accuracy, stopping the training of the generative adversarial network to obtain an image restoration model, so as to perform image restoration on a to-be-restored image using the image restoration model; wherein the adjusting of the control parameters of the PID controller using the fuzzy algorithm based on the error parameter comprises: according to the current error and the current error rate, checking a pre-created fuzzy rule table to determine the parameter variable value corresponding to each control item of the PID controller; the fuzzy rule table comprises a proportional control parameter fuzzy rule table, an integral control parameter fuzzy rule table, and a differential control parameter fuzzy rule table; the control items include proportion, integral, and differential; based on the parameter variable value, adjusting the control parameters of the PID controller based on the last historical control parameters of each control item of the PID controller to obtain the latest proportional control parameters, integral control parameters, and differential control parameters; wherein the training of the generative adversarial network using the training set further comprises: constructing a joint loss function based on a reconstruction loss function and an adversarial loss function; the reconstruction loss function is a function constructed according to the Euclidean distance under the precondition mask layer; using the joint loss function as the loss function in the training process of the generative adversarial network; wherein the determining of the current error parameter of the generative adversarial network according to the training result comprises: determining a first learning rate corresponding to a generator in the generative adversarial network and a second learning rate corresponding to a discriminator in the generative adversarial network according to the training result; determining the current error and the current error rate of the generative adversarial network according to the first learning rate and the second learning rate, and the configured target learning rate of the generator and the target learning rate of the discriminator, to obtain the error parameter.

2. The image inpainting method of claim 1, wherein, judging whether the training result of the generative adversarial network reaches the expected accuracy comprises: using a GAN evaluation index to evaluate the accuracy of the training result of the generative adversarial network to determine whether the training result of the generative adversarial network reaches the expected accuracy.

3. The image inpainting method of claim 1, wherein, The training parameters of the generative adversarial network include any one or more of the following: the number of training times per unit cycle of the generator, the number of training times per unit cycle of the discriminator, the Adam optimizer parameter, the generator noise dimension, the generator learning rate, the discriminator learning rate, and the total number of training times.

4. The image inpainting method of claim 1, wherein, the adjusting of the training parameters of the generative adversarial network using the adjusted PID controller comprises: Adjust the generator target learning rate and the discriminator target learning rate corresponding to the generative adversarial network by using the adjusted PID controller, so as to train the generator of the generative adversarial network according to the adjusted generator learning rate, and train the discriminator of the generative adversarial network according to the adjusted discriminator learning rate.

5. An image inpainting apparatus characterized by comprising: Comprise: A blurred image acquisition module is configured to acquire a blurred image to generate a training set, and train a generative adversarial network by using the training set; An error parameter determination module is configured to acquire a training result of the generative adversarial network at a target training node in a training process, and determine a current error parameter of the generative adversarial network according to the training result; The error parameter comprises a current error and a current error rate; A control parameter adjustment module is configured to adjust control parameters of a PID controller by using a fuzzy algorithm based on the error parameter; A network training parameter adjustment module is configured to adjust training parameters of the generative adversarial network by using the adjusted PID controller, and stop training the generative adversarial network when the training result of the generative adversarial network reaches an expected accuracy to obtain an image inpainting model, so as to perform image inpainting on a to-be-inpainted image by using the image inpainting model; The control parameter adjustment module is configured to determine a parameter variable value corresponding to each control item of the PID controller according to the current error and the current error rate, and view a pre-created fuzzy rule table; the fuzzy rule table comprises a proportional control parameter fuzzy rule table, an integral control parameter fuzzy rule table, and a differential control parameter fuzzy rule table; the control items comprise proportion, integral, and differential; and the control parameters of the PID controller are adjusted based on the parameter variable value and the latest historical control parameters of each control item of the PID controller to obtain the latest proportional control parameter, integral control parameter, and differential control parameter; The blurred image acquisition module is configured to construct a joint loss function based on a reconstruction loss function and an adversarial loss function; the reconstruction loss function is a function constructed according to a Euclidean distance under a prerequisite mask layer; and the joint loss function is used as a loss function in the training process of the generative adversarial network. The error parameter determination module is configured to determine a first learning rate corresponding to a generator in the generative adversarial network and a second learning rate corresponding to a discriminator in the generative adversarial network according to the training result; and determine the current error and the current error rate of the generative adversarial network according to the first learning rate and the second learning rate, and a configured generator target learning rate and discriminator target learning rate, to obtain the error parameter.

6. An electronic device, comprising: Comprise: A memory is configured to save a computer program; A processor is configured to execute the computer program to implement the image inpainting method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored in the memory and executed by the processor to implement the image inpainting method according to any one of claims 1 to 4.

Images