A High-Resolution Image Restoration Method and Device

Abstract

The present invention discloses a high-resolution image restoration method and device. The method performs restoration training on the basis of adding a random mask to a real image and finally forms a target image restoration model, which solves the problem of poor restoration quality when an already trained image restoration network is applied to the restoration of super-resolution pictures. When the target image restoration model is used to restore a high-resolution picture, there is no need to redesign the restoration network structure, retrain the network, or introduce additional variables and parameters. The idea is simple and effective.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and particularly to a high-resolution image restoration method and device. Background Art

[0002] Image restoration technology is to use the information of the undamaged image area to infer the color and structure that the damaged area of the image should have, and then fill the damaged area to achieve the purpose of restoring the image. In actual industrial production and applications, images are often damaged due to noise corrosion, dust on the lens, human occlusion, or image graffiti.

[0003] The quality of the image obtained by the restoration network needs to be evaluated to evaluate the pros and cons of the algorithm. This helps the model training. Image evaluation calculates a certain measure value according to a certain criterion or index. Image evaluation metrics are standards for objectively evaluating the pros and cons of images and their processing algorithms, and they are basically universal in restoration processes such as image denoising, deblurring, defogging, hole filling, and super-resolution tasks.

[0004] In the engineering field, digital image restoration technology can be applied to aspects such as image super-resolution analysis, error concealment during image and video transmission, image coding compression, and image and video stabilization. Image super-resolution analysis is a technology that uses low-resolution images to reconstruct high-resolution images. In this field, digital restoration technology can be used to identify pixels in low-resolution images and use them as known information for high-resolution images, and then fill the damaged information areas to obtain better results.

[0005] Currently, most image restoration technologies are based on traditional technologies that extract features manually. The two main traditional digital image restoration algorithms are one based on structure and the other based on texture. The structure-based method is mainly used to restore images with small damage scales and cannot restore textures; the texture-based restoration method is mainly for restoring larger damage scales, but it ignores the image edge structure and requires a large amount of time.

[0006] With the continuous enhancement of computing power and the emergence of emerging models such as image restoration networks, deep learning technology has been increasingly emphasized by researchers. Generally speaking, when training an image restoration network, it is required that the image resolutions of the training set and the test set are roughly the same. Usually, the test set images should not exceed the image resolution of the training set, otherwise, the restoration effect will be poor.

[0007] However, in the current image inpainting technology based on deep learning, there is a large gap in the image resolution between the training set and the test set, which easily makes the restored image have lower pixels than the original image, resulting in poor restoration effects. The current main solution is to reduce the resolution of the high-resolution test set to a resolution close to that of the training set for inpainting, and then upsample and restore it to the original resolution of the test set image through interpolation and other methods. The image restored in this way is bound to be blurred in many details, resulting in poor effects; or to cover the large-resolution test set by increasing the receptive field, but this method changes the network structure of the original training model and requires retraining the network to be effectively used. Summary of the Invention

[0008] In view of the above problems, the present invention provides a high-resolution image inpainting method and device for overcoming the above problems or at least partially solving the above problems.

[0009] The present invention provides the following solutions:

[0010] A high-resolution image inpainting method, comprising:

[0011] Using a mask generator to randomly generate masks of different shapes through a large mask generation strategy;

[0012] Using the mask to occlude the training original image to form four-channel input tensor data;

[0013] Putting the input tensor data into an image inpainting network for training to form an image inpainting model; the image inpainting network includes intermediate features;

[0014] Processing the high-resolution image to be inpainted to obtain a low-resolution image to be inpainted, and the resolution of the low-resolution image to be inpainted is adapted to the image inpainting network;

[0015] Using the high-resolution image to be inpainted and the low-resolution image to be inpainted to optimize the intermediate features in the image inpainting model to obtain target intermediate features;

[0016] Using the target intermediate features to replace the intermediate features in the image inpainting model to obtain a target image inpainting model;

[0017] Inputting the high-resolution image to be inpainted into the target image inpainting model so that the target image inpainting model outputs a target image inpainting result.

[0018] Preferably: Stacking the mask and the training original image to form a mask image; using the mask image and the mask to form the input tensor data.

[0019] Preferably: putting the input tensor data into an inpainting network for training to form an inpainting model, including:

[0020] Putting the input tensor data into the inpainting network and performing bicubic interpolation downsampling processing;

[0021] The downsampled data passes through intermediate features, and the input tensor data is convolved using a convolutional layer to obtain a convolutional image;

[0022] Using an upsampling module to upsample the features of the convolutional image, and combining with the objective function and the original training image for supervised learning to obtain the inpainting model.

[0023] Preferably: the mathematical model of the bicubic interpolation downsampling is shown as the following formula:

[0024]

[0025] Where P LR is the low-resolution image, P HR is the high-resolution image, δ is the mapping parameter (noise or scaling factor), represents the convolution operation of the blur kernel k and the high-resolution image, η α represents Gaussian white noise with a standard deviation of α, ↓ S is the downsampling operation with a reduction factor of s for the image.

[0026] Preferably: the mathematical model of the upsampling module is shown as the following formula:

[0027]

[0028] In the formula: T(i,j) is the position projection function, (i’,j’) is the pixel value coordinate after projection calculated using the magnification factor; W(i,j) is the weight estimation function; Ω is the feature function; F is the image feature value; W is the weight estimation.

[0029] Preferably: the objective function is shown as the following formula:

[0030]

[0031] In the formula: θ represents the parameter; Φ(θ) represents the regularization term; is the restored image; L is the loss function, G(i,j) represents the pixel restoration value at the coordinate (i,j) position, and F(i,j) represents the actual pixel value at the coordinate (i,j) position.

[0032] Preferably: the loss function L includes the image evaluation function MSE.

[0033] Preferably: Processing the high-resolution image to be restored to obtain a low-resolution image to be restored, including:

[0034] Processing the high-resolution image to be restored through Gaussian filtering and linear interpolation to obtain the low-resolution image to be restored.

[0035] Preferably: Optimizing the intermediate features in the image restoration model by using the high-resolution image to be restored and the low-resolution image to be restored to obtain target intermediate features; including:

[0036] Inputting the low-resolution image to be restored into the image restoration model to obtain a low-resolution restored image;

[0037] Inputting the high-resolution image to be restored into the image restoration model to obtain a prediction result;

[0038] Calculating the L2 loss between the prediction result and the low-resolution restored image, and optimizing the intermediate features by using the backpropagation algorithm and the Adam gradient descent algorithm;

[0039] Iteratively optimizing the intermediate features until the preset number of optimization times and the L2 loss are small enough to obtain the target intermediate features.

[0040] A high-resolution image restoration device, including:

[0041] A mask generation unit, configured to randomly generate masks of different shapes by using a large mask generation strategy through a mask generator;

[0042] An input tensor data formation unit, configured to occlude the training original image by using the mask to form four-channel input tensor data;

[0043] An image restoration model formation unit, configured to put the input tensor data into an image restoration network for training to form an image restoration model; the image restoration network includes intermediate features;

[0044] A low-resolution image to be restored acquisition unit, configured to process the high-resolution image to be restored to obtain a low-resolution image to be restored, and the resolution of the low-resolution image to be restored is adapted to the image restoration network;

[0045] A target intermediate feature acquisition unit, configured to optimize the intermediate features in the image restoration model by using the high-resolution image to be restored and the low-resolution image to be restored to obtain target intermediate features;

[0046] A target image restoration model acquisition unit, configured to replace the intermediate features in the image restoration model with the target intermediate features to obtain a target image restoration model;

[0047] A target image restoration result output unit for inputting the high-resolution image to be restored into the target image restoration model, so that the target image restoration model outputs a target image restoration result.

[0048] According to the specific embodiments provided by the present invention, the following technical effects are disclosed:

[0049] A high-resolution image restoration method and apparatus provided by an embodiment of the present application. The method performs restoration training on the basis of adding a random mask to a real image and finally forms a target image restoration model, which solves the problem of poor restoration quality when an already trained image restoration network is applied to the restoration of super-resolution pictures. When using the target image restoration model to restore a high-resolution picture, there is no need to redesign the restoration network structure, retrain the network, or introduce additional variables and parameters, and the idea is simple and effective.

[0050] Of course, it is not necessary for any product implementing the present invention to achieve all the above advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0052] Figure 1 is a flowchart of a high-resolution image restoration method provided by an embodiment of the present invention;

[0053] Figure 2 is a framework diagram of an image restoration model provided by an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of the model training stage and the intermediate feature optimization stage provided by an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of a high-resolution image restoration apparatus provided by an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of a high-resolution image restoration device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0058] See Figure 1 , a high-resolution image restoration method provided by an embodiment of the present invention. As Figure 1 shown, the method may include:

[0059] S101: Use a mask generator to randomly generate masks of different shapes through a large mask generation strategy;

[0060] S102: Use the mask to occlude the training original image to form four-channel input tensor data; specifically, stack the mask and the training original image to form a mask image; use the mask image and the mask to form the input tensor data.

[0061] S103: Put the input tensor data into an image restoration network for training to form an image restoration model; the image restoration network includes intermediate features; specifically, putting the input tensor data into an image restoration network for training to form an image restoration model includes:

[0062] Put the input tensor data into the image restoration network and perform bicubic interpolation downsampling processing;

[0063] The data after downsampling processing passes through intermediate features, and a convolutional layer is used to perform convolutional calculation on the input tensor data to obtain a convolutional image;

[0064] Use an upsampling module to upsample the features of the convolutional image, and perform supervised learning in combination with the objective function and the training original image to obtain the image restoration model.

[0065] The mathematical model of the bicubic interpolation downsampling is shown as the following formula:

[0066]

[0067] where P LR is the low-resolution image, P HR is the high-resolution image, δ is the mapping parameter (noise or scaling factor), represents the convolution operation of the blur kernel k and the high-resolution image, η α represents Gaussian white noise with a standard deviation of α, ↓ S is the downsampling operation with a reduction factor of s for the image.

[0068] The mathematical model of the upsampling module is shown as follows:

[0069]

[0070] Where: T(i, j) is the position projection function, (i’, j’) is the pixel value coordinate after projection calculated using the magnification factor; W(i, j) is the weight estimation function; Ω is the feature function; F is the image feature value; W is the weight estimation.

[0071] The objective function is shown as follows:

[0072]

[0073] Where: θ represents the parameter; Φ(θ) represents the regularization term; is the restored image; L is the loss function, G(i, j) represents the pixel restoration value at the coordinate (i, j), and F(i, j) represents the actual pixel value at the coordinate (i, j).

[0074] Furthermore, the loss function L includes the image evaluation function MSE.

[0075] S104: Process the high-resolution image to be restored to obtain a low-resolution image to be restored, and the resolution of the low-resolution image to be restored is adapted to the image restoration network; specifically, processing the high-resolution image to be restored to obtain a low-resolution image to be restored includes:

[0076] Pass the high-resolution image to be restored through Gaussian filtering and linear interpolation to obtain the low-resolution image to be restored.

[0077] S105: Optimize the intermediate features in the image restoration model using the high-resolution image to be restored and the low-resolution image to be restored to obtain target intermediate features; specifically, optimizing the intermediate features in the image restoration model using the high-resolution image to be restored and the low-resolution image to be restored to obtain target intermediate features includes:

[0078] Input the low-resolution image to be restored into the image restoration model to obtain a low-resolution restored image;

[0079] Input the high-resolution image to be restored into the image restoration model to obtain a prediction result.

[0080] Calculate the L2 loss between the prediction result and the low-resolution restored image, and optimize the intermediate features using the backpropagation algorithm and the Adam gradient descent algorithm;

[0081] Iteratively optimize the intermediate feature until the target intermediate feature is obtained when the preset number of optimization times and the L2 loss are small enough.

[0082] S106: Replace the intermediate feature in the image inpainting model with the target intermediate feature to obtain a target image inpainting model;

[0083] S107: Input the high-resolution image to be inpainted into the target image inpainting model so that the target image inpainting model outputs a target image inpainting result.

[0084] The high-resolution image inpainting method provided by the embodiments of this application performs inpainting training on the basis of adding a random mask to a real image and finally forms a target image inpainting model.

[0085] The specific technical idea steps are as follows:

[0086] (1) Train a mask generator. Adopt a large mask generation strategy to randomly generate a mask with a large occlusion to occlude the training original image and form a four-channel input tensor data;

[0087] (2) Put the input tensor data into an image inpainting network for training to form an image inpainting model;

[0088] (3) Input the super-resolution image to be inpainted into the trained image inpainting model and iteratively optimize the intermediate feature M;

[0089] (4) Finally, the target image inpainting network with the optimized intermediate feature outputs the final target image inpainting result.

[0090] This method can use a mask generator to generate a random mask and stack it with the masked image to form a four-channel input tensor data as the input. An image inpainting network that has been trained needs to be available before optimizing the intermediate feature. The optimization of the intermediate feature calculates the L2 loss between the restoration result of the non-downsampled image to be inpainted and the restoration result of the downsampled image to be inpainted, and iteratively optimizes the intermediate feature through the backpropagation algorithm and the Adam gradient descent algorithm.

[0091] The image inpainting model framework provided by the embodiments of this application is as Figure 2 shown, demonstrating the overall idea of the solution implementation. The overall network structure provided by the embodiments of this application is generally similar to an encoder-decoder network. Use a mask generator to stack the mask m with the masked image to form a four-channel input tensor x'. The mask generator will randomly generate different masks to occlude the image. The downsampling strategy uses bicubic interpolation, and the upsampling strategy uses multiple factors. Before upsampling, the inpainting network processes the intermediate feature M in a fully convolutional manner, and outputs a restored three-channel color image after upsampling.

[0092] The present invention adopts a bicubic interpolation downsampling operation D, specifically:

[0093]

[0094] where P LR is a low-resolution image, P HR is a high-resolution image, δ is a mapping parameter (noise or scaling factor), represents the convolution operation of the blur kernel k and the high-resolution image, η α represents Gaussian white noise with a standard deviation of α, ↓ S is a downsampling operation with a reduction factor of s for the image.

[0095] After the intermediate feature M is processed by convolution, the upsampling module magnifies the image. For a certain pixel point in the LR image (super-resolution image), the upsampling module always first finds its position, then estimates the corresponding weight according to the super-resolution factor r, and then calculates the pixel value of the corresponding pixel point in the HR image (video blurred image) based on this weight. The upsampling module mainly includes three important functions, namely weight estimation W, position projection T, and feature mapping Ω, where F represents the feature and G represents the target. The mathematical model is as follows:

[0096]

[0097] In the formula: The position projection function T(i, j) calculates the pixel value coordinates (i’, j’) after projection using the magnification factor; the weight estimation function W(i, j), which predicts the required weight value according to the result of the position projection function; the feature function Ω will calculate the upsampled pixel value and the corresponding position using the image feature value F and the weight estimation W, so as to realize the transformation from the LR image to the HR image.

[0098] The purpose of training is to obtain the parameter θ under the condition of the minimum loss, and its objective function is:

[0099]

[0100] In the formula: Φ(θ) represents the regularization term; is the restored image; L is the loss function, which is represented by the image evaluation function MSE here, G(i, j) represents the pixel restoration value at the coordinate (i, j) position, and F(i, j) represents the actual pixel value at the coordinate (i, j) position.

[0101] Generally, for an image inpainting network trained based on deep learning, there is a problem that when the resolution gap between the image to be inpainted and the input image of the training model is large, the restored image is likely to have lower pixels than the original image, resulting in poor restoration effect. To solve the above problem, in the embodiments of the present application, by optimizing the intermediate feature M, the problems that the model needs to modify the network structure and undergo additional training before it can be used, and introducing additional variables and parameters are cleverly avoided.

[0102] To fully illustrate more details of this method, the technical process of this method is described below.

[0103] Figure 3 The technical flow chart of this method is shown. The process mainly includes two stages. One is the training stage of the pre - model, and the other is the intermediate feature optimization stage using the trained model. The specific process is as follows:

[0104] Pre - model training stage:

[0105] S1: Use the mask generator to randomly generate masks of different shapes through the large - mask generation strategy.

[0106] S2: Stack the mask m and the training original image x to form the masked image y.

[0107] S3: Combine the masked image y and the mask m to form a 4 - channel input tensor data.

[0108] S4: Take the 4 - channel input tensor data as the input, and first perform bicubic interpolation downsampling processing.

[0109] S5: The downsampled data passes through the intermediate feature M, and the convolutional layer is used to perform convolution calculation on the input tensor data to obtain the convolutional image.

[0110] S6: Use the upsampling module to upsample the convolutional image features, combine the objective function and the training original image for supervised learning, and finally form an image inpainting model.

[0111] Intermediate feature optimization stage:

[0112] S7: Combine the image inpainting model trained in S6. The high - resolution image to be inpainted is processed by Gaussian filtering and linear interpolation to reduce its resolution to the most suitable resolution for the network, and then the inpainted image at this resolution is input into the image inpainting model to obtain a low - resolution inpainted image.

[0113] S8: Input the high - resolution image to be inpainted into the image inpainting model as well to obtain the prediction result.

[0114] S9: Calculate the L2 loss between the prediction result and the low - resolution inpainted image, and use the backpropagation algorithm and the Adam gradient descent algorithm to optimize the intermediate feature M.

[0115] S10: Repeat the steps of S9 for iterative optimization. Stop the optimization and obtain the final intermediate feature M~ when a certain number of optimization iterations is reached and the L2 loss is small enough.

[0116] S11: Replace the intermediate feature M in the image inpainting model with the final intermediate feature M~, and input the high-resolution image to be inpainted again to obtain the final improved inpainting effect (the target image inpainting result).

[0117] In summary, the high-resolution image inpainting method provided in this application performs inpainting training based on adding a random mask to a real image and finally forms a target image inpainting model. It solves the problem of poor inpainting quality when an already trained image inpainting network is applied to super-resolution image inpainting. When using this target image inpainting model to inpaint high-resolution images, there is no need to redesign the inpainting network structure, retrain the network, or introduce additional variables and parameters. The idea is simple and effective.

[0118] See Figure 4 , this embodiment of the application can also provide a high-resolution image inpainting device, as Figure 4 shown. The device may include:

[0119] A mask generation unit 401, configured to randomly generate masks of different shapes by using a large mask generation strategy through a mask generator;

[0120] An input tensor data forming unit 402, configured to occlude the training original image by using the mask to form four-channel input tensor data;

[0121] An image inpainting model forming unit 403, configured to put the input tensor data into an image inpainting network for training to form an image inpainting model; the image inpainting network includes an intermediate feature;

[0122] A low-resolution image to be inpainted acquisition unit 404, configured to process the high-resolution image to be inpainted to obtain a low-resolution image to be inpainted, and the resolution of the low-resolution image to be inpainted is adapted to the image inpainting network;

[0123] A target intermediate feature acquisition unit 405, configured to optimize the intermediate feature in the image inpainting model by using the high-resolution image to be inpainted and the low-resolution image to be inpainted to obtain a target intermediate feature;

[0124] A target image inpainting model acquisition unit 406, configured to replace the intermediate feature in the image inpainting model with the target intermediate feature to obtain a target image inpainting model;

[0125] The target image restoration result output unit 407 is configured to input the high-resolution image to be restored into the target image restoration model, so that the target image restoration model outputs a target image restoration result.

[0126] An embodiment of the present application may further provide a high-resolution image restoration device, where the device includes a processor and a memory:

[0127] The memory is used to store program code and transmit the program code to the processor;

[0128] The processor is configured to execute the steps of the above high-resolution image restoration method according to the instructions in the program code.

[0129] As Figure 5 shown, a high-resolution image restoration device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all complete communication with each other through the communication bus 13.

[0130] In an embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices, etc.

[0131] The processor 10 may call the program stored in the memory 11. Specifically, the processor 10 may execute the operations in the embodiment of the high-resolution image restoration method.

[0132] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operation instructions. In an embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0133] Using a mask generator, different-shaped masks are randomly generated through a large mask generation strategy;

[0134] Using the mask to occlude the training original image to form four-channel input tensor data;

[0135] Putting the input tensor data into an image restoration network for training to form an image restoration model; the image restoration network includes intermediate features;

[0136] Processing the high-resolution image to be restored to obtain a low-resolution image to be restored, where the resolution of the low-resolution image to be restored is adapted to the image restoration network;

[0137] Optimize the intermediate features in the image restoration model by using the high-resolution image to be restored and the low-resolution image to be restored to obtain target intermediate features;

[0138] Replace the intermediate features in the image restoration model with the target intermediate features to obtain a target image restoration model;

[0139] Input the high-resolution image to be restored into the target image restoration model so that the target image restoration model outputs a target image restoration result.

[0140] In a possible implementation, the memory 11 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function (such as a file creation function and a data reading and writing function), etc.; the data storage area may store data created during use, such as initialization data, etc.

[0141] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage devices.

[0142] The communication interface 12 may be an interface of a communication module for connecting to other devices or systems.

[0143] Of course, it should be noted that Figure 5 the structure shown does not constitute a limitation on the high-resolution image restoration device in the embodiments of the present application. In actual applications, the high-resolution image restoration device may include more or fewer components than Figure 5 those shown, or combine some components.

[0144] The embodiments of the present application may also provide a computer-readable storage medium for storing program codes for executing the steps of the above high-resolution image restoration method.

[0145] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0146] As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0147] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for a system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0148] The above is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A high-resolution image restoration method, characterized in that, Including: Using a mask generator, different-shaped masks are randomly generated through a large mask generation strategy; Using the mask to occlude the training original image to form four-channel input tensor data; Putting the input tensor data into an image inpainting network for training to form an image inpainting model; The image inpainting network includes intermediate features; The method for forming the image inpainting model includes: Putting the input tensor data into the image inpainting network and performing bicubic interpolation downsampling processing; The data after downsampling processing passes through the intermediate features, and a convolutional layer is used to perform convolutional calculation on the input tensor data to obtain a convolutional image; Using an upsampling module to upsample the features of the convolutional image, and combining with an objective function and the training original image for supervised learning to obtain the image inpainting model; Processing a high-resolution image to be inpainted to obtain a low-resolution image to be inpainted, and the resolution of the low-resolution image to be inpainted is adapted to the image inpainting network; Using the high-resolution image to be inpainted and the low-resolution image to be inpainted to optimize the intermediate features in the image inpainting model to obtain target intermediate features; The method for optimizing the target intermediate features includes: Inputting the low-resolution image to be inpainted into the image inpainting model to obtain a low-resolution inpainted image; Inputting the high-resolution image to be inpainted into the image inpainting model to obtain a prediction result; Performing L2 loss calculation between the prediction result and the low-resolution inpainted image, and using the backpropagation algorithm and the Adam gradient descent algorithm to optimize the intermediate features; Iteratively optimizing the intermediate features until the preset number of optimization times and the L2 loss are small enough to obtain the target intermediate features; Using the target intermediate features to replace the intermediate features in the image inpainting model to obtain a target image inpainting model; Inputting the high-resolution image to be inpainted into the target image inpainting model so that the target image inpainting model outputs a target image inpainting result.

2. The high-resolution image restoration method according to claim 1, characterized in that, Stacking the mask and the training original image to form a mask image; Using the mask image and the mask to form the input tensor data.

3. The high-resolution image restoration method according to claim 1, wherein The mathematical model of the bicubic interpolation downsampling is shown as the following formula: Among them P LR is a low-resolution image, P HR is a high-resolution image, δ is the mapping parameter noise or the scaling factor, represents the convolution operation of the blur kernel k and the high-resolution image, η α represents that the standard deviation is α of white Gaussian noise, ↓ S is the downsampling operation with a reduction factor of s for the image.

4. The high-resolution image restoration method according to claim 1, characterized in that The mathematical model of the upsampling module is shown as the following formula: where: T ( i , j ) is the position projection function, is the pixel value coordinate after projection calculated using the magnification factor; W ( i , j ) is the weight estimation function; Ω is the feature function; F is the image feature value; W For weight estimation.

5. The high-resolution image restoration method according to claim 1, wherein The objective function is shown as the following formula: In the formula: θ represents a parameter; Φ( θ ) represents a regularization term; is the restored image; L is the loss function, G ( i , j ) represents the pixel restoration value at the coordinate ( i , j ), F ( i , j ) represents the actual pixel value at the coordinate ( i , j ).

6. The high-resolution image restoration method according to claim 5, characterized in that The loss function L includes the image evaluation function MSE.

7. The high-resolution image restoration method according to claim 1, characterized in that, Processing a high-resolution image to be inpainted to obtain a low-resolution image to be inpainted includes: Passing the high-resolution image to be inpainted through Gaussian filtering and linear interpolation processing to obtain the low-resolution image to be inpainted.

8. A high-resolution image restoration device, characterized in that For implementing the high-resolution image inpainting method according to any one of claims 1-7, the apparatus includes: A mask generation unit, configured to use a mask generator to randomly generate masks with different shapes through a large mask generation strategy; An input tensor data formation unit, configured to use the mask to occlude the training original image to form four-channel input tensor data; An image inpainting model formation unit, configured to put the input tensor data into an image inpainting network for training to form an image inpainting model; The image inpainting network includes intermediate features; A low-resolution image to be restored acquisition unit, which is used to process a high-resolution image to be restored to obtain a low-resolution image to be restored, and the resolution of the low-resolution image to be restored is adapted to the image restoration network; A target intermediate feature acquisition unit, which is used to optimize the intermediate features in the image restoration model by using the high-resolution image to be restored and the low-resolution image to be restored to obtain target intermediate features; A target image restoration model acquisition unit, which is used to replace the intermediate features in the image restoration model with the target intermediate features to obtain a target image restoration model; A target image restoration result output unit, which is used to input the high-resolution image to be restored into the target image restoration model so that the target image restoration model outputs a target image restoration result.

Images