Image restoration method and device based on potential mask, equipment and medium

Abstract

The invention relates to the technical field of image restoration, and particularly provides an image restoration method, device and equipment based on a potential mask, and a medium, and the method comprises the steps: carrying out the mask processing of images in a big data set, carrying out the pre-training of an encoder-decoder architecture model after grouping according to different mask proportions, and carrying out the pre-training of an encoder-decoder architecture model; obtaining image restoration models with different mask proportions; evaluating the image restoration models of different mask proportions to obtain an optimal image restoration model of each group of mask proportion images; evaluating the damage proportion of the to-be-repaired image, and selecting an optimal image repair model of the corresponding mask proportion image as a basic model of the to-be-repaired image based on the damage proportion; performing mask processing on the visible part of the to-be-restored image, and then performing fine adjustment on the basic model; and inputting a to-be-restored image into the fine-tuned model for image restoration. And it can be ensured that the model has higher adaptability to images with different damage degrees.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and in particular relates to an image restoration method, device, equipment and medium based on a latent mask. Background Art

[0002] In the field of image processing, image restoration is a crucial technology that aims to restore damaged or missing parts of an image to make it look complete and natural. Traditional image restoration methods usually follow a fixed process: first, the model is pre-trained using existing large datasets to enable it to have basic image understanding and generation capabilities; then, the model is fine-tuned for specific application scenarios or datasets to further improve its performance in this field.

[0003] However, these traditional methods have a significant limitation in the specific execution steps of image restoration. That is, after the model is pre-trained and fine-tuned, the model weights remain unchanged when the actual image restoration operation is performed. This strategy ignores the important information contained in the visible part of the image to be restored. This information is extremely valuable to the model because it directly reflects the unique features and contextual information of the image and is the key basis for the model to learn and restore the missing parts.

[0004] Because traditional methods fail to fully utilize the information of these visible parts, the quality of the restored image is not satisfactory in some cases. Specifically, the degree of integration between the restored area and the surrounding environment is not high enough, the edge processing is not natural enough, or the overall tone and texture are obviously different from the original image. These problems not only affect the visual effect of the image, but also limit the application of image restoration technology in more practical scenarios. Summary of the invention

[0005] In view of the problem that traditional image restoration methods cannot effectively utilize the information of the visible part of the image to be restored, resulting in poor quality of the restored image in some cases, the present invention provides an image restoration method, device, equipment and medium based on latent mask.

[0006] In a first aspect, the technical solution of the present application provides an image restoration method based on a latent mask, comprising: After masking the images in the large data set and grouping them according to different mask ratios, the encoder-decoder architecture model is pre-trained to obtain image restoration models with different mask ratios. The image inpainting models with different mask ratios are evaluated to obtain the optimal image inpainting model for each group of mask ratio images; Evaluate the damage ratio of the image to be repaired, and select the optimal image repair model corresponding to the mask ratio image as the basic model of the image to be repaired based on the damage ratio; After masking the visible part of the image to be repaired, the base model is fine-tuned; The image to be repaired is input into the fine-tuned model for image repair.

[0007] As a further limitation of the technical solution of the present invention, after performing mask processing on the images in the large data set and grouping them according to different mask ratios, the encoder-decoder architecture model is pre-trained to obtain the image restoration model with different mask ratios, including the following steps: The encoder based on the attention mechanism encodes the images in the large data set; Select some images from the large data set for mask processing and group them according to different mask ratios; Use the decoder to restore the masked portion of the image for each set of mask ratios; Obtain the difference between the image restored by the decoder and the image of the original masked part, and express it using the square error loss function; and optimize the loss function through the gradient descent method; When the difference between the image restored by the decoder and the image of the original masked part is less than a set threshold, the training is completed and an image restoration model with different mask ratios is obtained; wherein the image restoration model includes basic restoration knowledge.

[0008] By encoding the image with an encoder based on the attention mechanism, the model can more accurately capture the key information and features in the image. This helps to restore the masked part more accurately in the subsequent decoding process, thereby improving the accuracy of image restoration. Grouping training for images with different mask ratios enables the model to adapt to a variety of image damage conditions of different degrees. This training method enhances the generalization ability of the model, so that it can give relatively satisfactory restoration results when facing images with different degrees of damage. The square error loss function is used to represent the difference between the image restored by the decoder and the image of the original masked part, and the loss function is optimized by the gradient descent method. This training strategy can efficiently guide the update of model parameters, so that the model continuously approaches the optimal solution during the training process, thereby optimizing the training process. The image restoration model obtained through pre-training contains basic restoration knowledge, which provides strong support for subsequent model fine-tuning. In practical applications, the corresponding image restoration model can be selected for fine-tuning according to the damage of the image to be repaired, thereby further improving the pertinence and accuracy of the restoration.

[0009] As a further limitation of the technical solution of the present invention, the step of encoding the images in the large data set by the encoder based on the attention mechanism includes: Divide the image into multiple small blocks and straighten each small block into a one-dimensional vector; Combining the one-dimensional vectors to obtain a two-dimensional vector; Perform three matrix operations on the two-dimensional vector to obtain the basic matrices Q, K, and V for the three attention mechanism operations; so that each original one-dimensional vector contains the information of the entire image.

[0010] As a further limitation of the technical solution of the present invention, the step of restoring the masked portion of the image of each set of mask ratios using a decoder comprises: The decoder generates a two-dimensional vector by reversely converting the one-dimensional vector, and restores the two-dimensional vector back to multiple small image blocks and then splices them into an image.

[0011] As a further limitation of the technical solution of the present invention, the step of evaluating the image restoration models of different mask ratios to obtain the optimal image restoration model for each group of mask ratio images includes: Perform mask processing of a set mask ratio on part of the image, input different mask ratio images into the decoder of the image inpainting model to restore the masked image and calculate the difference loss value; The image restoration model with the smallest difference value is selected as the optimal image restoration model for the mask ratio image.

[0012] As a further limitation of the technical solution of the present invention, after performing mask processing on the visible part of the image to be repaired, the step of fine-tuning the basic model includes: The visible part of the image to be repaired is taken as the original image, the original image is masked, and the decoder of the basic model predicts the masked part and calculates the difference value, which is evaluated using the mean square error loss function, and the gradient descent algorithm is used to optimize the loss function until the model contains the feature information of the picture to obtain a fine-tuned model, and the fine-tuned model contains the feature information of the repaired picture.

[0013] As a further limitation of the technical solution of the present invention, the step of inputting the image to be repaired into the fine-tuned model for image repair includes: The image to be repaired is received, and based on basic repair knowledge and repair picture feature information decoder, the part of the image that needs to be repaired is restored and the repaired image is output.

[0014] In a second aspect, the technical solution of the present invention further provides an image restoration device based on a latent mask, comprising a pre-training module, an optimal matching module, a basic model confirmation module, a fine-tuning module and an execution restoration module; The pre-training module is used to mask the images in the large data set and group them according to different mask ratios, and then pre-train the encoder-decoder architecture model to obtain image restoration models with different mask ratios; The optimal matching module is used to evaluate the image restoration models with different mask ratios to obtain the optimal image restoration model for each group of mask ratio images; A basic model confirmation module is used to evaluate the damage ratio of the image to be repaired, and select the optimal image repair model corresponding to the mask ratio image as the basic model of the image to be repaired based on the damage ratio; The fine-tuning module is used to fine-tune the basic model after masking the visible part of the image to be repaired; The execution restoration module is used to input the image to be restored into the fine-tuned model for image restoration.

[0015] As a further limitation of the technical solution of the present invention, the pre-training module includes an encoding unit, a mask processing unit, a decoding unit, a loss optimization unit and an image restoration model output unit; The encoding unit is used to encode images in a large data set based on an attention mechanism encoder; A mask processing unit, used to select some images in the large data set for mask processing and group them according to different mask ratios; A decoding unit, configured to restore the masked portion of the image of each set of mask ratios using a decoder; The loss optimization unit is used to obtain the difference between the image restored by the decoder and the image of the original masked part, expressed using the square error loss function; and optimize the loss function using the gradient descent method; The image restoration model output unit is used to determine that when the difference between the image restored by the decoder and the image of the original masked part is less than a set threshold, the training is completed and an image restoration model with different mask ratios is obtained; wherein the image restoration model includes basic restoration knowledge.

[0016] The present invention uses latent mask technology to ensure that the model can learn the information of the image to be repaired before repairing the image, rather than just based on the previous data set. Application of contrast loss function: solve the problem of imbalanced training data and improve the robustness of the model.

[0017] As a further limitation of the technical solution of the present invention, the encoding unit is specifically used to divide the image into multiple small blocks and straighten each small block into a one-dimensional vector; combine the one-dimensional vectors to obtain a two-dimensional vector; perform three matrix operations on the two-dimensional vector to obtain the basic matrices Q, K, and V of the three attention mechanism operations; so that each original one-dimensional vector contains the information of the entire image.

[0018] As a further limitation of the technical solution of the present invention, the decoding unit is specifically used to reversely generate a two-dimensional vector through a one-dimensional vector, and restore the two-dimensional vector back to multiple small picture blocks and then splice them into a picture.

[0019] As a further limitation of the technical solution of the present invention, the optimal matching module is specifically used to perform mask processing on a part of the image with a set mask ratio, input the decoder of the image restoration model of different mask ratio images, restore the masked image and calculate the difference loss value; select the image restoration model with the smallest difference value as the optimal image restoration model of the mask ratio image.

[0020] As a further limitation of the technical solution of the present invention, the fine-tuning module is specifically used to take the visible part of the image to be repaired as the original image, mask the original image, and let the decoder of the basic model predict the masked part and calculate the difference value, use the mean square error loss function for evaluation, and use the gradient descent algorithm to optimize the loss function until the model contains the feature information of the picture to obtain a fine-tuned model, and the fine-tuned model contains the feature information of the repaired picture.

[0021] As a further limitation of the technical solution of the present invention, an execution repair module is specifically used to receive an image to be repaired, restore the part of the image that needs to be repaired based on basic repair knowledge and a repair picture feature information decoder, and output the repaired image.

[0022] In a third aspect, the technical solution of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the image restoration method based on latent mask as described in the first aspect.

[0023] In a fourth aspect, the technical solution of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the latent mask-based image restoration method as described in the first aspect.

[0024] It can be seen from the above technical solutions that the present invention has the following advantages: by processing images in a large data set with different mask ratios and training them to obtain image restoration models with different mask ratios, it can be ensured that the model has a stronger adaptability to images with different degrees of damage. When repairing the image to be repaired, the optimal restoration model can be selected according to its damage ratio, thereby significantly improving the restoration effect. By pre-training on a large-scale data set and group training for different mask ratios, the method can obtain multiple image restoration models with different generalization capabilities. These models can jointly cope with image restoration tasks in various complex scenes and improve the generalization performance of the model.

[0025] This method can select the optimal repair model for fine-tuning according to the damage ratio of the image to be repaired. This adaptive repair mechanism enables the method to flexibly deal with images of different degrees of damage, improving the flexibility and accuracy of repair. Since this method adopts a combination of pre-training and fine-tuning, it is not necessary to train the model from scratch when repairing new images, which greatly reduces the cost and time of repair. At the same time, since the model has been fully pre-trained, it can converge quickly in the fine-tuning stage, further improving the efficiency of repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0028] Figure 2 is a schematic block diagram of an apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Traditional image restoration methods, after model pre-training and fine-tuning, do not change the model weights during the image restoration step. This method cannot effectively utilize the information of the visible part of the image to be restored, and this information is the most helpful information for the model to learn the characteristics of the image, resulting in poor quality of the restored image in some cases. To this end, the present application proposes a method in which a model first learns the information of the visible part of the image to be restored and then performs image restoration. In order to enable personnel in this technical field to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0030] In order to clearly describe the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below: Encoder: In the encoder-decoder architecture, the encoder is responsible for encoding the input data (such as an image) into a fixed-length continuous representation, usually used to extract features.

[0031] Masked: A technique used in encoder-decoder architectures to mask parts of the data to train the model so that it can predict or recover the masked data.

[0032] Latent Masked: First mask the visible part of the image that needs to be repaired and decode it, and then repair the missing part of the image after training is completed.

[0033] Decoder: Corresponding to the encoder, the decoder is responsible for decoding the output of the encoder (i.e., the potential representation) back to the form of original data, or generating new data, such as missing parts in image restoration.

[0034] Image Inpainting: An image processing technique that focuses on filling in missing or damaged parts of an image to restore the integrity and beauty of the image.

[0035] Self-attention: A mechanism that allows a model to dynamically pay attention to other elements in a sequence while processing a sequence of data. This is particularly useful when processing images because it can capture long-range dependencies between different parts of an image.

[0036] The technical solution of the present application provides an image restoration method based on latent masks. First, the model of the encoder-decoder architecture is trained by adding mask technology to learn how to predict missing images. Then, the information of the image is learned by fine-tuning the mask based on the visible part of the pre-repaired image, and finally the image of the damaged part is predicted by the decoder. A model with better restoration effect is obtained, which avoids the problem of reduced transfer ability caused by traditional image restoration methods only trained on pre-training datasets, and effectively improves the model's learning of prior knowledge of pre-repaired images; Figure 1 As shown, the method includes: Step 1: After masking the images in the large data set and grouping them according to different mask ratios, the encoder-decoder architecture model is pre-trained to obtain image restoration models with different mask ratios; This step specifically includes: Step 11: The encoder based on the attention mechanism encodes the images in the large data set; Step 12: Select some images from the large data set for mask processing and group them according to different mask ratios; Step 13: Use the decoder to restore the masked part of the image of each set of mask ratios; Step 14: Obtain the difference between the image restored by the decoder and the image of the original masked part, and express it using the square error loss function; and optimize the loss function by the gradient descent method; Step 15: When the difference between the image restored by the decoder and the image of the original masked part is determined to be less than a set threshold, the training is completed and an image restoration model with different mask ratios is obtained; wherein the image restoration model includes basic restoration knowledge.

[0037] Step 2: Evaluate the image restoration models with different mask ratios to obtain the optimal image restoration model for each group of mask ratio images; Specifically, this step includes the following: Perform mask processing of a set mask ratio on part of the image, input different mask ratio images into the decoder of the image inpainting model to restore the masked image and calculate the difference loss value; The image restoration model with the smallest difference value is selected as the optimal image restoration model for the mask ratio image.

[0038] Step 3: Evaluate the damage ratio of the image to be repaired, and select the optimal image repair model corresponding to the mask ratio image based on the damage ratio as the basic model of the image to be repaired; Step 4: After masking the visible part of the image to be repaired, fine-tune the basic model; The visible part of the image to be repaired is taken as the original image, the original image is masked, and the decoder of the basic model predicts the masked part and calculates the difference value, which is evaluated using the mean square error loss function, and the gradient descent algorithm is used to optimize the loss function until the model contains the feature information of the picture to obtain a fine-tuned model, and the fine-tuned model contains the feature information of the repaired picture.

[0039] Step 5: Input the image to be repaired into the fine-tuned model for image repair. Specifically, receive the image to be repaired, restore the part of the image that needs to be repaired based on basic repair knowledge and repair image feature information decoder, and output the repaired image.

[0040] By encoding the image with an encoder based on the attention mechanism, the model can more accurately capture the key information and features in the image. This helps to restore the masked part more accurately in the subsequent decoding process, thereby improving the accuracy of image restoration. Grouping training for images with different mask ratios enables the model to adapt to a variety of image damage conditions of different degrees. This training method enhances the generalization ability of the model, so that it can give relatively satisfactory restoration results when facing images with different degrees of damage. The square error loss function is used to represent the difference between the image restored by the decoder and the image of the original masked part, and the loss function is optimized by the gradient descent method. This training strategy can efficiently guide the update of model parameters, so that the model continuously approaches the optimal solution during the training process, thereby optimizing the training process. The image restoration model obtained through pre-training contains basic restoration knowledge, which provides strong support for subsequent model fine-tuning. In practical applications, the corresponding image restoration model can be selected for fine-tuning according to the damage of the image to be repaired, thereby further improving the pertinence and accuracy of the restoration.

[0041] In some embodiments, the step of encoding images in a large data set based on an attention mechanism encoder includes: Divide the image into multiple small blocks and straighten each small block into a one-dimensional vector; Combining the one-dimensional vectors to obtain a two-dimensional vector; Perform three matrix operations on the two-dimensional vector to obtain the basic matrices Q, K, and V for the three attention mechanism operations; so that each original one-dimensional vector contains the information of the entire image.

[0042] Accordingly, the steps of restoring the masked portion of each set of masked ratio images using the decoder include: The decoder generates a two-dimensional vector by reversely converting the one-dimensional vector, and restores the two-dimensional vector back to multiple small image blocks and then splices them into an image.

[0043] It should be noted that the present invention constructs an image restoration scheme divided into two key stages: training and restoration. This method is applicable to the encoder-decoder architecture model based on the attention mechanism, which is one of the current mainstream architecture models and has certain applicability.

[0044] Before introducing the training, let's first introduce the structure and function of the model. This method is applicable to the model of the encoder-decoder architecture. And the model as a whole is based on information extraction implemented by the attention mechanism.

[0045] First, let's introduce the encoder of the model. The role of the encoder is to extract and condense information. Images are usually considered to be a three-dimensional matrix, while encoders based on the attention mechanism can only process two-dimensional matrices. So the first thing to do is to perform matrix transformation. The common practice is to divide the image into multiple small blocks, then straighten each small block into a one-dimensional vector, and then combine these one-dimensional vectors into a two-dimensional vector. This two-dimensional vector can be considered as the most original data information. It must first undergo three matrix operations to obtain the three basic matrices Q, K, and V for the attention mechanism operation. The Q, K, and V matrices are the same as the original matrices in terms of matrix specifications, where Q represents the information that should be paid attention to, K represents the information provided, and V represents the information that can be obtained after attention. The formula of the attention mechanism is:

[0046] The attention score is calculated using the Q and K matrices. This score represents how much attention should be given to each element in the sequence when generating the output for the current element. This is achieved by calculating the dot product of the query and key vectors and then applying a softmax function, which converts these scores into a probability distribution. dK is the dimension of the key vector. The output at each position is the weighted sum of the value vectors, with the weights determined by the attention scores calculated in the previous step. This means that each output vector is a weighted combination of all elements in the sequence, with the weights reflecting the relevance of each element to the current element. In this way, the output vector of each small patch is not just its own representation, but a weighted representation of the entire sequence (in this case, the entire image). So after the attention mechanism is applied, each original one-dimensional vector is considered to contain information about the entire image.

[0047] The information extracted by a single encoder layer is often not sufficient, so the encoder layer is usually composed of multiple layers of attention mechanism operations to ensure the accuracy of information extraction. The two-dimensional vector information extracted by the multi-layer encoder is compressed into a one-dimensional vector and contains the information of the entire image. Then the decoder of the model is introduced. The function of the decoder is exactly the opposite of the encoder. Its main job is to generate two-dimensional vectors from one-dimensional vectors, and then restore these two-dimensional vectors to multiple small image blocks and splice them into images. The principle of being able to repair masks and images at the same time is that when it becomes a two-dimensional vector, the scale can be larger than the original input, thereby generating some information to fill in the masked or repaired areas in the image.

[0048] During the training phase, the model is first trained on a large-scale public dataset. The training method is to select a part of the image for masking, and then use the decoder to try to restore the masked part of the image. In this process, the difference between the image restored by the decoder and the original masked part of the image is called the training loss. In order to quantify this loss value more accurately, a more suitable loss function will be used for calculation. Currently, the square error loss function is more commonly used in image restoration.

[0049] The squared error loss function is a commonly used performance function in artificial neural network training. It is used to measure the difference between the predicted output and the actual target value. The squared error loss function formula is shown below.

[0050]

[0051] SELoss in the formula stands for squared error. p represents the total number of training samples, that is, the number of samples or instances in the data set used to train the neural network. k represents the total number of output units, which corresponds to the number of output neurons in the last layer of the neural network, that is, the number of predicted target variables. It represents the kth target value of the pth training pattern. It represents the kth expected output of the pth training pattern. Represents the kth output obtained by the pth pattern.

[0052] By using the gradient descent method, the image generated by the decoder is made to look more and more like the original appearance of the masked part. The gradient descent algorithm selected is the stochastic gradient descent algorithm, which has two cores: learning rate and momentum. Generally, when the model is pre-trained, the learning rate needs to be larger, while the learning rate needs to be smaller during the fine-tuning period.

[0053] During the entire training period, in order to perfectly adapt to different scenes and image restoration tasks of different restoration difficulties, multiple mask ratios must be carefully adjusted and deeply trained. The common incremental mask ratio is 10%. With a mask ratio of 10% as a group, multiple groups of control experiments are conducted to select the best mask ratio for different scenes. The preset mask table is shown in Table 1.

[0054] Table 1: Mask table

[0055] In terms of the selection of mask styles, it is recommended to use random masking. When using random masking, you can shuffle the mask and then fix the front part of the mask. In this way, you can successfully achieve the ideal effect of close time complexity to a constant, thereby effectively avoiding the masking step from taking up too much time resources. At the same time, when using random masking, you need to add position information to the attention model. Adding position information can ensure that the shuffling operation will not have a negative impact on the accuracy of the model.

[0056] After completing the training on a large-scale dataset, the image restoration model is obtained. At this point, multiple image restoration models with different mask ratios will be obtained, so it is necessary to conduct in-depth evaluation on image restoration tasks with different ratios. The evaluation process is also carried out on a public dataset. At this stage, although part of the image is still masked, and then the decoder is allowed to restore this part of the image and calculate the difference loss value, the model will no longer perform gradient descent at this time. The difference value only exists as an important criterion at this stage, and its role is to accurately judge how much mask ratio is more suitable for repairing the image to what specific degree of damage. In the end, the most suitable pre-trained mask ratio for a certain scenario can be determined. For example, when repairing a damaged image with a ratio of 10%, the models with mask ratios of 10%, 20%, and 30% during training can be compared to select the model with the best accuracy. Thanks to the fact that the operation matrices Q, K, and V of the attention mechanism all come from the initial matrix, and the specifications are the same, the different mask ratios for training and evaluation will not cause problems in the training and evaluation of the model. In subsequent repairs, the ratio to be repaired can be evaluated first, and then the model with the best accuracy can be selected.

[0057] During the restoration phase of the model, the images used in the restoration task should be images that the model has never seen during the training phase. In this way, the real scene of restoring images can be simulated with high fidelity.

[0058] When performing image restoration, we first need to evaluate the proportion of the image that is damaged. Then, select the model with the best accuracy of the proportion that was previously evaluated as the base model. Then let the model start to learn the basic information of the image to be restored. This step is achieved through fine-tuning. Fine-tuning is a technology in the field of machine learning. Its basic idea is to further train on the basis of pre-trained models (such as BERT, GPT, etc.) to adapt to specific downstream tasks (such as text classification, sentiment analysis, etc.). In this experiment, fine-tuning is to learn the basic knowledge of the image to be restored.

[0059] When fine-tuning, first use the visible part of the image to be repaired as the original image, then mask the image, and let the decoder predict the masked part. At this time, the predicted difference value is still evaluated using the mean square error loss function, and the stochastic gradient descent is also used to reduce the loss value. After fine-tuning for a period of time, the model basically knows the information contained in the image, and then the image repair task is performed. Then, the repaired image is used as a training set for fine-tuning. In this process, the decoder needs to restore the masked part and evaluate the model's loss value and operations related to gradient descent. (In this way, the model can fully learn the feature information of the image so that it can repair the image better.

[0060] After training is completed, no masking operations will be performed on the image, and the decoder needs to restore the part of the image that needs to be repaired. Based on the basic repair knowledge learned in pre-training and the feature information of the repaired image learned during fine-tuning, the repair effect can be better than the traditional repair effect. When repairing an image, the pre-trained weight needs to be saved as a basic weight, and then loaded, retrained, and re-evaluated each time the user calls it. This saves space by not having to save multiple weights, and also ensures that the model can repair based on the feature information of the repaired image.

[0061] like Figure 2 As shown, an embodiment of the present invention further provides an image restoration device based on a latent mask, comprising a pre-training module, an optimal matching module, a basic model confirmation module, a fine-tuning module and an execution restoration module; The pre-training module is used to mask the images in the large data set and group them according to different mask ratios, and then pre-train the encoder-decoder architecture model to obtain image restoration models with different mask ratios; The optimal matching module is used to evaluate the image restoration models with different mask ratios to obtain the optimal image restoration model for each group of mask ratio images; A basic model confirmation module is used to evaluate the damage ratio of the image to be repaired, and select the optimal image repair model corresponding to the mask ratio image as the basic model of the image to be repaired based on the damage ratio; The fine-tuning module is used to fine-tune the basic model after masking the visible part of the image to be repaired; The execution restoration module is used to input the image to be restored into the fine-tuned model for image restoration.

[0062] In some embodiments, the pre-training module includes an encoding unit, a mask processing unit, a decoding unit, a loss optimization unit, and an image restoration model output unit; The encoding unit is used to encode images in a large data set based on an attention mechanism encoder; A mask processing unit, used to select some images in the large data set for mask processing and group them according to different mask ratios; A decoding unit, configured to restore the masked portion of the image of each set of mask ratios using a decoder; The loss optimization unit is used to obtain the difference between the image restored by the decoder and the image of the original masked part, expressed using the square error loss function; and optimize the loss function using the gradient descent method; The image restoration model output unit is used to determine that when the difference between the image restored by the decoder and the image of the original masked part is less than a set threshold, the training is completed and an image restoration model with different mask ratios is obtained; wherein the image restoration model includes basic restoration knowledge.

[0063] In some embodiments, the encoding unit is specifically used to divide the image into multiple small blocks and straighten each small block into a one-dimensional vector; combine the one-dimensional vectors to obtain a two-dimensional vector; perform three matrix operations on the two-dimensional vector to obtain the basic matrices Q, K, and V of the three attention mechanism operations; so that each original one-dimensional vector contains the information of the entire image. The decoding unit is specifically used to reversely generate a two-dimensional vector through a one-dimensional vector, and restore the two-dimensional vector back to multiple small image blocks and then splice them into an image.

[0064] In some embodiments, the optimal matching module is specifically used to perform mask processing on a portion of the image with a set mask ratio, input a decoder of the image restoration model of images with different mask ratios, restore the masked image and calculate the difference loss value; and select the image restoration model with the smallest difference value as the optimal image restoration model for the mask ratio image.

[0065] In some embodiments, the fine-tuning module is specifically used to take the visible part of the image to be repaired as the original image, mask the original image, and let the decoder of the basic model predict the masked part and calculate the difference value, use the mean square error loss function for evaluation, and use the gradient descent algorithm to optimize the loss function until the model contains the characteristic information of the picture to obtain a fine-tuned model, and the fine-tuned model contains the characteristic information of the repaired picture.

[0066] In some embodiments, a restoration module is executed, specifically for receiving an image to be restored, restoring the portion of the image that needs to be restored based on basic restoration knowledge and a restoration picture feature information decoder, and outputting the restored image.

[0067] The embodiment of the present invention also provides an electronic device, which includes: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The communication bus can be used for information transmission between the electronic device and the sensor. The processor can call the logic instructions in the memory to execute the following method: Step 1: After masking the images in the large data set and grouping them according to different mask ratios, the encoder-decoder architecture model is pre-trained to obtain image restoration models with different mask ratios; Step 2: Evaluate the image restoration models with different mask ratios to obtain the optimal image restoration model for each group of mask ratio images; Step 3: Evaluate the damaged ratio of the image to be restored, and select the optimal image restoration model of the corresponding mask ratio image based on the damaged ratio as the basic model of the image to be restored; Step 4: After masking the visible part of the image to be restored, fine-tune the basic model; Step 5: Input the image to be restored into the fine-tuned model for image restoration.

[0068] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0069] An embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute the method provided by the above method embodiment, for example, including: Step 1: After masking the images in a large data set and grouping them according to different mask ratios, pre-training the encoder-decoder architecture model to obtain image restoration models with different mask ratios; Step 2: Evaluating the image restoration models with different mask ratios to obtain the optimal image restoration model for each group of mask ratio images; Step 3: Evaluating the damaged ratio of the image to be restored, and selecting the optimal image restoration model of the corresponding mask ratio image based on the damaged ratio as the basic model of the image to be restored; Step 4: After masking the visible part of the image to be restored, fine-tuning the basic model; Step 5: Inputting the image to be restored into the fine-tuned model for image restoration.

[0070] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0071] An embodiment of an image restoration device based on a latent mask provided in an embodiment of the present invention belongs to the same inventive concept as the image restoration method based on a latent mask in the above-mentioned embodiments. For details not described in detail in the embodiment of the image restoration device based on a latent mask, reference can be made to the embodiment of the image restoration method based on a latent mask in the above-mentioned embodiments.

[0072] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A latent mask-based image restoration method, characterized in that: include: After masking the images in the large data set and grouping them according to different mask ratios, the encoder-decoder architecture model is pre-trained to obtain image restoration models with different mask ratios. The image inpainting models with different mask ratios are evaluated to obtain the optimal image inpainting model for each group of mask ratio images; Evaluate the damage ratio of the image to be repaired, and select the optimal image repair model corresponding to the mask ratio image as the basic model of the image to be repaired based on the damage ratio; After masking the visible part of the image to be repaired, the base model is fine-tuned; The image to be repaired is input into the fine-tuned model for image repair.

2. The image restoration method based on latent mask according to claim 1, characterized in that: After masking the images in the large data set and grouping them according to different mask ratios, the encoder-decoder architecture model is pre-trained to obtain the image restoration model with different mask ratios. The steps include: The encoder based on the attention mechanism encodes the images in the large data set; Select some images from the large data set for mask processing and group them according to different mask ratios; Use the decoder to restore the masked portion of the image for each set of mask ratios; Obtain the difference between the image restored by the decoder and the image of the original masked part, and express it using the square error loss function; and optimize the loss function through the gradient descent method; When the difference between the image restored by the decoder and the image of the original masked part is less than a set threshold, the training is completed and an image restoration model with different mask ratios is obtained; wherein the image restoration model includes basic restoration knowledge.

3. The image restoration method based on latent mask according to claim 2, characterized in that: The steps of encoding images in a large data set based on the attention mechanism encoder include: Divide the image into multiple small blocks and straighten each small block into a one-dimensional vector; Combining the one-dimensional vectors to obtain a two-dimensional vector; Perform three matrix operations on the two-dimensional vector to obtain the basic matrices Q, K, and V for the three attention mechanism operations; so that each original one-dimensional vector contains the information of the entire image.

4. The image restoration method based on latent mask according to claim 3, characterized in that: The steps of restoring the masked portion of the image of each set of mask ratios using the decoder include: The decoder generates a two-dimensional vector by reversely converting the one-dimensional vector, and restores the two-dimensional vector back to multiple small image blocks and then splices them into an image.

5. The image restoration method based on latent mask according to claim 4, characterized in that: The steps of evaluating the image inpainting models with different mask ratios to obtain the optimal image inpainting model for each set of mask ratio images include: Perform mask processing of a set mask ratio on part of the image, input different mask ratio images into the decoder of the image inpainting model to restore the masked image and calculate the difference loss value; The image restoration model with the smallest difference value is selected as the optimal image restoration model for the mask ratio image.

6. The image restoration method based on latent mask according to claim 5, characterized in that: After masking the visible part of the image to be inpainted, the steps to fine-tune the base model include: The visible part of the image to be repaired is taken as the original image, the original image is masked, and the decoder of the basic model predicts the masked part and calculates the difference value, which is evaluated using the mean square error loss function, and the gradient descent algorithm is used to optimize the loss function until the model contains the feature information of the picture to obtain a fine-tuned model, and the fine-tuned model contains the feature information of the repaired picture.

7. The image restoration method based on latent mask according to claim 6, characterized in that: The steps of inputting the image to be repaired into the fine-tuned model for image repair include: The image to be repaired is received, and based on basic repair knowledge and repair picture feature information decoder, the part of the image that needs to be repaired is restored and the repaired image is output.

8. An image restoration device based on latent mask, characterized in that: It includes pre-training module, optimal matching module, basic model confirmation module, fine-tuning module and execution repair module; The pre-training module is used to mask the images in the large data set and group them according to different mask ratios, and then pre-train the encoder-decoder architecture model to obtain image restoration models with different mask ratios; The optimal matching module is used to evaluate the image restoration models with different mask ratios to obtain the optimal image restoration model for each group of mask ratio images; A basic model confirmation module is used to evaluate the damage ratio of the image to be repaired, and select the optimal image repair model corresponding to the mask ratio image as the basic model of the image to be repaired based on the damage ratio; The fine-tuning module is used to fine-tune the basic model after masking the visible part of the image to be repaired; The execution restoration module is used to input the image to be restored into the fine-tuned model for image restoration.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the latent mask-based image restoration method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the latent mask-based image restoration method according to any one of claims 1 to 7.

Images