Image inpainting method and device, electronic equipment and readable storage medium

Abstract

This application provides an image restoration method, apparatus, electronic device, and readable storage medium, relating to the field of computer vision technology. The method includes: acquiring an image to be processed; preprocessing the image to be processed; inputting the image to be processed and the preprocessed image to be processed into an image restoration model to obtain a restored image with target features removed; wherein the image restoration model includes an image correction module and an image restoration module, the image correction module being used to extract the target features from the image to be processed and the preprocessed image to be processed, and outputting a feature image; the image restoration module being used to decode and restore the feature image using a pre-acquired target matrix as a constraint, and outputting the restored image; the target matrix is ​​determined based on the one-hot encoding corresponding to the target features. Embodiments of this application can remove various scene content and background information existing in an image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image restoration method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] In reality, paper documents face significant risks of loss and damage. Uploading paper documents to data support systems is often used to maintain a record of these documents, with common methods including scanning and direct photographing. However, paper documents are extremely prone to damage during storage, such as creases, water stains, and blemishes caused by human error. Even without human error during storage, exposure to air can lead to yellowing and curling over time. Such damaged paper documents often result in unclear text and incomplete information when uploaded to the system. Current technologies use image restoration algorithms to repair damaged images, but traditional algorithms can only address one type of damage, and many invalid information features in images, such as scene content and background information, are difficult to remove. Summary of the Invention

[0003] This application provides an image restoration method, apparatus, electronic device, and readable storage medium to address the problem that traditional image restoration algorithms can only address one type of loss, while a large number of invalid information features in the image are difficult to remove.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide an image restoration method, including:

[0006] Obtain the image to be processed;

[0007] The image to be processed is preprocessed;

[0008] The image to be processed and the preprocessed image to be processed are input into the image restoration model to obtain the restored image with the target features removed;

[0009] The image restoration model includes an image correction module and an image restoration module. The image correction module is used to extract the target features from the image to be processed and the preprocessed image to be processed, and output a feature image. The image restoration module is used to decode and restore the feature image with a pre-acquired target matrix as a constraint, and output the restored image. The target matrix is ​​determined based on the one-hot encoding corresponding to the target features.

[0010] Optionally, the preprocessing of the image to be processed includes:

[0011] Obtain the position coordinates of the range where the image information is located within the sample image to be processed;

[0012] The image is labeled based on the location coordinates to obtain the labeled sample image to be processed.

[0013] The labeled sample image to be processed is then subjected to image binarization.

[0014] Optionally, the method further includes:

[0015] Identify the target features in the image to be processed;

[0016] Determine the one-hot encoding corresponding to the target feature;

[0017] Obtain the first size information of the feature image;

[0018] Based on the one-hot encoding corresponding to the target feature and the first size information, the target matrix corresponding to the target feature is obtained.

[0019] Optionally, the training process of the image restoration model includes:

[0020] Obtain the image restoration model to be trained;

[0021] Obtain a sample training set, wherein the sample training set includes a first image and a second image corresponding to the first image, the first image is an image with sample features, and the second image is an image with the sample features pre-removed;

[0022] The first image is input into the image correction module to obtain a first feature map, the second image is input into the image correction module to obtain a second feature map, and a first loss function is obtained based on the first feature map and the second feature map;

[0023] The first feature map and the pre-acquired sample matrix are input into the image restoration module to obtain the restored third image corresponding to the first image;

[0024] The parameters of the image restoration model are adjusted based on the first loss function and the third image to obtain the trained image restoration model.

[0025] Optionally, the sample training set includes sample images to be processed that have the sample features, and obtaining the first image from the sample images to be processed includes:

[0026] The preprocessing is performed on the sample image to be processed to obtain the fourth image;

[0027] The sample image to be processed is segmented to obtain sample image information in the sample image to be processed, and the fourth image is segmented to remove image information in the fourth image;

[0028] The sample image information and the fourth image from which the sample image information has been removed are fused together to obtain the first image.

[0029] Optionally, obtaining the first loss function based on the first feature map and the second feature map includes:

[0030] Obtain the first feature matrix corresponding to the first feature map, and obtain the second feature matrix corresponding to the second feature map;

[0031] The first loss function is obtained based on the first feature matrix and the second feature matrix.

[0032] Optionally, the method further includes:

[0033] Identify the sample features in the image to be processed;

[0034] Determine the one-hot encoding corresponding to the sample features;

[0035] Obtain the second size information of the first feature map;

[0036] Based on the one-hot encoding corresponding to the sample features and the second size information, the sample matrix corresponding to the sample features is obtained.

[0037] Optionally, the image restoration model further includes a discriminator, and adjusting the parameters of the image restoration model based on the first loss function and the third image includes:

[0038] The third image is input into the discriminator to obtain the discrimination result output by the discriminator. The discrimination result includes a first result and a second result. The first result is that the discriminator identifies the third image as a real image, and the second result is that the discriminator identifies the third image as a fake image.

[0039] If the discrimination result of the third image is the second result, a second loss function is obtained based on the third image and the first loss function, and the parameters of the image restoration model are adjusted based on the second loss function.

[0040] Optionally, the step of obtaining the second loss function based on the third image and the first loss function, and adjusting the parameters of the image inpainting model based on the second loss function, includes:

[0041] Based on the third image and the second image, a third loss function is obtained;

[0042] The model parameters of the image restoration module are adjusted based on the third loss function and the first loss function to obtain the trained image restoration module;

[0043] The model parameters of the image correction module are adjusted based on the first loss function to obtain the trained image correction module;

[0044] Based on the third image, the first loss function, and the third loss function, the second loss function is obtained;

[0045] The model parameters of the discriminator are adjusted based on the second loss function.

[0046] Secondly, embodiments of this application also provide an image restoration apparatus, the apparatus comprising:

[0047] The acquisition module is used to acquire the image to be processed;

[0048] The preprocessing module is used to preprocess the image to be processed;

[0049] The model repair module is used to input the image to be processed and the preprocessed image to be processed into the image repair model to obtain a repaired image with the target features removed.

[0050] The image restoration model includes an image correction module and an image restoration module. The image correction module is used to extract the target features from the image to be processed and the preprocessed image to be processed, and output a feature image. The image restoration module is used to decode and restore the feature image with a pre-acquired target matrix as a constraint, and output the restored image. The target matrix is ​​determined based on the one-hot encoding corresponding to the target features.

[0051] Thirdly, embodiments of this application also provide an electronic device, including a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the image restoration method as described in any one of the first aspects.

[0052] Fourthly, embodiments of this application also provide a readable storage medium storing a program that, when executed by a processor, implements the steps of the image restoration method as described in any one of the first aspects.

[0053] In this embodiment, the image to be processed is preprocessed and annotated. The preprocessed image and the image to be processed are then input into an image inpainting model. The image inpainting model outputs an inpainted image that has had multiple target features removed. Specifically, the image inpainting model extracts target features using an image correction module and outputs a feature image. This feature image is then input into the image inpainting module, using a target matrix representing multiple target features as conditional parameters. The feature image is then inpainted based on these target features, achieving the effect of repairing various losses using the image inpainting model. This effectively removes invalid information features such as scene content and background information from the image to be processed. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic flowchart of an image restoration method provided in an embodiment of this application;

[0056] Figure 2 yes Figure 1 One of the flowcharts illustrating the training process of an image restoration model;

[0057] Figure 3 yes Figure 2 The second flowchart illustrating the training process of the image restoration model in China;

[0058] Figure 4 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0062] See Figure 1 , Figure 1 This is a flowchart illustrating an image restoration method provided in an embodiment of this application. The image restoration method includes the following steps:

[0063] Step 101: Obtain the image to be processed.

[0064] It is understood that the aforementioned images to be processed can be images of paper documents with damage, stains, creases, etc.; images with missing information; images with messy edges; images with a large amount of invalid scene or background information; or images with several of the aforementioned damage conditions simultaneously. In this application embodiment, the images to be processed can be acquired using a scanner, camera, video camera, or other image-capturing devices, converting paper documents into electronic images. Using a scanner to acquire the images can avoid problems such as curled edges that may occur in paper documents and is unaffected by scene brightness, resulting in electronic images with better display effects. Furthermore, images acquired using cameras or other devices can also be used for image restoration. This application does not impose excessive restrictions on the acquisition method of the images to be processed, allowing acquisition based on the actual condition of the original paper document, thus expanding the application scope and enhancing practicality.

[0065] Step 102: Preprocess the image to be processed.

[0066] In specific embodiments, preprocessing the image to be processed can involve content annotation, which facilitates the image restoration model to quickly and accurately identify feature information and determine the scope of the content to be restored when extracting feature information within the image. Preprocessing can automatically identify the area containing valid information within the image to be processed, annotate this area, remove areas outside the annotations, or generate a bounding box along the annotations to define the image information within this range. Furthermore, the image to be processed can be displayed on the front page of the image display system, providing a visual representation to the user. The user can manually annotate the area containing valid information within the image, for example, by providing annotation functionality through an html2canvas plugin. After obtaining the coordinates of the user's annotations, a bounding box can be generated for the image to be processed. It should be noted that while defining valid information in the image during preprocessing can initially remove some invalid background information or scene content, the preprocessed image may still contain invalid information. The bounding box generated after preprocessing may still contain invalid information such as water stains, dirt, damage, and creases.

[0067] Step 103: Input the image to be processed and the preprocessed image to be processed into the image restoration model to obtain the restored image with the target features removed;

[0068] The image restoration model includes an image correction module and an image restoration module. The image correction module is used to extract the target features from the image to be processed and the preprocessed image to be processed, and output a feature image. The image restoration module is used to decode and restore the feature image with a pre-acquired target matrix as a constraint, and output the restored image. The target matrix is ​​determined based on the one-hot encoding corresponding to the target features.

[0069] It should be noted that the aforementioned target features can be various forms of damage in the image to be processed. Specific forms of damage include yellowing, aging, water stains, dirt, creases, and tears in the paper, as well as redundant background information, scene content, and other invalid information in the image. This application addresses various target features appearing on paper documents. After acquiring an electronic image of the paper document, image restoration processing is performed on the electronic image to remove various target features, thus restoring the electronic image. This facilitates the preservation of the original paper document's electronic archive, allowing for direct recall of the restored electronic image during subsequent use, reducing secondary damage to the damaged paper document.

[0070] In one specific embodiment of this application, the aforementioned image inpainting model can be a Conditional Generative Adversarial Network (CGAN) model. The pre-trained image inpainting model performs image inpainting processing on the image to be processed. The image inpainting model in this application can include an image correction module and an image inpainting module. The image correction module can serve as the encoder part of the CGAN, and the image inpainting module can serve as the decoder part of the CGAN. Different constraints are introduced during the training of this image inpainting model to represent different target features; the constraints can be the target matrix corresponding to the target features. In this way, the image inpainting model performs different inpainting processes on the feature image according to different constraints, thereby achieving the purpose of using the aforementioned image inpainting model to repair multiple target features existing in the image to be processed.

[0071] Specifically, after importing the image to be processed and the preprocessed image into the image correction module, the two images can be fused into a single image for feature extraction. The extracted features may include valid information features or target features from the image to be processed. Thus, fusing the image to be processed and the preprocessed image before feature extraction improves the feature extraction performance of the image correction module.

[0072] Furthermore, the constraints introduced in the aforementioned image inpainting model can be a target matrix. The pre-acquisition process of the target matrix can be as follows: first, the target features present in the image to be processed are labeled, and each target feature corresponds to a one-hot encoding. Then, all the determined one-hot encodings are converted into a target matrix. Finally, the target matrix is ​​imported into the image inpainting model as a constraint to inpaint the image to be processed. In this way, by setting different one-hot encodings for different target features and forming a target matrix using these one-hot encodings as constraints for the image inpainting model, multiple target features present in the image to be processed can be inpainted.

[0073] Optionally, the preprocessing of the image to be processed includes:

[0074] Obtain the position coordinates of the area containing the image information within the image to be processed;

[0075] The image is annotated based on the location coordinates to obtain the annotated image to be processed.

[0076] The labeled image to be processed is then subjected to image binarization.

[0077] In practice, the image to be processed undergoes preprocessing. This involves determining the coordinates of the area containing the image information within the image to be processed, and then labeling the image information within these coordinates to obtain the labeled image. Finally, the labeled image is binarized, making it a single-channel mask image. This mask image is then fused with the original image and used as input to the image inpainting model. This facilitates the extraction of effective features by the image correction model, improving the restoration effect.

[0078] Optionally, the method further includes:

[0079] Identify the target features in the image to be processed;

[0080] Determine the one-hot encoding corresponding to the target feature;

[0081] Obtain the first size information of the feature image;

[0082] Based on the one-hot encoding corresponding to the target feature and the first size information, the target matrix corresponding to the target feature is obtained.

[0083] In one embodiment of this application, the pre-acquired target matrix can be determined using one-hot encoding and feature images. First, after determining the target features in the image to be processed, the one-hot encoding corresponding to each target feature is determined. Based on the size information of the feature image combined with the one-hot encoding, the target matrix corresponding to the target features in the image to be processed is determined. For example, after acquiring the image to be processed, the target features in the image are labeled. When inputting the image restoration model, a matrix of size [missing information] can be used. One-hot encoding converted to The target matrix, where, The number of target features. This represents the length of the feature image output by the image correction model. This represents the width of the feature image output by the image correction model. By introducing one-hot encoding corresponding to the target features, the image processing model uses the target matrix determined by this one-hot encoding as a constraint to repair the target features within the image to be processed, thus achieving the effect of repairing multiple types of losses using a single image inpainting network.

[0084] Optionally, the training process of the image restoration model includes:

[0085] Step 201: Obtain the image restoration model to be trained.

[0086] Please see Figure 2The image restoration model to be trained can include two parts: an image correction module and an image restoration module. By training the image restoration model, its corresponding loss function is determined so that the obtained image restoration model can remove cluttered backgrounds and invalid information, and output a restored image that retains only valid information. The image correction model can include a four-layer structure, each layer consisting of a 3*3 convolutional layer, a ReLU activation function, and a Batch Normalization (BN) network layer. The image restoration model can include three residual blocks and four decoding layers, each decoding layer consisting of an upsampling layer, a convolutional layer with a kernel of 1, a ReLU activation function layer, and a BN network layer.

[0087] Step 202: Obtain a sample training set, wherein the sample training set includes a first image and a second image corresponding to the first image, the first image is an image with sample features, and the second image is an image with the sample features removed beforehand.

[0088] It is worth noting that the training set used in the image restoration model training process can consist of two parts: a first image and a second image corresponding to the first image. The first image includes pre-set sample features, which can be consistent with the target features in the above embodiments, such as aging, water stains, yellowing, creases, damage, etc., or may contain invalid information or invalid scene content. The first image can be a pre-processed image, or an image fused from a pre-processed image and an unprocessed image. The second image can be an image with pre-removed sample features. The second image corresponds to the first image and can serve as a comparison, representing the result of the first image after correction and restoration. Alternatively, the second image can be a pre-corrected and restored ground truth image.

[0089] Step 203: Input the first image into the image correction module to obtain a first feature map, input the second image into the image correction module to obtain a second feature map, and obtain a first loss function based on the first feature map and the second feature map.

[0090] Understandably, the first and second images are input into the image correction module, respectively, to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image. Based on the output first and second feature maps, a loss function can be applied to obtain the corresponding loss function for training the image correction module. Specifically, the first loss function can be calculated by determining the difference in intermediate parameters between the first and second feature maps. Utilizing the different processing results of the image correction module for the first and second images can solve the gradient vanishing problem commonly encountered in image restoration, while simultaneously improving the training effect of the image correction module.

[0091] Step 204: Input the first feature map and the pre-acquired sample matrix into the image restoration module to obtain the restored third image corresponding to the first image.

[0092] In this step, the first feature map and a pre-acquired sample matrix are input into the image inpainting module as input parameters. The first feature map is then inpainted under the constraint of the sample matrix. The pre-acquired sample matrix can be obtained in the same way as the target matrix in the previous embodiments, by determining the one-hot encoding of the sample features and generating the sample matrix based on the one-hot encoding. Thus, under the constraint of the sample matrix, the image inpainting module inpaints the first feature map and outputs the inpainted third image.

[0093] Step 205: Adjust the parameters of the image restoration model based on the first loss function and the third image to obtain the trained image restoration model.

[0094] In one specific embodiment, the model parameters of the image restoration model are adjusted based on the aforementioned first loss function and the restored third image to improve the image restoration effect. Specifically, the first loss function can be used to adjust the model parameters of the image correction module. By judging the restoration effect of the obtained restored third image, if the restoration effect is good, it is determined that the parameter settings of the image restoration model are appropriate; if the restoration effect is poor, the parameters of the image restoration model can be adjusted to obtain a better restoration effect. Thus, the first loss function and the restored third image can effectively adjust the parameters of the image restoration model to improve its restoration effect.

[0095] Optionally, the sample training set includes sample images to be processed that have the sample features, and obtaining the first image from the sample images to be processed includes:

[0096] The preprocessing is performed on the sample image to be processed to obtain the fourth image;

[0097] The sample image to be processed is segmented to obtain sample image information in the sample image to be processed, and the fourth image is segmented to remove sample image information in the fourth image;

[0098] The sample image information and the fourth image from which the sample image information has been removed are fused together to obtain the first image.

[0099] It must be noted that during the training of the image restoration model, the sample training set includes the sample images to be processed. These sample images serve as the base images for training the image restoration model, and other images used in the training can be obtained based on these sample images. The sample images to be processed can be directly obtained electronic images of paper documents, or three-channel RGB images. The first image, as image data in the sample training set, can be obtained by processing the sample images to be processed. Specifically, the processing can involve preprocessing the sample images to be processed to obtain a fourth image. This fourth image can be an image with bounding boxes and binarized, and it can be a single-channel image. Subsequently, the content within the bounding boxes of the fourth image, i.e., the sample image information, can be removed. This sample image information can include valid image information and invalid information; the invalid information can be the aforementioned sample features. Simultaneously, the sample image information within the sample images to be processed corresponding to the fourth image can be segmented, retaining the sample image information within the sample images to be processed and the other parts of the fourth image excluding the sample image information. These are then fused together to finally obtain the first image. In this way, the image correction module can more accurately determine the feature extraction region from the first input image.

[0100] Optionally, obtaining the first loss function based on the first feature map and the second feature map includes:

[0101] Obtain the first feature matrix corresponding to the first feature map, and obtain the second feature matrix corresponding to the second feature map;

[0102] The first loss function is obtained based on the first feature matrix and the second feature matrix.

[0103] In another embodiment of this application, the first loss function is obtained through the first feature map and the second feature map, specifically:

[0104]

[0105] in, The first loss function;

[0106] The feature matrix represents the first feature map;

[0107] The feature matrix represents the second feature map.

[0108] Specifically, by using the matrix parameters in the first feature map and the matrix parameters in the second feature map, the output of the image correction module is used as a loss function to train the image correction module. The parameters of the first feature map and the second feature map are shared to determine the intermediate loss result between them. In this way, the gradient vanishing problem that often exists in the image restoration process can be effectively solved, and the training effect of the image correction module can be improved.

[0109] Optionally, the method further includes:

[0110] Identify the sample features in the image to be processed;

[0111] Determine the one-hot encoding corresponding to the sample features;

[0112] Obtain the second size information of the first feature map;

[0113] Based on the one-hot encoding corresponding to the sample features and the second size information, the sample matrix corresponding to the sample features is obtained.

[0114] In practical implementation, the sample matrix can serve as a constraint on the image inpainting module to be trained, constraining the module's inpainting of the first image. The process of pre-obtaining the sample matrix can refer to the aforementioned process of obtaining the target matrix. Specifically, it can be determined based on the sample features in the image to be processed. First, a one-hot encoding can be set for each sample feature. Combining the one-hot encoding and the size information of the first feature map, the sample matrix corresponding to the sample feature is determined. The specific details can be found in the aforementioned process of obtaining the target matrix.

[0115] Optionally, the image restoration model further includes a discriminator, and adjusting the parameters of the image restoration model based on the first loss function and the third image includes:

[0116] The third image is input into the discriminator to obtain the discrimination result output by the discriminator. The discrimination result includes a first result and a second result. The first result is that the discriminator identifies the third image as a real image, and the second result is that the discriminator identifies the third image as a fake image.

[0117] If the discrimination result of the third image is the second result, a second loss function is obtained based on the third image and the first loss function, and the parameters of the image restoration model are adjusted based on the second loss function.

[0118] Both the generator and discriminator are given additional information as conditions. This additional information is fed into the discriminator and generator models as part of the input layer, thus forming a conditional adversarial network.

[0119] In yet another embodiment of this application, please refer to Figure 3 Image inpainting models can also include a discriminator. The discriminator can determine the restoration effect of the restored third image within the model. The discriminator's judgment process can involve comparing the third image with the second image. If the discriminator outputs a first result—that is, the third image is identical or similar to the second image, or if the image features within the third image are essentially the same as those within the second image—then the third image is judged as a genuine image. Conversely, if the discriminator outputs a second result—that is, the third image is judged as a fake image—this could be because the third image still contains sample features, or because the third image is more blurred than the second image.

[0120] Furthermore, when the third image is identified as a genuine image, the image inpainting model has achieved good restoration results, and no adjustment to the image inpainting model is necessary. However, when the third image is identified as a fake image, the image inpainting model's restoration effect is poor. In this case, a second loss function related to the model parameters of the image inpainting module can be determined using the third image and the first loss function. The image inpainting module can then be adjusted using the second loss function to improve its restoration effect. In this way, the discriminator can judge whether the restored third image is realistic and clear, and then provide feedback to the image inpainting model for parameter adjustment, enabling the model to generate a clearer restored image that closely resembles the second image.

[0121] Optionally, the step of obtaining the second loss function based on the third image and the first loss function, and adjusting the parameters of the image inpainting model based on the second loss function, includes:

[0122] Based on the third image and the second image, a third loss function is obtained;

[0123] The model parameters of the image restoration module are adjusted based on the third loss function and the first loss function to obtain the trained image restoration module;

[0124] The model parameters of the image correction module are adjusted based on the first loss function to obtain the trained image correction module;

[0125] Based on the third image, the first loss function, and the third loss function, the second loss function is obtained;

[0126] The model parameters of the discriminator are adjusted based on the second loss function.

[0127] In practical implementation, the third loss function is determined based on the third image and the second image. This can be achieved by processing the feature matrices in the third image and the second image. Specifically, the third loss function can be:

[0128]

[0129] in, This is the loss of the image inpainting module, also known as the third loss function;

[0130] This is the repaired third image, which is the result output by the image restoration module;

[0131] This is the second image used for training.

[0132] Specifically, the second loss function is determined based on the third image, the first loss function, and the third loss function. It can be used to adjust the process of insulating the acquired first feature map during the training of the entire image inpainting model; specifically, it can be used to adjust the training of the image inpainting module. Therefore, the second loss function can be:

[0133]

[0134] The first loss function;

[0135] and This is a hyperparameter used to adjust the weight ratio between the image correction module and the image inpainting module in the third loss function.

[0136] Furthermore, training the image restoration model can be a process of continuously adjusting the model's parameters. In this application, the image correction module can be trained using a first loss function, and the image restoration module can be adjusted using a second loss function determined by the first and third loss functions. Additionally, the discriminator in the entire image restoration model can be trained using a loss function determined by the second loss function and cross-entropy loss, thereby improving the image restoration effect. The loss function used to train the discriminator in this application is:

[0137]

[0138] in, Cross-entropy loss;

[0139] This is the second loss function;

[0140] is a hyperparameter that can represent the weights of the cross-entropy loss and the second loss function.

[0141] See Figure 4 , Figure 4 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application, as shown below. Figure 4 As shown, the image restoration device 300 includes:

[0142] The acquisition module 301 is used to acquire the image to be processed;

[0143] Preprocessing module 302 is used to preprocess the image to be processed;

[0144] The model repair module 303 is used to input the image to be processed and the preprocessed image to be processed into the image repair model to obtain a repaired image with the target features removed.

[0145] The image restoration model includes an image correction module and an image restoration module. The image correction module is used to extract the target features from the image to be processed and the preprocessed image to be processed, and output a feature image. The image restoration module is used to decode and restore the feature image with a pre-acquired target matrix as a constraint, and output the restored image. The target matrix is ​​determined based on the one-hot encoding corresponding to the target features.

[0146] Optionally, the preprocessing module 302 is used for:

[0147] Obtain the position coordinates of the area containing the image information within the image to be processed;

[0148] The image is annotated based on the location coordinates to obtain the annotated image to be processed.

[0149] The labeled image to be processed is then subjected to image binarization.

[0150] Optionally, the device is further used for:

[0151] Identify the target features in the image to be processed;

[0152] Determine the one-hot encoding corresponding to the target feature;

[0153] Obtain the first size information of the feature image;

[0154] Based on the one-hot encoding corresponding to the target feature and the first size information, the target matrix corresponding to the target feature is obtained.

[0155] Optionally, the training device for the image restoration model is used for:

[0156] Obtain the image restoration model to be trained;

[0157] Obtain a sample training set, wherein the sample training set includes a first image and a second image corresponding to the first image, the first image is an image with sample features, and the second image is an image with the sample features pre-removed;

[0158] The first image is input into the image correction module to obtain a first feature map, the second image is input into the image correction module to obtain a second feature map, and a first loss function is obtained based on the first feature map and the second feature map;

[0159] The first feature map and the pre-acquired sample matrix are input into the image restoration module to obtain the restored third image corresponding to the first image;

[0160] The parameters of the image restoration model are adjusted based on the first loss function and the third image to obtain the trained image restoration model.

[0161] Optionally, the training device for the image restoration model is used for:

[0162] The preprocessing is performed on the sample image to be processed to obtain the fourth image;

[0163] The sample image to be processed is segmented to obtain sample image information in the sample image to be processed, and the fourth image is segmented to remove image information in the fourth image;

[0164] The sample image information and the fourth image from which the sample image information has been removed are fused together to obtain the first image.

[0165] Optionally, the training device for the image restoration model is used for:

[0166] Obtain the first feature matrix corresponding to the first feature map, and obtain the second feature matrix corresponding to the second feature map;

[0167] The first loss function is obtained based on the first feature matrix and the second feature matrix.

[0168] Optionally, the device is further used for:

[0169] Identify the sample features in the image to be processed;

[0170] Determine the one-hot encoding corresponding to the sample features;

[0171] Obtain the second size information of the first feature map;

[0172] Based on the one-hot encoding corresponding to the sample features and the second size information, the sample matrix corresponding to the sample features is obtained.

[0173] Optionally, the image restoration model further includes a discriminator, and the training device for the image restoration model is used for:

[0174] The third image is input into the discriminator to obtain the discrimination result output by the discriminator. The discrimination result includes a first result and a second result. The first result is that the discriminator identifies the third image as a real image, and the second result is that the discriminator identifies the third image as a fake image.

[0175] If the discrimination result of the third image is the second result, a second loss function is obtained based on the third image and the first loss function, and the parameters of the image restoration model are adjusted based on the second loss function.

[0176] Optionally, the training device for the image restoration model is used for:

[0177] Based on the third image and the second image, a third loss function is obtained;

[0178] The model parameters of the image restoration module are adjusted based on the third loss function and the first loss function to obtain the trained image restoration module;

[0179] The model parameters of the image correction module are adjusted based on the first loss function to obtain the trained image correction module;

[0180] Based on the third image, the first loss function, and the third loss function, the second loss function is obtained;

[0181] The model parameters of the discriminator are adjusted based on the second loss function.

[0182] The image restoration device 300 can implement each process in the above-described image restoration method embodiments and achieve the corresponding beneficial effects. To avoid repetition, it will not be described again here.

[0183] like Figure 5 As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a program or instructions stored in the memory 402 and executable on the processor 401. When the program or instructions are executed by the processor 401, they implement the various processes of the aforementioned image restoration method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0184] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0185] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the aforementioned image restoration method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0186] The processor is the processor in the electronic device described in the above embodiments. A readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image restoration method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is preprocessed; The image to be processed and the preprocessed image to be processed are input into the image restoration model to obtain the restored image with the target features removed; The image restoration model includes an image correction module and an image restoration module. The image correction module is used to extract the target features from the image to be processed and the preprocessed image to be processed, and output a feature image. The image restoration module is used to decode and restore the feature image with a pre-acquired target matrix as a constraint, and output the restored image. The target matrix is ​​determined based on the one-hot encoding corresponding to the target features. The training process of the image restoration model includes: Obtain the image restoration model to be trained; Obtain a sample training set, wherein the sample training set includes a first image and a second image corresponding to the first image, the first image is an image with sample features, and the second image is an image with the sample features pre-removed; The first image is input into the image correction module to obtain a first feature map, the second image is input into the image correction module to obtain a second feature map, and a first loss function is obtained based on the first feature map and the second feature map; The first feature map and the pre-acquired sample matrix are input into the image restoration module to obtain the restored third image corresponding to the first image; The parameters of the image restoration model are adjusted based on the first loss function and the third image to obtain the trained image restoration model; The image restoration model further includes a discriminator, and adjusting the parameters of the image restoration model based on the first loss function and the third image includes: The third image is input into the discriminator to obtain the discrimination result output by the discriminator. The discrimination result includes a first result and a second result. The first result is that the discriminator identifies the third image as a real image, and the second result is that the discriminator identifies the third image as a fake image. If the discrimination result of the third image is the second result, a second loss function is obtained based on the third image and the first loss function, and the parameters of the image restoration model are adjusted based on the second loss function.

2. The method according to claim 1, characterized in that, The preprocessing of the image to be processed includes: Obtain the position coordinates of the range of image information within the image to be processed; The image frame of the image to be processed is annotated based on the position coordinates to obtain the annotated image to be processed. The labeled image to be processed is then subjected to image binarization.

3. The method according to claim 1, characterized in that, The method further includes: Identify the target features in the image to be processed; Determine the one-hot encoding corresponding to the target feature; Obtain the first size information of the feature image; Based on the one-hot encoding corresponding to the target feature and the first size information, the target matrix corresponding to the target feature is obtained.

4. The method according to claim 1, characterized in that, The sample training set includes sample images to be processed that have the sample features, and obtaining the first image from the sample images to be processed includes: The preprocessing is performed on the sample image to be processed to obtain the fourth image; The sample image to be processed is segmented to obtain sample image information in the sample image to be processed, and the fourth image is segmented to remove image information in the fourth image; The sample image information and the fourth image from which the sample image information has been removed are fused together to obtain the first image.

5. The method according to claim 4, characterized in that, The step of obtaining the first loss function based on the first feature map and the second feature map includes: Obtain the first feature matrix corresponding to the first feature map, and obtain the second feature matrix corresponding to the second feature map; The first loss function is obtained based on the first feature matrix and the second feature matrix.

6. The method according to claim 4, characterized in that, The method further includes: Identify sample features in the image to be processed; Determine the one-hot encoding corresponding to the sample features; Obtain the second size information of the first feature map; Based on the one-hot encoding corresponding to the sample features and the second size information, the sample matrix corresponding to the sample features is obtained.

7. The method according to claim 1, characterized in that, The step of obtaining the second loss function based on the third image and the first loss function, and adjusting the parameters of the image restoration model based on the second loss function, includes: Based on the third image and the second image, a third loss function is obtained; The model parameters of the image restoration module are adjusted based on the third loss function and the first loss function to obtain the trained image restoration module; The model parameters of the image correction module are adjusted based on the first loss function to obtain the trained image correction module; Based on the third image, the first loss function, and the third loss function, the second loss function is obtained; The model parameters of the discriminator are adjusted based on the second loss function.

8. An image restoration device, characterized in that, include: The acquisition module is used to acquire the image to be processed; The preprocessing module is used to preprocess the image to be processed; The model repair module is used to input the image to be processed and the preprocessed image to be processed into the image repair model to obtain a repaired image with the target features removed. The image restoration model includes an image correction module and an image restoration module. The image correction module is used to extract the target features from the image to be processed and the preprocessed image to be processed, and output a feature image. The image restoration module is used to decode and restore the feature image with a pre-acquired target matrix as a constraint, and output the restored image. The target matrix is ​​determined based on the one-hot encoding corresponding to the target features. The training device for the image restoration model is used for: Obtain the image restoration model to be trained; Obtain a sample training set, wherein the sample training set includes a first image and a second image corresponding to the first image, the first image is an image with sample features, and the second image is an image with the sample features pre-removed; The first image is input into the image correction module to obtain a first feature map, the second image is input into the image correction module to obtain a second feature map, and a first loss function is obtained based on the first feature map and the second feature map; The first feature map and the pre-acquired sample matrix are input into the image restoration module to obtain the restored third image corresponding to the first image; The parameters of the image restoration model are adjusted based on the first loss function and the third image to obtain the trained image restoration model; The image restoration model also includes a discriminator, and the training device for the image restoration model is used for: The third image is input into the discriminator to obtain the discrimination result output by the discriminator. The discrimination result includes a first result and a second result. The first result is that the discriminator identifies the third image as a real image, and the second result is that the discriminator identifies the third image as a fake image. If the discrimination result of the third image is the second result, a second loss function is obtained based on the third image and the first loss function, and the parameters of the image restoration model are adjusted based on the second loss function.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the image restoration method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the steps of the image restoration method as described in any one of claims 1 to 7.

Images