Image restoration model training method and device, computer equipment and storage medium

Abstract

The invention relates to an image restoration model training method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a sample image, and repairing the sample image through an image repairing model to obtain a repaired image; determining restoration loss based on the real image and the restoration image corresponding to the sample image; outputting a quality parameter for the repaired image through the target sorting network, and determining sorting loss according to the quality parameter; the target sorting network is obtained by training based on a plurality of reference images, the plurality of reference images are obtained by repairing the same original image according to different repairing modes, and quality parameters corresponding to the plurality of reference images are matched with a good-bad relationship between repairing effects corresponding to the plurality of reference images; and performing parameter adjustment on the image restoration model based on the restoration loss and the sorting loss to obtain a trained image restoration model. By adopting the method, the training effect of the image restoration model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image inpainting, and in particular to an image inpainting model training method and device, a computer device and a storage medium. BACKGROUND

[0002] With the development of image inpainting technology, various image inpainting technologies based on generative adversarial networks have appeared, and more objective indexes consistent with human perception have been proposed for the inpainting effect of images, such as NIQE (Natural Image Quality Evaluator), PI (Perceptual Index), etc. These objective indexes are highly correlated with human subjective scores.

[0003] In related technologies, an image inpainting model is usually constructed based on a single inpainting method and is optimized and trained through adversarial loss of a generative adversarial network. The model trained in this way can synthesize realistic details, but the objective index is poor, i.e., the image perceptual quality is low. As an un-differentiable artificial statistic, the objective index cannot be directly involved in model training, so that the optimization goal of the image inpainting model and the evaluation index are inconsistent, resulting in poor inpainting effect of the obtained image inpainting model. SUMMARY

[0004] Therefore, it is necessary to provide an image inpainting model training method and device that can fuse multiple inpainting methods and use objective indexes for model optimization, a computer device, a computer readable storage medium and a computer program product.

[0005] In a first aspect, the present application provides an image inpainting model training method, comprising:

[0006] obtaining a sample image, performing image inpainting on the sample image through an image inpainting model to obtain an inpainted image;

[0007] determining an inpainting loss based on a real image corresponding to the sample image and the inpainted image;

[0008] performing inpainting quality prediction on the inpainted image through a target ranking network to obtain a quality parameter, and determining a ranking loss according to the quality parameter; wherein the target ranking network is trained based on multiple reference images, the multiple reference images are obtained by inpainting the same original image in different inpainting manners, and the quality parameters corresponding to the multiple reference images respectively match the superior-inferior relationship between the inpainting effects corresponding to the multiple reference images respectively;

[0009] Based on the repair loss and the ranking loss, the image repair model is adjusted in parameters to obtain a trained image repair model.

[0010] In one of the embodiments, the method further comprises:

[0011] Obtaining an original image;

[0012] Repairing the original image based on a plurality of different repair methods to obtain a plurality of reference images corresponding to the original image;

[0013] Pairing the plurality of reference images two by two to obtain at least one image pair;

[0014] For each image pair, determining a superior-inferior relationship label of the image pair based on the repair effects of the two reference images corresponding to the image pair respectively;

[0015] Training a ranking network based on the image pair and the superior-inferior relationship label of the image pair to obtain a target ranking network.

[0016] In one of the embodiments, the training of the ranking network based on the image pair and the superior-inferior relationship label of the image pair to obtain a target ranking network comprises:

[0017] Outputting, by the ranking network, respective quality parameters of the two reference images included in the image pair respectively;

[0018] Determining a marginal ranking loss based on the respective quality parameters of the two reference images and the superior-inferior relationship label of the image pair;

[0019] Adjusting the ranking network in parameters based on the marginal ranking loss to obtain a target ranking network.

[0020] In one of the embodiments, the repair quality prediction of the repair image by the target ranking network to obtain a quality parameter and the determination of a ranking loss based on the quality parameter comprises:

[0021] Obtaining a target ranking network;

[0022] Performing, by the target ranking network, repair quality prediction on the repair image to obtain a quality parameter;

[0023] Converting based on the quality parameter to obtain a ranking loss.

[0024] In one of the embodiments, the image repair model comprises a discriminator.

[0025] The determination of a repair loss based on the real image corresponding to the sample image and the repair image comprises:

[0026] determine an adversarial loss based on discriminant feedback of the discriminator for the repaired image;

[0027] determine a perceptual loss based on feature similarity between the real image corresponding to the sample image and the repaired image;

[0028] perform weighted summation on the adversarial loss and the perceptual loss to obtain a repair loss.

[0029] In one of the embodiments, the parameter adjustment of the image repair model based on the repair loss and the ranking loss to obtain the trained image repair model comprises:

[0030] perform weighted summation on the repair loss and the ranking loss to obtain a target loss;

[0031] perform parameter adjustment of the image repair model based on the target loss to obtain the trained image repair model.

[0032] In one of the embodiments, the method further comprises:

[0033] obtain a to-be-repaired image;

[0034] perform image repair on the to-be-repaired image by the trained image repair model to obtain a repaired target image.

[0035] In a second aspect, the present application further provides an image repair model training device, comprising:

[0036] an obtaining module configured to obtain a sample image, perform image repair on the sample image by an image repair model to obtain a repaired image;

[0037] a repair loss determination module configured to determine a repair loss based on a real image corresponding to the sample image and the repaired image;

[0038] a ranking loss determination module configured to perform repair quality prediction on the repaired image by a target ranking network to obtain a quality parameter, and determine a ranking loss according to the quality parameter; wherein the target ranking network is trained based on multiple reference images, the multiple reference images are repaired according to different repair manners for the same original image, and the quality parameter corresponding to each of the multiple reference images matches the superior-inferior relationship between the repair effect corresponding to each of the multiple reference images;

[0039] an output module configured to perform parameter adjustment of the image repair model based on the repair loss and the ranking loss to obtain a trained image repair model.

[0040] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0041] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0042] In a fifth aspect, the present application also provides a computer program product comprising a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0043] The above image inpainting model training method and device, computer device, computer readable storage medium and computer program product, the target ranking network is trained by multiple reference images. Since the multiple reference images are obtained by inpainting the same original image in different inpainting manners, the multiple reference images can reflect the visual features concerned by different inpainting manners. The quality parameters corresponding to the multiple reference images output by the target ranking network match the advantages and disadvantages between the respective inpainting effects of the multiple reference images, so that the target ranking network can focus on the visual features highly related to the inpainting effect. Meanwhile, the target ranking model uses the relative advantages and disadvantages between different inpainting effects during training, rather than using the evaluation index values of different inpainting effects, thereby avoiding the numerical difference problem caused by the difference between the numerical ranges corresponding to different evaluation indexes included in the inpainting effect. When training the image inpainting model, not only the inpainting loss between the real image and the inpainting image is considered to ensure the image similarity between the inpainting image and the real image, but also the ranking loss determined based on the quality parameters of the inpainting image is considered to ensure the perceived quality of the inpainting effect of the inpainting image, so that the inpainting image generated by the obtained image inpainting model can take into account the image similarity and the perceived quality of the inpainting effect, thereby improving the inpainting effect of the image inpainting model.

[0044] In addition, the reference images are paired two by two to obtain image pairs, and the advantages and disadvantages of the image pairs are determined by using the respective repair effects of the reference images, and through label conversion, the problem of uneven distribution of absolute numerical values of the repair effects is avoided, the continuous score space is discretized into a clear ranking relationship, which is conducive to the target ranking network focusing on the relative advantages and disadvantages between the reference images; and through the marginal ranking loss, the parameters of the ranking network are adjusted, so that the quality parameters output by the target ranking network for the reference images not only meet the advantages and disadvantages between the reference images, but also maintain a sufficient safety margin, so that the target ranking network has stronger generalization ability and can adapt to various complex images; by converting the quality parameters into a function to obtain a ranking loss, the stability of the loss function in the numerical calculation process can be effectively enhanced, and at the same time, the ranking loss can be combined with other loss functions for combined optimization; by combining the perception loss and the adversarial loss to obtain the repair loss, the global authenticity of the repaired image, i.e., the global quality, is reflected through the adversarial loss, and the detail features of the repaired image, i.e., the local quality, are reflected through the perception loss, so that the global quality and the local quality of the repaired image are balanced. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 An application environment diagram of the image repair model training method in an embodiment;

[0047] Figure 2 A flowchart of the image repair model training method in an embodiment;

[0048] Figure 3 A flowchart of the image repair model training method in another embodiment;

[0049] Figure 4 An application flowchart involved in an embodiment;

[0050] Figure 5 A structural block diagram of the image repair model training device in an embodiment;

[0051] Figure 6 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0053] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.

[0054] The image repairing model training method provided by the embodiments of the present application can be applied to the application environment as shown in the figure. Figure 1 The terminal 101 communicates with the server 102 through the network. The data storage system can store the data required by the server 102 to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0055] The image repairing model can be deployed in the server 102. The image repairing model is trained by executing the method provided by the present application, specifically including: obtaining a sample image, repairing the sample image by the image repairing model to obtain a repaired image; determining a repairing loss based on the real image corresponding to the sample image and the repaired image; predicting the repairing quality of the repaired image by a target ranking network to obtain a quality parameter, and determining a ranking loss according to the quality parameter; wherein the target ranking network is trained based on multiple reference images, the multiple reference images are obtained by repairing the same original image in different repairing manners, and the quality parameters corresponding to the multiple reference images respectively match the superior-inferior relationship between the repairing effects corresponding to the multiple reference images respectively; adjusting the parameters of the image repairing model based on the repairing loss and the ranking loss to obtain a trained image repairing model; then, when the terminal 101 receives a target image to be repaired, the terminal 101 can send the target image to be repaired to the server 102, and the server 102 repairs the target image by using the trained image repairing model, and returns the repaired image to the terminal 101.

[0056] The terminal 101 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude aerial vehicles, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 102 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0057] In an exemplary embodiment, as shown in Figure 2 , a method for training an image inpainting model is provided. The method is described by taking the server in Figure 1 as an example, and includes the following steps 201 to 204. Wherein:

[0058] Step 201, obtaining a sample image, and performing image inpainting on the sample image by an image inpainting model to obtain an inpainting image.

[0059] The sample image can be a collected defect image, such as a blurred image or an occluded image in a monitoring video, or an image obtained by performing a series of image degradation processes on a real image. For example, a real image is obtained by randomly occluding, adding image noise, or adjusting the color of the real image to distort the real image.

[0060] In some embodiments, the image inpainting model is a model constructed based on a convolutional neural network, or a model constructed based on a generative adversarial network, which is not specifically limited here.

[0061] In some embodiments, the image inpainting model is a model constructed based on a generative adversarial network. Further, the generative adversarial network includes a generator and a discriminator, wherein the generator is configured to generate an inpainting image according to a sample image, and the discriminator is configured to determine whether the inpainting image is a real image corresponding to the sample image, so as to improve the quality of image inpainting through the confrontation between the generator and the discriminator.

[0062] Step 202, determining an inpainting loss based on the real image corresponding to the sample image and the inpainting image.

[0063] The inpainting loss is used to describe the difference in features between the real image and the inpainting image. It can be understood that the more similar the features between the real image and the inpainting image, the closer the inpainting image is to the real image.

[0064] In some embodiments, the repairing loss can include a VGG (Visual Geometry Group)-based perceptual loss, and can also include a style loss, a content loss, an adversarial loss, a pixel loss, etc.

[0065] In step 203, a quality parameter is obtained by performing a repairing quality prediction on the repaired image through a target ranking network, and a ranking loss is determined according to the quality parameter; wherein the target ranking network is trained based on multiple reference images, the multiple reference images are obtained by repairing the same original image in different repairing manners, and the quality parameters corresponding to the multiple reference images match the superior-inferior relationship between the repairing effects corresponding to the multiple reference images.

[0066] Different repairing manners refer to image repairing by using different image repairing algorithms; it can be understood that different image repairing algorithms focus on different aspects when performing image repairing, and the image features of the obtained repaired images also differ.

[0067] The quality parameter refers to a quality score of the repaired image in repairing quality output by the target ranking network.

[0068] The repairing effect of the reference image refers to a score of the reference image on an image evaluation objective index (such as NIQE, PI, etc.), specifically, the higher the score of the reference image on the image evaluation objective index, the better the repairing effect; it should be noted that since the image evaluation objective index often includes non-differentiable operations such as statistical calculation, matrix inversion, numerical solution, etc., it is impossible to directly establish an explicit mathematical mapping relationship between the image features and the index score, which further makes it difficult for the objective index score to directly participate in the gradient backpropagation optimization process of the image repairing model, and further limits the feasibility of directly guiding the model parameter update as a loss function.

[0069] The superior-inferior relationship of the repairing effect of the reference image refers to the score difference of different reference images on the image evaluation objective index; specifically, if the score of reference image a on the image evaluation objective index is higher than the score of image b on the image evaluation objective index, then the repairing effect of reference image a is superior to the repairing effect of reference image b.

[0070] The superior-inferior relationship between the quality parameters corresponding to the multiple reference images and the repairing effects corresponding to the multiple reference images is matched, which means that the size relationship between the quality parameters corresponding to the multiple reference images satisfies the superior-inferior relationship between the repairing effects corresponding to the multiple reference images; for example, if the repairing effect of reference image a is superior to the repairing effect of reference image b, then the quality score of reference image a is greater than the quality score of reference image b.

[0071] In some embodiments, the training process of the ranking training network includes: inputting multiple reference images corresponding to the same original image into the ranking network, having the ranking network output the quality parameters of each reference image, then calculating the loss based on the quality parameters of the multiple reference images and the superiority or inferiority of the restoration effects of each of the multiple reference images, and then adjusting the parameters of the ranking network based on the loss until the training termination condition is met, thereby obtaining the target ranking network.

[0072] Understandably, by determining the relative merits of the restoration effects of multiple reference images, the target ranking model can focus on the relative relationships between the restoration effects of multiple reference images, rather than the absolute values ​​of the restoration effects. This can effectively prevent the target ranking model from being unable to distinguish the quality of the restoration effects when the values ​​are small, thus affecting the training effect of the target ranking model.

[0073] Clearly, through the above training, the target ranking network can output better quality parameters for images with good restoration results and worse quality parameters for images with poor restoration results. Moreover, since the quality parameters output by the target ranking network are determined based on the image features of the restored image, meaning there is a correlation between the quality parameters and image features, and the quality parameters corresponding to multiple reference images are matched with the quality of restoration results for each of the multiple reference images, meaning there is a correlation between the quality parameters and restoration results, the target ranking network can learn from multiple reference images with quality relationships to uncover visual features highly correlated with restoration results. This allows it to transform the scores of non-differentiable objective image evaluation indicators into fully differentiable quality parameters, which can then participate in the gradient optimization process of the image restoration model.

[0074] Step 204: Based on the restoration loss and ranking loss, adjust the parameters of the image restoration model to obtain the trained image restoration model.

[0075] In some embodiments, the restoration loss and the ranking loss can be weighted and summed to obtain the target loss, and the parameters of the image restoration model can be adjusted based on the target loss to obtain the trained image restoration model.

[0076] In other embodiments, the image restoration model can be trained by adjusting the parameters of the image restoration model based on restoration loss and ranking loss respectively. For example, the parameters of the image restoration model can be adjusted first based on restoration loss until the restoration loss is less than a preset threshold, and then the parameters of the image restoration model can be adjusted based on ranking loss to obtain the trained image restoration model.

[0077] In other embodiments, a training termination condition may be set; for example, it may be that the number of training iterations reaches a threshold, or that the target loss is less than a loss threshold. The specific training termination condition is not limited here.

[0078] In the above image restoration model training method, a target ranking network is trained using multiple reference images. Since these reference images are obtained by restoring the same original image using different restoration methods, they can reflect the visual features that different restoration methods focus on. Furthermore, the quality parameters of each reference image output by the target ranking network match the relative merits of their respective restoration effects, allowing the target ranking network to focus on visual features highly correlated with the restoration effect. Simultaneously, during training, the target ranking model utilizes the relative merits of different restoration effects rather than their individual evaluation metrics, avoiding numerical discrepancies caused by differences in the numerical ranges of different evaluation metrics. Subsequently, when training the image restoration model, not only is the restoration loss between the real and restored images considered to ensure image similarity, but also the ranking loss determined by the quality parameters of the restored images is considered to ensure the perceptual quality of the restored images. Ultimately, the restored images generated by the image restoration model can balance image similarity and perceptual quality, improving the overall restoration effect.

[0079] In one exemplary embodiment, such as Figure 3 As shown, the image restoration model training method includes steps 301 to 305. Wherein:

[0080] Step 301: Obtain the original image.

[0081] The original image is the image that needs to be repaired; it can be a collected defective image or an image obtained by performing a series of image degradation processes on a real image.

[0082] Step 302: Perform image restoration on the original image based on various restoration methods to obtain multiple reference images corresponding to the original image.

[0083] Specifically, various image inpainting algorithms are used to inpaint the original image, resulting in multiple reference images. These algorithms include SRResNet (Super-Resolution Residual Network), SRGAN (Super-Resolution Generative Adversarial Network), and ESRGAN (Enhanced Super-Resolution Generative Adversarial Network).

[0084] Step 303: Pair multiple reference images together to obtain at least one image pair.

[0085] For example, if there are three reference images a, b and c, then pairing them together will yield three image pairs (a, b), (a, c) and (c, b).

[0086] Step 304: For each image pair, determine the superiority / inferiority relationship label of the image pair based on the restoration effect of each of the two reference images in the image pair.

[0087] The restoration effect of each reference image is calculated by professional evaluators based on objective indicators such as NIQE; the superiority / inferiority relationship label, such as ab, indicates that reference image a is superior to reference image b.

[0088] Step 305: Based on the image pairs and the labels indicating the superiority or inferiority of the image pairs, train the ranking network to obtain the target ranking network.

[0089] In other embodiments, an image group can be generated based on multiple reference images. The quality relationship label of the image group is determined according to the repair effect of each reference image. The ranking network is trained using the image group and the quality relationship label of the image group to obtain the target ranking network.

[0090] In the above embodiments, reference images are paired to obtain image pairs. Then, the corresponding restoration effects of each reference image are used to determine the superiority or inferiority relationship label of the image pairs. Through label transformation, the problem of uneven distribution of absolute values ​​of restoration effects is avoided. The continuous score space is discretized into a clear ranking relationship, which helps the target ranking network focus on the relative superiority or inferiority relationship between reference images.

[0091] In some embodiments, step 305 includes steps 401 to 403. Wherein:

[0092] Step 401: The sorting network outputs the corresponding quality parameters for each of the two reference images included in the image pair.

[0093] Specifically, the ranking network consists of three twin neural network branches, each of which is a convolutional neural network. For each reference image, the three branches are used to extract visual features at different scales (spatial scale, receptive field scale, and semantic scale) of the image, and finally output a quality parameter corresponding to the reference image as the quality parameter of the reference image.

[0094] It is understandable that since the quality parameters of the reference image are determined based on the visual features of the reference image, a functional mapping relationship between the quality parameters and the visual features is established.

[0095] Step 402: Determine the marginal ranking loss based on the quality parameters corresponding to the two reference images and the label of the image pair's superiority or inferiority.

[0096] Marginal ranking loss improves the model's performance in ranking tasks by maximizing the difference in predicted scores between positive and negative samples while maintaining a certain margin. In other words, based on the quality parameters corresponding to the two reference images, it is necessary not only to determine the superiority or inferiority relationship between the two reference images, but also to ensure that the difference in the quality parameters corresponding to the two reference images meets the difference threshold.

[0097] Step 403: Adjust the parameters of the sorting network based on the marginal sorting loss to obtain the target sorting network.

[0098] Specifically, the parameters of the ranking network are adjusted in the direction of minimizing the marginal loss until the training termination condition of the ranking network is met, thus obtaining the target ranking network. The training termination condition of the ranking network can be that the marginal ranking loss is less than the loss threshold, or that the number of training iterations reaches the number threshold.

[0099] In the above embodiments, the parameters of the ranking network are adjusted by the marginal ranking loss so that the quality parameters output by the target ranking network for the reference image can not only meet the superiority-inferiority relationship between the reference images, but also maintain a sufficient safety margin, so that the target ranking network has a stronger generalization ability and can adapt to various complex and ever-changing images.

[0100] In some embodiments, step 203 includes steps 501 to 503. Wherein:

[0101] Step 501: Obtain the target sorting network.

[0102] In some embodiments, before using the image restoration model for image restoration, a target ranking network can be invoked from the database to form a joint model with the image restoration model, and then the joint model can be used for image restoration.

[0103] Step 502: The target ranking network performs restoration quality prediction on the restored image to obtain quality parameters.

[0104] Specifically, the repaired image is input into the target ranking network, which extracts features from the repaired image and predicts the repair quality based on the extracted features, outputting the quality score of the repaired image as a quality parameter.

[0105] Step 503: Perform a function transformation based on the quality parameters to obtain the ranking loss.

[0106] For example, the quality parameter is transformed using the Sigmoid function to obtain the ranking loss; furthermore, the larger the quality parameter, the higher the ranking loss; the smaller the quality parameter, the smaller the ranking loss.

[0107] Understandably, since the output quality parameters may have a large numerical range, the instability of this numerical range can cause difficulties in gradient calculation and backpropagation, potentially leading to gradient explosion or gradient vanishing problems, which in turn affect the model's convergence speed and training effect. The Sigmoid function, on the other hand, has the property of mapping any real number to the (0, 1) interval. After transforming the quality parameters through it, the loss value is limited to a relatively small range, avoiding the computational problems caused by excessively large or small values. This makes gradient calculation more stable and reliable, which helps the model to update parameters and converge more smoothly.

[0108] In the above embodiments, by transforming the quality parameters into a function to obtain the ranking loss, the stability of the loss function in the numerical calculation process can be effectively enhanced. At the same time, it is convenient to combine and jointly optimize the ranking loss with other loss functions.

[0109] In some embodiments, the image inpainting model is built based on a generative adversarial network, and the image inpainting model includes a generator and a discriminator; step 202 includes steps 601 to 603. Wherein:

[0110] Step 601: Determine the adversarial loss based on the discriminator's discrimination feedback for the repaired image.

[0111] The adversarial loss is used to encourage the generator to produce the most realistic restored image possible, and to help the discriminator to distinguish between the real image and the restored image as accurately as possible.

[0112] Specifically, the image inpainting model includes a generator that outputs a repaired image based on a sample image; a discriminator that distinguishes whether the repaired image is a real image and provides discriminative feedback, which is the probability that the repaired image is a real image. Then, the probability provided by the discriminative feedback is transformed by a function to obtain the adversarial loss.

[0113] Step 602: Determine the perceptual loss based on the feature similarity between the real image and the repaired image corresponding to the sample image.

[0114] Perceptual loss is used to describe the similarity between the real image and the restored image in the feature space; perceptual loss can ensure the basic image quality of the restored image in terms of features.

[0115] In some embodiments, perceptual loss can be determined by the similarity between the real image and the restored image in the VGG feature space.

[0116] Step 603: Perform a weighted summation based on the adversarial loss and the perceptual loss to obtain the repair loss.

[0117] Among them, adversarial loss focuses on the global realism of the restored image, i.e. global quality, while perceptual loss focuses on the detailed features of the restored image, i.e. local quality. Therefore, by combining perceptual loss and adversarial loss, the restoration loss can be obtained, which can effectively balance the global quality and local quality of the restored image.

[0118] In some embodiments, the adversarial loss has a smaller weight to facilitate maintaining training stability, while the perceptual loss has a larger weight to facilitate ensuring the most basic image quality.

[0119] In the above embodiments, since adversarial loss can improve global realism but may ignore local details, while perceptual loss can focus on more detailed image information, combining perceptual loss and adversarial loss to obtain inpainting loss can balance global quality and local quality.

[0120] In some embodiments, step 204 includes steps 701 to 702. Wherein:

[0121] Step 701: Perform a weighted summation based on the repair loss and the sorting loss to obtain the target loss.

[0122] It is understandable that the restoration loss reflects the image quality of the restored image in the feature dimension; the ranking loss reflects the image quality of the restored image in the human perception dimension; by weighted summing of the restoration loss and the ranking loss, the target loss is obtained. The target loss can be used to balance the restoration image in the feature dimension and the human perception dimension, so that the restored image achieves a balance between image distortion and image quality.

[0123] Step 702: Adjust the parameters of the image inpainting model based on the target loss to obtain the trained image inpainting model.

[0124] Specifically, the parameters of the image inpainting model are adjusted in a way that minimizes the target loss to obtain the trained image inpainting model.

[0125] In the above embodiments, the target loss is obtained by weighted summing of the restoration loss and the ranking loss. The target loss reflects the image quality of the restored image in the feature dimension through the restoration loss, and reflects the image quality of the restored image in the human perception dimension through the ranking loss. In this way, the parameters of the image restoration model are adjusted using the target loss to obtain the trained image restoration model. The trained image restoration model can balance image integrity and image quality, thereby improving the restoration effect of the image restoration model.

[0126] In some embodiments, the image inpainting model training method further includes steps 801 to 802. Wherein:

[0127] Step 801: Obtain the image to be repaired.

[0128] For example, the image to be repaired can be a blurry or obscured image from a surveillance video; it can also be a low-resolution old photo.

[0129] Step 802: The trained image inpainting model performs image inpainting based on the image to be inpainted, and obtains the inpainted target image.

[0130] The image restoration model can be trained according to the training method described in the above embodiments to obtain the trained image restoration model.

[0131] In this embodiment, the trained image inpainting model performs image inpainting based on the image to be inpainted, resulting in a repaired target image. This repaired target image can have high image quality in terms of image features and is more in line with objective indicators of human perception, achieving a balance between image quality and image distortion prevention.

[0132] For easier understanding, please refer to Figure 4 , Figure 4 The following is an application flowchart illustrating an embodiment of this application, including steps one through three, wherein:

[0133] Step 1: Construct a ranking learning dataset. First, collect a large number of original images. For each original image, generate results using three different levels of super-resolution methods. Use metrics such as NIQE to score the quality of all generated super-resolution images. Pair the three results of the same original image together, determine the quality relationship of each pair of images based on the NIQE score, and generate ranking labels.

[0134] Step two: Train the ranking network. Specifically, the ranking network uses three Siamese neural network branches. Each neural network branch includes a Feature Pyramid Network, a global average pooling layer, and a fully connected layer (FC layer). The Feature Pyramid Network is used to extract visual features of different scales of the image. After global average pooling, the fully connected layer outputs a scalar quality score.

[0135] In the training process of the ranking network, marginal ranking loss is used. This ensures that the ranking network not only ranks correctly but also that its output scalar quality score has a sufficient safety margin.

[0136] Step 3, Joint Training. The image inpainting model and the trained object ranking network are combined to obtain a joint model, with the object ranking network serving as an additional quality assessment component. Specifically, the joint model consists of four main parts:

[0137] Generator: Responsible for generating high-resolution results from low-resolution images;

[0138] Discriminator: Distinguishes generated images from real images and provides adversarial learning signals;

[0139] VGG Feature Extractor: Calculates perceptual loss to ensure that the generated image is similar to the real image in the feature space;

[0140] Ranker network: evaluates the perceptual quality of the generated images, and its parameters are frozen at this stage.

[0141] Building upon this, the joint model's loss function uses a weighted combination of three loss functions: perceptual loss (weight 1.0), adversarial loss (weight 0.005), and ranking loss (weight 0.03). The perceptual loss, as the dominant loss, ensures the similarity between the generated and real images in the VGG feature space, guaranteeing basic image quality. The adversarial loss, using feedback from the discriminator, increases the realism and detail richness of the generated images; its weight is relatively small to maintain training stability. The ranking loss converts the output of the target ranking network into a loss value using the Sigmoid function; the loss value is higher when the target ranking network considers the generated image quality poor, and lower when the quality is good.

[0142] In the above application process, a ranking model is trained by using a large number of images ranked by perceptual metrics such as NIQE. On the one hand, ranking learning overcomes the problem of uneven distribution of perceptual metrics; on the other hand, the ranking model can automatically discover visual features highly correlated with NIQE, transforming non-differentiable perceptual metrics into fully differentiable quality parameters, thus achieving end-to-end optimization of perceptual quality. Simultaneously, marginal ranking loss is used to allow the target ranking network to focus on learning the relative relationships of perceptual metrics between different restored images, rather than absolute values, avoiding the problem of differences in the numerical ranges of different metrics. Finally, when training the image restoration model, not only is the restoration loss between the real and restored images considered to ensure image similarity between the restored and real images, but also the ranking loss determined based on the quality parameters of the restored images is considered to ensure the perceptual quality of the restored images in terms of restoration effect. Ultimately, the restored images generated by the image restoration model can balance image similarity and perceptual quality of restoration effect, improving the restoration performance of the image restoration model.

[0143] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0144] Based on the same inventive concept, this application also provides an image restoration model training apparatus for implementing the image restoration model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the image restoration model training apparatus provided below can be found in the limitations of the image restoration model training method described above, and will not be repeated here.

[0145] In one exemplary embodiment, such as Figure 5 As shown, a schematic diagram of an image restoration model training device is provided. The image restoration model training device 900 includes:

[0146] The acquisition module 901 is used to acquire sample images and perform image restoration on the sample images using an image restoration model to obtain restored images.

[0147] The restoration loss determination module 902 is used to determine the restoration loss based on the real image and the restored image corresponding to the sample image;

[0148] The ranking loss determination module 903 is used to predict the restoration quality of the restored image through the target ranking network, obtain quality parameters, and determine the ranking loss based on the quality parameters. The target ranking network is trained based on multiple reference images, which are obtained by restoring the same original image according to different restoration methods. The quality parameters of each of the multiple reference images are matched with the restoration effect of each of the multiple reference images.

[0149] Output module 904 is used to adjust the parameters of the image restoration model based on restoration loss and ranking loss to obtain the trained image restoration model.

[0150] In some embodiments, the image restoration model training device 900 further includes a ranking network training module for acquiring the original image; performing image restoration on the original image based on multiple different restoration methods to obtain multiple reference images corresponding to the original image; pairing the multiple reference images in pairs to obtain at least one image pair; for each image pair, determining the superiority / inferiority relationship label of the image pair based on the restoration effects corresponding to the two reference images in the image pair; and training the ranking network based on the image pair and the superiority / inferiority relationship label of the image pair to obtain the target ranking network.

[0151] In some embodiments, the ranking network training module is specifically used to have the ranking network output corresponding quality parameters for each of the two reference images included in the image pair; determine the marginal ranking loss based on the quality parameters corresponding to the two reference images and the quality relationship label of the image pair; and adjust the parameters of the ranking network based on the marginal ranking loss to obtain the target ranking network.

[0152] In some embodiments, the ranking loss determination module 903 is specifically used to obtain a target ranking network; the target ranking network performs restoration quality prediction on the restored image to obtain quality parameters; and a function transformation is performed based on the quality parameters to obtain the ranking loss.

[0153] In some embodiments, the image inpainting model includes a discriminator; an inpainting loss determination module 902, specifically used to determine adversarial loss based on the discriminator's discrimination feedback for the inpainted image; determine perceptual loss based on the feature similarity between the real image corresponding to the sample image and the inpainted image; and obtain the inpainting loss by performing a weighted summation based on the adversarial loss and the perceptual loss.

[0154] In some embodiments, the output module 904 is specifically used to perform a weighted summation based on the restoration loss and the ranking loss to obtain the target loss; and to adjust the parameters of the image restoration model based on the target loss to obtain the trained image restoration model.

[0155] In some embodiments, the image inpainting model training device 900 further includes a deployment module for acquiring the image to be repaired; the trained image inpainting model performs image inpainting based on the image to be repaired to obtain the repaired target image.

[0156] Each module in the aforementioned image restoration model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0157] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores image restoration data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements an image restoration model training method.

[0158] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0161] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0165] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training an image restoration model, characterized in that, The method includes: A sample image is acquired, and an image restoration model is used to restore the sample image to obtain a restored image. Based on the real image corresponding to the sample image and the repaired image, determine the repair loss; The restoration quality of the restored image is predicted by a target ranking network to obtain quality parameters, and the ranking loss is determined based on the quality parameters. The target ranking network is trained on multiple reference images, which are obtained by restoring the same original image using different restoration methods. The quality parameters of each of the multiple reference images are matched with the restoration effect of each of the multiple reference images. Based on the restoration loss and the ranking loss, the parameters of the image restoration model are adjusted to obtain the trained image restoration model.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the original image; The original image is restored using various restoration methods to obtain multiple reference images corresponding to the original image. The multiple reference images are paired up to obtain at least one image pair; For each image pair, a quality relationship label for the image pair is determined based on the restoration effect corresponding to each of the two reference images in the image pair. Based on the image pairs and the labels indicating their relative merits, the ranking network is trained to obtain the target ranking network.

3. The method according to claim 2, characterized in that, The process of training the ranking network based on the image pairs and their relative quality labels to obtain the target ranking network includes: The sorting network outputs corresponding quality parameters for each of the two reference images included in the image pair; Based on the quality parameters corresponding to the two reference images and the quality relationship labels of the image pair, the marginal ranking loss is determined; The ranking network is adjusted based on the marginal ranking loss to obtain the target ranking network.

4. The method according to claim 1, characterized in that, The step of predicting the restoration quality of the restored image using a target ranking network to obtain quality parameters, and determining the ranking loss based on the quality parameters, includes: Obtain the target sorting network; The target ranking network performs restoration quality prediction on the restored image to obtain quality parameters; Based on the quality parameters, a function transformation is performed to obtain the ranking loss.

5. The method according to claim 1, characterized in that, The image restoration model includes a discriminator; The step of determining the restoration loss based on the real image corresponding to the sample image and the restored image includes: Based on the discriminator's feedback on the repaired image, the adversarial loss is determined; Based on the feature similarity between the real image corresponding to the sample image and the restored image, the perceptual loss is determined; The repair loss is obtained by weighted summation of the adversarial loss and the perceptual loss.

6. The method according to claim 1, characterized in that, The step of adjusting the parameters of the image restoration model based on the restoration loss and the ranking loss to obtain the trained image restoration model includes: The target loss is obtained by weighted summation of the repair loss and the sorting loss. The parameters of the image restoration model are adjusted based on the target loss to obtain the trained image restoration model.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the image to be repaired; The trained image inpainting model performs image inpainting based on the image to be inpainted, resulting in the inpainted target image.

8. An image restoration model training device, characterized in that, The device includes: The acquisition module is used to acquire sample images and perform image restoration on the sample images using an image restoration model to obtain restored images. The restoration loss determination module is used to determine the restoration loss based on the real image corresponding to the sample image and the restored image; The ranking loss determination module is used to predict the restoration quality of the restored image through the target ranking network, obtain quality parameters, and determine the ranking loss based on the quality parameters; wherein, the target ranking network is trained based on multiple reference images, which are obtained by restoring the same original image according to different restoration methods, and the quality parameters corresponding to each of the multiple reference images are matched with the superiority or inferiority relationship between the restoration effects corresponding to each of the multiple reference images. The output module is used to adjust the parameters of the image restoration model based on the restoration loss and the ranking loss to obtain the trained image restoration model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images