Model training method and platform, image inpainting method and device, equipment and medium

Abstract

The model training method and platform, the image repairing method and device, the equipment and the medium belong to the technical field of image processing. The model training method comprises the following steps: acquiring a plurality of image sample pairs, wherein the image sample pair comprises a first image sample and a second image sample of the same image; the image quality of the second image sample is higher than that of the first image sample; training a first preset model by taking the plurality of image sample pairs as training samples, wherein the first preset model is used to improve the image quality of the first image sample, and the training process comprises the following steps: acquiring text data corresponding to a predicted image currently output by the first preset model; the text data comprises data for evaluating the image quality of the predicted image; and updating the parameters of the first preset model based on the text data, the predicted image and the second image sample.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular to a model training method and platform, an image restoration method and device, equipment and a medium. BACKGROUND

[0002] With the development of image processing technology, it is necessary to restore images according to requirements, such as increasing the resolution of an image from low to high to improve the clarity of the image, or repairing scratches, noise and the like in the image to improve the image quality, or restoring missing parts in the image. In related technologies, image restoration is usually achieved through deep learning methods. However, the training mechanism used in existing deep learning methods is limited. SUMMARY

[0003] The present disclosure provides a model training method, which comprises:

[0004] obtaining a plurality of image sample pairs, wherein each image sample pair comprises a first image sample and a second image sample of the same image, and the image quality of the second image sample is higher than that of the first image sample;

[0005] training a first preset model using the plurality of image sample pairs as training samples, wherein the first preset model is used to improve the image quality of the first image sample, and the training process comprises:

[0006] obtaining text data corresponding to a predicted image output by the first preset model, wherein the text data comprises data for evaluating the image quality of the predicted image;

[0007] updating the parameters of the first preset model based on the text data, the predicted image and the second image sample.

[0008] For example, the text data is text input by a user for the predicted image, and the updating of the parameters of the first preset model based on the text data, the predicted image and the second image sample comprises:

[0009] determining the similarity between a target text and the predicted image, wherein the target text is the text data or a text with a semantic opposite to the text data;

[0010] determining a loss value based on the predicted image and the second image sample;

[0011] updating the parameters of the first preset model based on the similarity and the loss value.

[0012] For example, the determining the similarity between the target text and the predicted image comprises:

[0013] encoding the target text to obtain a text feature vector of the target text;

[0014] encoding the predicted image to obtain an image feature vector of the predicted image; wherein the text feature vector and the image feature vector have consistent dimensions;

[0015] determining the similarity based on the text feature vector and the image feature vector.

[0016] For example, the training the first preset model with the plurality of image sample pairs as training samples comprises:

[0017] training the first preset model with part of the image sample pairs as training samples; wherein, in the first training, based on the predicted image output by the first preset model and the second image sample, the parameters of the first preset model are updated;

[0018] training the first preset model obtained by the first training with part of the image sample pairs as training samples;

[0019] wherein, in the second training, based on the text data, the predicted image and the second image sample, the parameters of the first preset model obtained by the first training are updated.

[0020] For example, the method further comprises:

[0021] obtaining a plurality of third image samples and text data samples corresponding to the third image samples; wherein the text data samples are used to describe the image quality of the third image samples;

[0022] training a second preset model based on the plurality of third image samples and the text data samples; wherein the second preset model is used to determine the similarity between the third image samples and the text data samples;

[0023] the updating the parameters of the first preset model based on the text data, the predicted image and the second image sample comprises:

[0024] inputting the predicted image and the text data into the second preset model after the third training is completed;

[0025] updating the parameters of the first preset model based on the similarity output by the second preset model, the predicted image and the second image sample.

[0026] The text data sample carries a category label indicating whether the description of the text data sample conforms to the image quality of the third image sample.

[0027] The third image sample and the text data sample are input into the second preset model to obtain a predicted similarity between the third image sample and each text data sample output by the second preset model.

[0028] The parameters of the second preset model are updated based on the predicted similarity and the category label.

[0029] The third image sample corresponds to two text data samples, including a first category of text data sample and a second category of text data sample, the first category representing that the description of the text data sample conforms to the image quality of the third image sample, and the second category representing that the description of the text data sample does not conform to the image quality of the third image sample.

[0030] The second preset model includes a text encoder and an image encoder, and a similarity determination module connected to the text encoder and the image encoder respectively.

[0031] The text encoder is configured to perform text encoding on the text data sample to obtain a predicted text vector.

[0032] The image encoder is configured to perform image encoding on the third image sample to obtain a predicted image vector, wherein the predicted image vector and the predicted text vector have consistent dimensions.

[0033] The similarity determination module is configured to determine a predicted similarity between the predicted image vector and the predicted text vector.

[0034] During the training of the first preset model, the third training of the second preset model is performed, and the third image samples include at least one of the first image sample, the second image sample, and a predicted image output by the first preset model before the current time.

[0035] The third training is performed in a gap of training the first preset model, the third image sample includes a predicted image output by the first preset model for the first image sample, and the third training of the second preset model based on the third image samples and the text data samples includes:

[0036] inputting a plurality of the first image samples into the first preset model;

[0037] inputting a predicted image output by the first preset model and the text data sample corresponding to the predicted image into the second preset model to perform the third training on the second preset model;

[0038] In the third training, parameters of the first preset model are fixed.

[0039] For example, after the third training, the training on the first preset model comprises:

[0040] inputting a plurality of the first image samples into the first preset model;

[0041] inputting a predicted image output by the first preset model and the text data into the second preset model;

[0042] updating parameters of the first preset model based on the predicted image, the second image sample, and a predicted similarity output by the second preset model;

[0043] In the updating of the parameters of the first preset model, parameters of the second preset model are fixed.

[0044] For example, the obtaining of the plurality of third image samples and the text data samples corresponding to the third image samples comprises:

[0045] obtaining first text data corresponding to the third image sample;

[0046] determining a category of the first text data, the category being used to indicate whether a description of the first text data is consistent with an image quality of the third image sample;

[0047] generating second text data based on the first text data; wherein a category of the second text data is different from a category of the first text data;

[0048] inputting the first text data and the second text data as the text data sample corresponding to the third image sample.

[0049] For example, the obtaining of the text data corresponding to a predicted image currently output by the first preset model comprises:

[0050] determining, from the plurality of third image samples, whether there is a target third image sample corresponding to the predicted image; wherein the target third image sample and the predicted image correspond to a same first image sample;

[0051] If yes, a text data sample corresponding to the target third image sample is taken as the text data;

[0052] If no, text data input for the predicted image is obtained.

[0053] For example, the text data corresponding to the predicted image output by the first preset model is obtained, comprising:

[0054] The predicted image is displayed.

[0055] Text data input for the predicted image is obtained.

[0056] For example, after the predicted image is displayed, the method further comprises:

[0057] An input operation on an operation interface displaying the predicted image is listened to.

[0058] If the input operation is not listened to, preset text data is taken as the text data corresponding to the predicted image.

[0059] The text data input for the predicted image comprises:

[0060] If the input operation is listened to, text data input for the predicted image is obtained.

[0061] For example, the text data comprises at least one word; the at least one word is used to describe the image quality of the predicted image in different image regions and / or image quality of different quality dimensions.

[0062] For example, the method further comprises at least one of:

[0063] In response to the first training, the predicted image output by the first preset model is displayed, and in response to a first preset operation performed on the predicted image, the first training is ended;

[0064] In response to the second training, the predicted image output by the first preset model is displayed, and in response to a second preset operation performed on the predicted image, the second training is ended.

[0065] For example, in the process of the third training on the second preset model, the method further comprises:

[0066] A loss value corresponding to at least one training of the second preset model before the current time is displayed; wherein the gradient is determined by the predicted similarity and the category label.

[0067] In response to a third preset operation performed on each of the displayed loss values, the third training is ended.

[0068] The present disclosure also provides an image inpainting method, comprising:

[0069] a target image to be inpainted;

[0070] inputting the target image into an image inpainting model; wherein the image inpainting model is a first preset model trained according to the model training method;

[0071] obtaining an inpainted image output by the image inpainting model; wherein the image quality of the inpainted image is higher than that of the target image.

[0072] The present disclosure also provides a model training platform, comprising:

[0073] a sample library for storing a plurality of image sample pairs, each image sample pair comprising a first image sample and a second image sample of the same image; wherein the image quality of the second image sample is higher than that of the first image sample;

[0074] a training module for training a first preset model multiple times using the plurality of image sample pairs as training samples, the first preset model being used to improve the image quality of the first image sample; wherein the training process comprises:

[0075] obtaining text data corresponding to a predicted image currently output by the first preset model; wherein the text data comprises data for evaluating the image quality of the predicted image;

[0076] updating the parameters of the first preset model based on the text data, the predicted image, and the second image sample.

[0077] The present disclosure also provides an image inpainting device, comprising:

[0078] a first obtaining module for obtaining a target image to be inpainted;

[0079] an input module for inputting the target image into an image inpainting model; wherein the image inpainting model is a first preset model trained according to the model training method;

[0080] a second obtaining module for obtaining an inpainted image output by the image inpainting model; wherein the image quality of the inpainted image is higher than that of the target image.

[0081] The model training method provided in the present disclosure can obtain a plurality of image sample pairs, and train a first preset model by taking the plurality of image sample pairs as training samples. The process of training includes: obtaining text data corresponding to a predicted image output by the first preset model currently; and updating parameters of the first preset model based on the text data, the predicted image and a second image sample. Specifically, the image sample pair includes a first image sample with low quality and a second image sample with high quality of the same image, and the first preset model is used to improve the image quality of the first image sample.

[0082] In the process of training the first preset model by using the plurality of image sample pairs, the parameters of the first preset model can be updated by using the text data, the predicted image and the second image sample according to the training method provided in the present disclosure. Since the text data includes data for evaluating the image quality of the predicted image, that is, the text data can be used to evaluate the image quality of the predicted image, the training can be supervised not only based on the difference between the predicted image output by the model and the second image sample, but also based on the text data for evaluating the predicted image, so as to supervise the training according to the image quality evaluation provided by the text data. In this way, the model training can be supervised from the pixel difference dimension between the predicted image and the second image sample, and the quality evaluation dimension of the predicted image, so that the model can be combined with the quality evaluation provided, the direction of model optimization is provided through the text data, so as to optimize the model effect and improve the image restoration quality of the trained model.

[0083] The present disclosure also discloses an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the model training method or the image restoration method as described.

[0084] The present disclosure also discloses a computer readable storage medium, which stores a computer program for enabling a processor to execute the model training method or the image restoration method as described in the present disclosure.

[0085] The above description is only a summary of the technical solutions of the present disclosure. In order to more clearly understand the technical means of the present disclosure, the specific embodiments of the present disclosure can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specific embodiments of the present disclosure are described. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those of ordinary skill in the art without creative labor under the premise of the drawings.

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or related technologies, the following will briefly introduce the drawings needed to be used in the description of the embodiments or related technologies. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those of ordinary skill in the art without creative labor under the premise of the drawings. It should be noted that the proportions in the drawings are only for illustration and do not represent the actual proportions.

[0088] Figure 1 The overall flowchart of the model training method in the embodiments of the present disclosure is shown;

[0089] Figure 2 The step flowchart of the model training method in the embodiments of the present disclosure is shown;

[0090] Figure 3 The process diagram of the first preset model training in stages in the embodiments of the present disclosure is shown;

[0091] Figure 4 The process diagram of adding the training stage of the second preset model in the training process of the first preset model in the embodiments of the present disclosure is shown;

[0092] Figure 5 The step flowchart of training the second preset model in the embodiments of the present disclosure is shown;

[0093] Figure 6 The composition diagram of the third image sample input to the second preset model in the embodiments of the present disclosure is shown;

[0094] Figure 7 The acquisition process diagram of the text data sample corresponding to the third image sample in the embodiments of the present disclosure is shown;

[0095] Figure 8 The model structure diagram of the second preset model in the embodiments of the present disclosure is shown;

[0096] Figure 9 The complete process diagram of the first training, the second training and the third training in the embodiments of the present disclosure is shown;

[0097] Figure 10 Another process diagram of training the second preset model in the embodiments of the present disclosure is shown;

[0098] Figure 11 Fig. 2 shows a loss value change trend diagram of the second preset model in the third training process in the embodiment of the present disclosure;

[0099] Figure 12 Fig. 3 shows a process schematic diagram of a model training method in the embodiment of the present disclosure;

[0100] Figure 13 Fig. 4 shows a framework structure schematic diagram of a model training platform in the embodiment of the present disclosure;

[0101] Figure 14a Fig. 5 shows a schematic diagram of a first operation interface of the model training platform in the embodiment of the present disclosure in the execution of the model training process;

[0102] Figure 14b Fig. 6 shows a schematic diagram of a second operation interface of the model training platform in the embodiment of the present disclosure in the execution of the model training process;

[0103] Figure 14c Fig. 7 shows a schematic diagram of a third operation interface of the model training platform in the embodiment of the present disclosure in the execution of the model training process;

[0104] Figure 14d Fig. 8 shows a schematic diagram of a fourth operation interface of the model training platform in the embodiment of the present disclosure in the execution of the model training process;

[0105] Figure 15 Fig. 9 shows a step flow schematic diagram of an image repairing method in the embodiment of the present disclosure;

[0106] Figure 16 Fig. 10 shows a framework structure schematic diagram of an image repairing device in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0107] In order to make the above objectives, features and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure.

[0108] In related technologies, image repairing is usually achieved through deep learning methods, such as obtaining an image repairing model through training of an auto-encoding fully convolutional neural network or an adversarial generative network. However, the training effect of the model is usually adjusted and improved through means such as neural network structure design, loss function, training parameters, so that the quality of the model for repairing images is difficult to be greatly improved.

[0109] In view of this, this disclosure improves the model training process. Specifically, during the model training process, human quality feedback language for image restoration is incorporated. This allows the model to supervise training not only by utilizing the pixel differences between the restored image and the original image, but also by using human quality feedback language for image restoration. By combining this quality feedback language with the model's evaluation of the restored image, the model can perceive human visual evaluation, assisting in updating parameters and thus optimizing the model training effect and improving the quality of image restoration.

[0110] Reference Figure 1 and Figure 2 As shown, Figure 1 A schematic diagram illustrating the overall process of the model training method of this disclosure is shown. Figure 2 A schematic diagram illustrating the steps of the model training method of this disclosure is shown, as follows: Figure 1 and Figure 2 As shown, the model training method disclosed herein can be applied to electronic devices, and may specifically include the following steps:

[0111] Step S201: Obtain multiple image sample pairs, where each image sample pair includes a first image sample and a second image sample of the same image;

[0112] The image quality of the second image sample is higher than that of the first image sample.

[0113] In this embodiment, since the image sample pair includes two image samples of the same image but with different image qualities—a first image sample and a second image sample—the image content included in the first image sample and the second image sample is the same, only the image quality differs. Image quality can include image quality descriptions such as resolution, noise, and scratches. For example, the image quality of the second image sample being higher than that of the first image sample may include: the resolution of the second image sample being higher than that of the first image sample; or the noise of the second image sample being less than that of the first image sample; or, the second image sample having no scratches while the first image sample has scratches.

[0114] For example, two images with different image qualities can be captured at the same angle for the same scene, such as taking a low-resolution image and a high-resolution image of the same person, so that the low-resolution image is taken as the first image sample and the high-resolution image is taken as the second image sample; for another example, a high-definition image can be blurred, and the blurred image is taken as the first image sample and the high-definition image is taken as the second image sample; for another example, a high-definition image can be processed with scratches, noise, etc., and the image with scratches and noise is taken as the first image sample and the high-definition image is taken as the second image sample.

[0115] For example, in the application scenario of old photo restoration, the goal of model training is to train a model that can restore old photos, which needs to improve the resolution of old photos and repair missing parts in old photos. Therefore, when obtaining multiple image sample pairs, a low-resolution image and a high-resolution image of the same person can be taken, so that the low-resolution image is taken as the first image sample and the high-resolution image is taken as the second image sample; and a high-resolution image of the same person is taken, and the high-resolution image is blurred, and appropriate noise, scratches, and random extraction of part of the area are added, so that the first image sample is obtained, and the high-resolution image is taken as the second image sample.

[0116] The size of the first image sample and the second image sample can be processed to a preset size to meet the model training requirements.

[0117] Step S202: training the first preset model with multiple image sample pairs as training samples, wherein the first preset model is used to improve the image quality of the first image sample.

[0118] After obtaining multiple image sample pairs, multiple first image samples in the multiple image sample pairs can be taken as the input of the first preset model, and the second image sample can be taken as the supervision of the model, so as to train the first preset model; wherein the first preset model can adopt an existing model structure, which will not be described here.

[0119] The first preset model is mainly used for image restoration of the input first image sample, so as to improve the image quality of the first image sample. Specifically, the image restoration can mainly include clarity restoration, scratch restoration, and noise elimination, wherein the clarity restoration can mean improving the resolution of the first image sample. Of course, in addition to the image restoration in the above examples, other types of restoration can also be included, such as restoration of missing parts of the image. In practice, the first preset model can be used to perform at least one of the above image restoration, i.e., at least one of clarity restoration, scratch restoration, noise elimination, and restoration of missing parts.

[0120] The first preset model is used for outputting a predicted image, and a difference between the predicted image and the second image sample can be used for supervising the training of the first preset model, such as updating parameters of the first preset model.

[0121] In the process of training the first preset model, at least in one training, text data of image quality evaluation of the predicted image output by the first preset model is obtained, and then, based on the text data, a difference between the predicted image and an expected image quality level (an image quality level represented by the second image sample) is determined, and the difference is used together with the difference between the predicted image and the second image sample for supervising the training of the first preset model.

[0122] Correspondingly, in the process of training the first preset model, at least in one training, the following steps are performed:

[0123] Step S2021: obtaining text data corresponding to the predicted image currently output by the first preset model;

[0124] Step S2022: updating parameters of the first preset model based on the text data, the predicted image and the second image sample;

[0125] The text data includes data of image quality evaluation of the predicted image.

[0126] In one training, the predicted image can be displayed, for example, the predicted image output by the first preset model is displayed on a display interface of an electronic device, so that a user can visually observe the effect of the first preset model on the first image sample, and then text of image quality evaluation of the predicted image can be inputted to obtain text data corresponding to the predicted image.

[0127] The user can input the text data through an input tool, such as typing the text data on an input interface, or the user's voice can be collected through a voice collection module, and then the voice is recognized to obtain the text data, in which case, the user can speak the language of quality evaluation of the predicted image to the voice collection module without manual input.

[0128] The text data can include a sentence for evaluating the predicted image. For example, the text data includes a sentence of "the image is not clear enough", and when the parameters of the first preset model are updated in combination with the text data, the first preset model needs to be optimized in the direction of "the image is clearer". For another example, the text data includes a sentence of "the hair is not clear enough", and when the parameters of the first preset model are updated in combination with the text data, the first preset model needs to be optimized in the direction of "the hair is clearer". For another example, the text data includes a sentence of "the image noise is not completely eliminated", and when the parameters of the first preset model are updated in combination with the text data, the first preset model needs to be optimized in the direction of "completely eliminating the noise".

[0129] In practice, when the parameters of the first preset model are updated according to the text data, the predicted image and the second image sample, since the text data can represent the image quality gap between the predicted image and the second image sample, according to the text data and the predicted image, the first gap between the predicted image and the second image sample can be determined at the user visual angle level, and according to the predicted image and the second image sample, the second gap between them at the pixel level can be determined. When updating the parameters of the first preset model, the first gap and the second gap can be used, so that the first preset model can be optimized not only in the optimization direction provided by the user vision, but also in the optimization direction provided by the second image sample, so that the first preset model can be combined with the provided quality evaluation to obtain the ability to perceive human vision, thereby providing multiple optimization directions for the first preset model, improving the model training method in the image restoration field, and helping to improve the image restoration quality of the first preset model.

[0130] In some embodiments, the text data can include at least one term; the at least one term is used to describe the image quality of the predicted image in different image regions and / or the image quality of different quality dimensions.

[0131] In this embodiment, the text data can include at least one term, and these terms can be different due to different image restoration tasks to be performed by the first preset model, that is, the at least one term corresponds to the image restoration task. Specifically, it can include terms for describing the image quality of different image regions and / or the image quality of different quality dimensions.

[0132] The image region refers to different regions in the predicted image. In image restoration, the restoration effect of different image regions can be different for the same image restoration task, and the difference between different image regions in restoration can be indicated by the text data.

[0133] For example, in the old photo restoration, for the clarity restoration, the faces of two people need to be restored at the same time, one of which is restored clearly, and the other is restored less clearly; for example, for the clarity restoration, when the face of a person needs to be restored and the clothes and accessories of the person also need to be restored, the face of the person may be restored more clearly, but the clothes and accessories of the person may be restored less clearly. Correspondingly, the text data includes the image quality evaluation of the predicted image in different image regions, for example, the words related to the image region in the text data can include: clothes and accessories are not clear enough, face A is not clear enough, and the like.

[0134] The quality dimension is related to the image restoration task performed by the first preset model. One image restoration task can correspond to one quality dimension. For example, if the image restoration task is a clarity restoration task, the quality dimension includes a clarity dimension. If the image restoration task also includes a scratch restoration task, the quality dimension also includes a scratch dimension.

[0135] For example, the image restoration task of the first preset model includes clarity restoration and scratch restoration, and the words related to the quality dimension in the text data can include at least one of: the image is not clear enough, and there are scratches. For example, the image restoration task of the first preset model includes clarity restoration and noise elimination, and the words included in the text data can include at least one of: the image is not clear enough, and there are noises. For example, the image restoration task of the first preset model includes noise elimination and missing part restoration, and the words included in the text data can include at least one of: there are noises, and the missing part is not completely restored. For example, the image restoration task of the first preset model includes clarity restoration and missing part restoration, and the words included in the text data can include at least one of: not clear enough, and the missing part is not completely restored.

[0136] Of course, the above is only an example. In some other examples, other image restoration tasks can also be included, and the text data can include words describing the predicted image in other quality dimensions.

[0137] When the above method is used, the text data can indicate the weak link of the first preset model in the image restoration process. Therefore, when updating the first preset model, the text data can be used to determine the to-be-optimized region in the predicted image that is different from the expected image quality, so that the to-be-optimized region can be fed back to the first preset model to supervise the first preset model to perform reinforcement learning on the restoration of the to-be-optimized region, thereby guiding the first preset model to continuously optimize the weak link of the image restoration.

[0138] In some embodiments, the text data is used to evaluate the image quality of the predicted image, wherein the text data can include text for inversely evaluating the predicted image, which means that the text included is an evaluation opposite to the true image quality of the predicted image. For example, if the face in the predicted image is not clear, the inverse evaluation is that the face is clear. For another example, if the predicted image has a scratch, the inverse evaluation is that there is no scratch.

[0139] Alternatively, the text data can include text for positively evaluating the predicted image, which means that the text included is an evaluation consistent with the true image quality of the predicted image. For example, if the face in the predicted image is not clear, the positive evaluation is that the face is not clear. For another example, if the predicted image has a scratch, the positive evaluation is that there is a scratch.

[0140] In this way, the first preset model can be updated in combination with the text data during the training process, so that the weak link existing in the image repairing process can be determined in combination with the text data. It can be understood that the optimization direction of the model in the image repairing process needs to be determined in combination with the text data. Therefore, whether the text data is a positive evaluation or an inverse evaluation, it can indicate the direction in which the model needs to be optimized. Specifically, the image quality of the predicted image that needs to be improved can be determined according to the text data.

[0141] In this embodiment, the gap between the predicted image and the expected image quality level (which can be understood as the image quality level represented by the second image sample) can be determined based on the text data. This gap reflects the evaluation of the human visual angle on the predicted image, so that the first preset model perceives the evaluation of the human visual angle through the text data. Therefore, the parameters of the first preset model can be updated based on the gap and the pixel difference between the predicted image and the second image sample.

[0142] In this way, the first preset model can be updated in combination with the text data during the training process, so that the weak link existing in the image repairing process can be determined in combination with the text data. It can be understood that the optimization direction of the model in the image repairing process needs to be determined in combination with the text data. Therefore, whether the text data is a positive evaluation or an inverse evaluation, it can indicate the direction in which the model needs to be optimized. Specifically, the image quality of the predicted image that needs to be improved can be determined according to the text data.

[0143] In specific implementation, the similarity between the target text and the predicted image can be determined, and the loss value can be determined based on the predicted image and the second image sample. Then, the parameters of the first preset model can be updated based on the similarity and the loss value. The target text is the text data or a text opposite in semantics to the text data.

[0144] The target text contains a description of the expected image quality that needs to be further optimized. For example, if the background of the predicted image is not clear enough, the target text needs to contain the description of "clear background" to clearly indicate the optimization direction of the image repair.

[0145] In an embodiment, if the text data is positive evaluation data, the text opposite to the semantic of the text data can be determined, and the positive evaluation text data is converted into negative evaluation text data, so that the converted negative evaluation text data can be used as the target text to determine the similarity between the target text and the predicted image. For example, if the hair of the portrait in the predicted image is not clear enough, and the text data contains a positive description of "unclear hair", then "unclear hair" needs to be converted into "clear hair" to clearly indicate the optimization direction of the image repair.

[0146] If the text data is negative evaluation data, the text data can be used as the target text to determine the similarity between the target text and the predicted image.

[0147] In the embodiment, when updating the parameters of the first preset model, the difference between the predicted image and the second image in pixels is also needed, specifically, the loss value can be determined according to the predicted image and the second image sample, specifically, the loss function can be constructed according to the predicted image and the second image sample to obtain the loss value, wherein the loss function can use the loss function shown in the following formula (1) or formula (2):

[0148] Formula (1)

[0149] Formula (2)

[0150] In the above formula (1) and formula (2), C represents the number of channels (RGB image is 3 channels, then C=3, and gray image is single channel, then C=1), H represents the height of the image, and W represents the width of the image. y is the true value image, x is the input image, and f(x) is the output image.

[0151] Since the similarity can reflect the difference between the predicted image and the expected image quality, its value can be 0-1, and the loss value can reflect the difference between the predicted image and the second image sample, its value can also be 0-1. In practice, the similarity can also be regarded as a type of loss value, and then the parameters of the first preset model can be updated according to the two loss values. Specifically, the similarity and the loss value can be respectively preset with weights, and the weighted sum of the two is obtained to obtain the total loss, and the parameters of the first preset model are updated based on the total loss.

[0152] In an example, the weight corresponding to the similarity can be less than the weight corresponding to the loss value, that is, the importance of the first gap in the user perspective level of the predicted image determined based on the text data can be slightly less than the importance of the second gap in the pixel level between the predicted image and the second image sample.

[0153] In the embodiment, since the text data is text type data and the predicted image is image type data, when determining the similarity between the two, the two can be converted into the same feature space for comparison, that is, the text data is converted into target type data, and the predicted image is also converted into target type data, so that the similarity between the two converted into target type data can be determined.

[0154] In an example, the predicted image can be converted into text type data, so that the predicted image can be compared with the text data in the same text space. Specifically, the predicted image can be feature extracted to obtain a feature vector, and then the feature vector is converted into a text vector according to a certain rule, so that the text data can be compared.

[0155] In an example, in the embodiment, when determining the similarity between the target text and the predicted image, the two can be encoded first to obtain vectors of the two respectively after encoding, so that the similarity can be determined based on the distance between the vectors. Specifically, the target text can be encoded to obtain a text feature vector of the target text, and the predicted image can be encoded to obtain an image feature vector of the predicted image; then, the similarity can be determined based on the text feature vector and the image feature vector.

[0156] In the embodiment, the dimensions of the text feature vector and the image feature vector are consistent.

[0157] In the embodiment, when the target text is encoded, the keywords in the target text can be encoded, and the text feature vector obtained by encoding can be a one-dimensional vector; when the predicted image is encoded, the predicted image can be feature extracted first, and then the extracted image features can be encoded, and the one-dimensional image feature vector can also be obtained after encoding; then, the cosine distance between the text feature vector and the image feature vector can be calculated to obtain the similarity between the two.

[0158] Next, the training process of the first preset model is introduced.

[0159] In some examples, the first preset model can be trained in stages. In the process of training the first preset model in stages, two stages can be included. In the first stage, the first preset model can be trained using part of the image sample pairs. The training process can not include text data. After the first stage is completed, the second stage can be entered. In the second stage, the first preset model can be further trained using the remaining part of the image sample pairs. In the training process of the second stage, text data can be added for parameter updating.

[0160] Referring to Figure 3 FIG. 2 shows a process diagram of training the first preset model in stages, as shown in Figure 3 FIG. 2 shows a process diagram of training the first preset model in stages, as shown in

[0161] In the first training, the parameters of the first preset model can be updated based on the predicted image output by the first preset model and the second image sample. In the second training, the parameters of the first preset model obtained in the first training can be updated based on the text data, the predicted image, and the second image sample.

[0162] In the first training, the parameters of the first preset model can be updated based on the predicted image output by the first preset model and the second image sample. In the second training, the parameters of the first preset model obtained in the first training can be updated based on the text data, the predicted image, and the second image sample.

[0163] In some examples, the first preset model can be trained in stages. In the process of training the first preset model in stages, two stages can be included. In the first stage, the first preset model can be trained using part of the image sample pairs. The training process can not include text data. After the first stage is completed, the second stage can be entered. In the second stage, the first preset model can be further trained using the remaining part of the image sample pairs. In the training process of the second stage, text data can be added for parameter updating.

[0164] The prediction image output by the first preset model in the second training can be displayed, so that when there are repeated image samples between the image sample pairs used in the first training and the image sample pairs used in the second training, the prediction image corresponding to the same first image sample in the first training and the prediction image corresponding to the same first image sample in the second training can be displayed, to facilitate the user to determine whether the same image is more optimally repaired in the second training.

[0165] For example, when the prediction image corresponding to the first image sample is output in the second training, it can be determined whether the first image sample is input to the first preset model in the first training. If yes, the prediction image corresponding to the first image sample in the first training is also output, so that the prediction images corresponding to the first image sample in the first training and the second training can be displayed on the display interface at the same time, so that whether the first preset model is optimized in the second training can be determined by comparing the two prediction images.

[0166] In the first training, since text data is not needed to be added for supervision, it can be understood as pre-training of the first preset model. When the pre-training meets the end condition, the first training can be stopped. The end condition can be that the difference between the prediction image and the second image sample is small, such as the loss value is less than the preset loss threshold. Of course, in order to achieve that the first training can be completed as soon as possible to achieve the goal of pre-training, the loss threshold can be set to be larger, such as to achieve the basic image repairing capability, the first training can be ended.

[0167] The end condition can also be triggered by the user. As described above, in the first training process, the first preset model can output the prediction image corresponding to the first image sample, and the user can determine the image repairing capability of the first preset model through the prediction image, so that the user can determine when to end the first training.

[0168] In some embodiments, since it is needed to determine the gap between the prediction image and the expected image quality level (the image quality level represented by the second image sample) based on the text data, in practice, it is needed to determine the similarity between the target text and the prediction image. In some examples, the similarity between the target text and the prediction image can be determined through a neural network, that is, by using deep learning technology, the second preset model can be trained to obtain the text feature vector of the target text and the image feature vector of the prediction image, and then the similarity between the target text and the prediction image is determined.

[0169] Specifically, in the process of training the first preset model in stages, a stage of training the second preset model can also be included. The second preset model can be a CLIP (Contrastive Language-Image Pre-Training) model structure, which can include an image encoder for encoding the predicted image and a text encoder for encoding the text data.

[0170] Referring to Figure 4 and Figure 5 , it is shown that the process of adding a training stage of the second preset model in the training process of the first preset model, Figure 4 a process diagram of training the second preset model is shown, Figure 5 a step flow diagram of training the second preset model is shown, as Figure 4 and Figure 5 , before the first training or the second training of the first preset model, the following steps can also be included:

[0171] Step S501: Obtain a plurality of third image samples and text data samples corresponding to the third image samples;

[0172] Step S502: Based on the plurality of third image samples and the corresponding text data samples, the second preset model is third trained;

[0173] The second preset model at the completion of the third training is used to determine the similarity between the predicted image and the text data in the training of the first preset model.

[0174] Correspondingly, as Figure 4 shown, when updating the parameters of the first preset model based on the text data, the predicted image and the second image sample, the predicted image and the text data can be input to the second preset model at the completion of the third training; and based on the similarity output by the second preset model, the predicted image and the second image sample, the parameters of the first preset model are updated.

[0175] In this embodiment, the third image samples can be different from the above-mentioned image sample pairs, or can include the image samples in the above-mentioned image sample pairs, and can also include the predicted image output at the first training. In practice, the third image samples can not be required to have a connection with the above-mentioned image sample pairs, such as the image sample pairs being portrait photos, the third image samples can not be limited to portrait photos, but can also be animal images, etc., thereby reducing the difficulty of obtaining image samples.

[0176] Wherein, each third image sample corresponds to a text data sample, as described above, the text data can include text data generated by forward evaluation of the predicted image, and text data generated by reverse evaluation of the predicted image, in some examples, in the third training, the text data sample corresponding to the third image sample can be data generated by forward evaluation of the third image sample, or data generated by reverse evaluation of the third image sample, or both.

[0177] Wherein, the text data sample can also include at least one word; the at least one word is used to describe the image quality of the third image sample in different image regions, and / or the image quality of different quality dimensions.

[0178] In this case, when labeling the text data sample corresponding to the third image sample, the image restoration task of the first preset model can be used for labeling, for example, if the image restoration task is a clarity restoration task, the word in the quality dimension of the labeled text data sample can include the word in the clarity dimension, such as whether each image region is clear.

[0179] Taking forward evaluation as an example, the image restoration task of the first preset model includes clarity restoration and noise elimination, if the image quality of the third image sample is not high (not clear enough, with noise, with scratches or part of the region is not restored), the word included in the text data sample can include at least one of: the image is not clear enough, and there is noise; for example, the image restoration task of the first preset model includes noise elimination and missing part restoration, the word included in the text data sample can include at least one of: there is noise, and the missing part is not completely restored; for example, the image restoration task of the first preset model includes clarity restoration and missing part restoration, the word included in the text data sample can include at least one of: not clear enough, and the missing part is not completely restored.

[0180] Taking reverse evaluation as an example, the image restoration task of the first preset model includes clarity restoration and noise elimination, if the image quality of the third image sample is not high (not clear enough, with noise, with scratches or part of the region is not restored), the word included in the text data sample can include at least one of: the image is clear enough, and there is no noise; for example, the image restoration task of the first preset model includes noise elimination and missing part restoration, the word included in the text data sample can include at least one of: there is no noise, and the missing part is completely restored; for example, the image restoration task of the first preset model includes clarity restoration and missing part restoration, the word included in the text data sample can include at least one of: clear enough, and the missing part is completely restored.

[0181] The image region refers to different regions in the third image sample, and the text data sample in the image region can also be related to the image restoration task to be performed by the first preset model.

[0182] Taking a positive evaluation as an example, for the clarity restoration task, the third image sample can include multiple different image regions. If the third image sample is not clear in image region 1 and image region 2, the text data sample can include the following terms: image region 1 (such as clothing accessories) is not clear, image region 2 (such as face A) is not clear, and the like.

[0183] Taking a negative evaluation as an example, for the clarity restoration task, the third image sample can include multiple different image regions. If the third image sample is not clear in image region 1 and image region 2, the text data sample can include the following terms: image region 1 (such as clothing accessories) is clear, image region 2 (such as face A) is clear, and the like.

[0184] By using this text data sample labeling method, the text data sample labeled for the third image sample is related to the image restoration task of the first preset model. In this way, the second preset model can be optimized in the third training towards the similarity determination method required by the image restoration task of the first preset model. The second preset model can extract the image feature vector of the third image sample and the text feature vector of the text data sample in a direction related to the image restoration task, for example, to fully extract the features reflecting the text and image in the image restoration task as the optimization direction, and optimize the second preset model. In this way, the second preset model can match the needs of the first preset model, thereby providing the first preset model with a higher-accuracy prediction of the comparison result between the image and the text data.

[0185] In some embodiments, in the third training, the second preset model needs to be trained based on multiple third image samples and text data samples. Since the second preset model is used to determine the similarity between the third image sample and the text data sample, the parameters of the second preset model can be updated based on the similarity output by the second preset model and the true similarity between the third image sample and the text data sample. Figure 4 As shown in the figure, specifically, the text data sample carries a category label, which is used to indicate whether the description of the text data sample conforms to the image quality of the third image sample. In this way, in the third training, the third image sample and the text data sample can be input into the second preset model to obtain the predicted similarity between the third image sample and each of the text data samples output by the second preset model. Then, the parameters of the second preset model can be updated based on the predicted similarity and the category label.

[0186] In this embodiment, the category label carried by the text data sample is used to indicate whether the description of the text data sample conforms to the image quality of the third image sample. Specifically, if the text data sample conforms to the image quality of the third image sample, it is positively evaluated data, and its category label can be 1; if the text data sample does not conform to the image quality of the third image sample, it is negatively evaluated data, and its category label can be 0.

[0187] For example, taking the first image sample as an example, assuming that the repair task is sharpness repair, the sharpness of the first image sample is poor, if the text data sample is "image clear", it is negatively evaluated data, and its category label can be 0, if the text data sample is "image not clear", it is positively evaluated data, and its category label can be 1. Taking the second image sample as an example, assuming that the repair task is sharpness repair, the sharpness of the second image sample is high, if the text data sample is "image clear", it is positively evaluated data, and its category label can be 1, if the text data sample is "image not clear", it is negatively evaluated data, and its category label can be 0.

[0188] When updating the parameters of the second preset model based on the predicted similarity and the category label, the loss corresponding to the text data sample can be determined based on the predicted similarity and the category label, and then the parameters of the second preset model can be updated according to the loss. Specifically, the parameters corresponding to the text encoder and the image encoder in the second preset model can be updated.

[0189] After multiple parameter updates, the second preset model can align the image and the text, so that they are compared in the same feature space, and then an accurate comparison result is generated.

[0190] For example, there are at least two text data samples corresponding to at least one third image sample, and the at least two text data samples include text data samples of a first category and text data samples of a second category, the first category representing that the description of the text data sample conforms to the image quality of the third image sample, and the second category representing that the description of the text data sample does not conform to the image quality of the third image sample.

[0191] For example, referring to Figure 6 As shown in FIG. 6, a schematic diagram of the third image sample input to the second preset model is shown, as shown in Figure 6 The text data sample of the first category can be understood as being positively evaluated for the third image sample, and its category label can be 1, representing that the two are matched; the text data sample of the second category can be understood as being negatively evaluated for the third image sample, and its category label can be 0, representing that the third image sample and the text data sample are not matched.

[0192] Specifically, each third image sample can correspond to two text data samples, or part of the third image samples correspond to two text data samples, and the rest of the third image samples correspond to one text data sample; for example, including 500 third image samples, of which 200 third image samples correspond to two text data samples, the first category of text data samples and the second category of text data samples; the remaining 200 third image samples correspond to the first category of text data samples, and the remaining 100 third image samples correspond to the second category of text data samples.

[0193] Wherein, in the case that the third image sample corresponds to at least two text data samples, one of the text data samples can be automatically generated based on the other, for example, the first category of text data samples can be generated based on the second category of text data samples, or the second category of text data samples can be generated based on the first category of text data samples.

[0194] Correspondingly, referring to Figure 7 It is shown that the acquisition process of the text data sample corresponding to the third image sample is shown, as Figure 7 As shown in the figure, the method can specifically include the following steps:

[0195] Step S701: acquiring first text data corresponding to the third image sample;

[0196] Step S702: determining the category corresponding to the first text data, the category being used to indicate whether the description of the first text data is consistent with the image quality of the third image sample;

[0197] Step S703: generating second text data based on the first text data; wherein the category of the second text data is different from the category of the first text data;

[0198] Step S704: taking the first text data and the second text data as the text data sample corresponding to the third image sample.

[0199] Wherein, the first text data is input by the user, which can be the first category of data or the second category of data; then, the category corresponding to the first text data can be determined, wherein the category to which the first text data belongs can be determined by detecting whether the first text data contains a target keyword, the target keyword can be "no", "noise exists", "scratch exists" and the like, which are related to the image restoration task and have negative meaning.

[0200] After determining the category corresponding to the first text data, second text data opposite to the category can be generated. For example, the category corresponding to the first text data is a first category, indicating that the image quality of the third image sample is consistent, and the category corresponding to the second text data is a second category, indicating that the image quality of the third image sample is inconsistent. For another example, the category corresponding to the first text data is the second category, and the category corresponding to the second text data is the first category.

[0201] The manner of generating the second text data can be that if the first text data is the first category, the words with negative meanings in the first text data can be converted into words with positive meanings, so as to obtain the second text data. For example, the word "not" in the first text data is removed, and the word "existence of noise points" in the first text data is modified into the word "nonexistence of noise points".

[0202] If the first text data is the second category, the words with positive meanings in the first text data can be converted into words with negative meanings, so as to obtain the second text data. For example, the word "not" is added in the first text data, so that the word "clear hair" becomes the word "unclear hair". For another example, the word "nonexistence of noise points" in the first text data is modified into the word "existence of noise points".

[0203] In this way, for the first third image sample, although only one category of text data sample is labeled, another category of text data sample can be generated based on the labeled text data sample, so that the third image sample can correspond to two categories of text data samples.

[0204] Correspondingly, referring to FIG. 6, a schematic diagram of a model structure of a second preset model is shown. The second preset model can include a text encoder and an image encoder, and a similarity determination module connected with the text encoder and the image encoder respectively. Figure 8

[0205] The text encoder is configured to perform text encoding on the text data sample to obtain a predicted text vector.

[0206] The image encoder is configured to perform image encoding on the third image sample to obtain a predicted image vector. The predicted image vector and the predicted text vector have consistent dimensions.

[0207] The similarity determination module is configured to determine a predicted similarity between the predicted image vector and the predicted text vector.

[0208] ​In this embodiment, during the third training, the third image sample can be input into the image encoder, and the corresponding text data sample can be input into the text encoder. If the third image sample corresponds to two types of text data samples, then both text data samples corresponding to the third image sample can be input into the text encoder. The image encoder can encode the third image sample to obtain a predicted image vector, and the text encoder can encode the text data sample to obtain a predicted text vector. The similarity determination module can calculate the cosine distance between the predicted text vector and the predicted image vector to obtain the predicted similarity between the two.

[0209] In the case where the third image sample corresponds to two types of text data samples, the text encoder can output the predicted text vectors corresponding to the two types of text data samples respectively. Then, the similarity determination module can obtain two predicted similarities, which correspond to the two types of text data samples respectively.

[0210] Among them, reference Figure 9 As shown, a schematic diagram illustrating the complete process of the first, second, and third training sessions is presented. Figure 9 As shown, the third training can be performed before the first training or before the second training. That is, the second preset model can be trained before the first training begins, or the third training can be performed after the first training ends and before the second training begins.

[0211] For example, the third training can be performed after the first training has ended and before the second training begins. The third image samples used in this training may overlap with the image sample pairs used in the first training process, and / or overlap with the predicted images output by the first preset model in the first training process. Specifically, the multiple third image samples may include at least one of the following: the first image samples, the second image samples, and the predicted images output by the first preset model before the current time.

[0212] Exemplarily, the plurality of third image samples can include all or part of the first image samples in the plurality of image sample pairs, such as the first image samples used in the first training process; or the plurality of third image samples can include all or part of the second image samples in the plurality of image sample pairs, such as the second image samples used in the first training process. Alternatively, the plurality of third image samples can include the predicted image output by the first preset model for the first image sample in the first training; or the plurality of third image samples can include the first image sample and the second image sample, so that the third training and the first training can share the image samples; or the plurality of third image samples can include the first image sample and the predicted image; or the plurality of third image samples can include the second image sample and the predicted image; or the plurality of third image samples can include the first image sample, the second image sample, and the predicted image.

[0213] In the case where the predicted image output by the first preset model in the first training is included in the plurality of third image samples, the image sample pairs used in the second training can include the image sample pairs used in the first training. With this sample setting, since the third image sample needs to correspond to the text data sample, the text data sample needs to be labeled for the predicted image. Since the labeled text data sample is used to evaluate the quality of the predicted image, in practice, when the second training is performed, the text data for evaluating the predicted image also needs to be obtained. Therefore, the text data sample labeled for the third image sample in the third training can be directly used as the text data for the predicted image in the second training.

[0214] Exemplarily, the first image sample A1 corresponds to the predicted image A1 in the first training, and the predicted image A1 is used as the third image sample. The first image sample A1 corresponds to at least one text data sample T1. Then, the first image sample A1 participates in the second training and corresponds to the predicted image A2. Since the predicted image A2 and the predicted image A1 have little difference in image quality when the second training starts, the text data sample T1 corresponding to the predicted image A1 can be used as the text data corresponding to the predicted image A2, so as to participate in the parameter update of the first preset model.

[0215] When the third image sample includes the predicted image corresponding to the first image sample in the first training, the utilization of the labeled text data sample can be improved, and the training efficiency can be improved.

[0216] Exemplarily, the first preset model can be connected with the second preset model to form a new model, and the first training, the second training and the third training can be performed based on the new model. Specifically, when the first training and the second training are performed, the parameters of the second preset model can be fixed, and the parameters of the first preset model can be updated; when the third training is performed, the parameters of the first preset model can be fixed, and the parameters of the second preset model can be updated.

[0217] In the case that the third image sample comprises the predicted image, the labeling of the text data sample can be performed along with the first training, that is, the predicted image output by the first preset model during the first training can be used both in the parameter updating of the first preset model and in the labeling of the text data sample. For example, during the first training, the predicted image is associated with the text data sample, so that the third image sample used in the third training can be collected during the first training.

[0218] Exemplarily, the third training can be performed after the first training is completed and before the second training is started, so that the third training can be performed in the interval of the training of the first preset model. As described above, in the case that the third image sample comprises the predicted image output by the first preset model for the first image sample, the text data sample corresponding to the predicted image can be used again in the second training of the first preset model.

[0219] In the implementation, refer to Figure 10 , another process diagram for training the second preset model is shown, as Figure 10 shown, when the third training of the second preset model is performed based on the plurality of third image samples and the text data samples, the plurality of first image samples can be input to the first preset model; and the predicted image output by the first preset model and the at least two kinds of text data samples corresponding to the predicted image can be input to the second preset model to perform the third training of the second preset model; wherein, in the third training, the parameters of the first preset model are fixed.

[0220] In this embodiment, the third training can be performed in the first training. In this case, the predicted image output by the first preset model and the at least two kinds of text data samples corresponding to the predicted image can be input to the second preset model at the beginning of the first training, and then the parameters of the first preset model are updated according to the predicted image and the second image sample; and the parameters of the second preset model are updated according to the category label corresponding to the text data sample and the similarity output by the second preset model.

[0221] The process of updating the parameters of the first preset model and the parameters of the second preset model can be independent of each other, so that when the parameters of the first preset model are updated, the parameters of the second preset model are fixed; when the parameters of the second preset model are updated, the parameters of the first preset model are fixed.

[0222] In this way, the third training can be started in parallel in the process of the first training, so that the training progress can be improved.

[0223] Correspondingly, the third image sample can also include the first image sample, and further, in addition to inputting the predicted image output by the first preset model and the at least two text data samples corresponding to the predicted image into the second preset model, the first image sample and the text data sample corresponding to the first image sample can also be input into the second preset model when the first image sample is input into the first preset model.

[0224] In this way, in one first training, the first image sample and the predicted image corresponding to the first image sample can be input into the second preset model for training, so that when the first preset model is trained based on n first image samples, the second preset model can be trained based on 2n third image samples, thereby accelerating the training progress of the second preset model.

[0225] When the first training is completed, the third training can end or continue the third training according to the optimization effect, when the third training needs to continue, the parameters of the first preset model are first fixed, and then the remaining third image samples and the corresponding text data samples are input into the second preset model for retraining.

[0226] In some embodiments, the predicted image input into the second preset model for training can be the predicted image output by the first preset model when the first training is to be completed, so that the image quality difference between the predicted image for the third training and the predicted image output in the second training for the same first image sample is small, thereby the text data sample corresponding to the predicted image can be reused.

[0227] For example, the predicted images corresponding to the preset number of first image samples when the first training is to be completed are input into the second preset model for training, and the preset number of first image samples are also input into the first preset model in the second training, and the difference between the predicted image output in the second training and the predicted image input into the second preset model is small, therefore, the text data sample can be reused as the text data corresponding to the predicted image in the second training to update the parameters of the first preset model.

[0228] Thus, when the first preset model is trained by using the partial image samples, the predicted images output during the first training can be saved, and then the third training is started. During the third training, a plurality of third image samples can be obtained, wherein the plurality of third image samples can include the plurality of predicted images output during the first training, and a text data sample is labeled for each third image sample. In this way, the plurality of predicted images also have respective corresponding text data samples.

[0229] For example, after the third training, the second training needs to be started, and in the second training, the similarity between the predicted image and the text data needs to be determined by using the second preset model. In the second training, the plurality of first image samples can be input to the first preset model, and the predicted image output by the first preset model and the text data can be input to the second preset model, and the parameters of the first preset model are updated based on the predicted image, the second image sample, and the predicted similarity output by the second preset model.

[0230] In the updating of the parameters of the first preset model, the parameters of the second preset model are fixed.

[0231] In this example, after the training of the second preset model is completed, the parameters of the second preset model can be fixed, so that in the second training, the predicted image output by the first preset model and the text data corresponding to the predicted image can be input to the second preset model, and the second preset model outputs the similarity between the predicted image and the text data. The similarity can be used together with the loss value determined by the predicted image and the second image sample to participate in the updating of the first preset model.

[0232] For example, whether the third image sample contains the predicted image output by the first preset model in the first training process, in the second training, since the text data corresponding to the predicted image needs to be obtained, when the text data corresponding to the predicted image currently output by the first preset model is obtained, it can be determined from the plurality of third image samples whether there is a target third image sample corresponding to the predicted image. If there is, the text data sample corresponding to the target third image sample is used as the text data; if there is not, the text data input for the predicted image is obtained.

[0233] The target third image sample and the predicted image correspond to the same first image sample.

[0234] In this example, in the second training, the text data corresponding to the predicted image output by the first preset model needs to be obtained, and if the predicted image has been trained by the second preset model as a third image sample, it has a text data sample. In practice, the labeled text data sample is directly used as the text data corresponding to the predicted image.

[0235] In practice, for the same first image sample, the prediction image output by the first preset model in the first training and the prediction image output by the first preset model in the second training can have differences, and therefore, when searching for the target third image sample, the identification of the first image sample associated with the prediction image in the third training and the identification of the first image sample input to the first preset model in the current second training can be used as a basis to search for whether the prediction image output by the first preset model in the current second training can use the text data sample corresponding to the prediction image used in the third training, that is, to search for the target third image sample, and if the target third image sample is found, the text data sample corresponding to the target third image sample is used as the text data corresponding to the prediction image output by the first preset model in the current second training.

[0236] As shown in Figure 6 , in the first training, the first image sample A is input to the first preset model to obtain the prediction image A1, the prediction image A1 is labeled with the text data sample 3 and the text data sample 4, and the prediction image A1, the text data sample 3 and the text data sample 4 are input to the second preset model as training samples for third training; when the second training comes, the first image sample A is input to the first preset model again and outputs the prediction image A2, because the prediction image A1 and the prediction image A2 correspond to the same first image sample A, the text data sample 3 and the text data sample 4 can be input to the second preset model as text data corresponding to the prediction image A2 to calculate the similarity.

[0237] It should be noted that if the target third image sample is associated with multiple text data samples, the text data sample of the reverse evaluation can be used as the text data corresponding to the prediction image. If the text data sample associated with the target third image sample is a text data sample of forward evaluation, the text data sample can be converted into a text data sample of reverse evaluation, and the converted text data sample can be used as the text data corresponding to the prediction image.

[0238] If the currently output prediction image has not been trained on the second preset model as a third image sample, the text data corresponding to the prediction image needs to be reacquired, because the text data can be input by the user, the prediction image needs to be displayed for the user to observe, specifically, the prediction image can be displayed on a display interface, and the text data input by the user for the prediction image can be acquired.

[0239] Of course, even if the predicted image is used as a third image sample in the third training, it has a text data sample, but for the same first image sample, the predicted image output by the first preset model in the first training and the predicted image output in the second training can have a large difference. For example, the first image sample B input to the first preset model at the beginning of the first training, the corresponding predicted image B1 is input to the second preset model, however, as the first training deepens, the image restoration quality of the first preset model will be optimized, then in the second training, if the first image sample B is input to the first preset model after the first training ends, the predicted image B2 output by the first preset model has a large difference from the predicted image B1, then the text data sample corresponding to the predicted image B2 is no longer suitable for use.

[0240] In this case, the text data sample corresponding to the predicted image can not be used as the text data for updating the parameters of the first preset model, but the predicted image can be displayed, so that the user re-enters the text data for the predicted image.

[0241] As described above, since the first training and the second training can be performed in stages for the first preset model, and the third training can be performed before the first training or between the first training and the second training, the third image sample used in the third training can include the predicted image output in the first training, so that the predicted image output in the training process can be displayed in real time during the first training, the second training and the third training, and the text data input by the user can be listened to during the display of the predicted image, so that the entire training stage can be visualized by the user.

[0242] In an example, when obtaining the text data corresponding to the predicted image currently output by the first preset model, the predicted image can be displayed, and the text data input for the predicted image can be obtained.

[0243] In this example, the predicted image output by the first preset model during the training can be displayed in response to the training of the first preset model, specifically, the predicted image output by the first preset model can be displayed in the first training or the second training, so that the user can input the text data for the predicted image. As described above, the way to input the text data can be to input on the display interface displaying the predicted image using an input tool, or to input by voice.

[0244] In another example, after displaying the predicted image, an input operation on the operation interface where the predicted image is displayed can also be monitored; if no input operation is monitored, preset text data is taken as the text data corresponding to the predicted image; accordingly, when obtaining the text data input for the predicted image, the text data input for the predicted image can be obtained in the case where the input operation is monitored.

[0245] In this embodiment, the interface where the predicted image is displayed can be referred to as an operation interface, in which the user can input text data through an input operation, which can be an operation of typing text in the operation interface, or can be an operation of clicking a voice collection control in the operation interface, in which case the device starts to collect voice data and starts to recognize the voice data to obtain text data.

[0246] In this embodiment, when the predicted image is displayed, a preset time length countdown can be started, and whether an input operation is received during the countdown can be monitored; if not, the preset text data can be taken as the text data corresponding to the predicted image. The preset text data can be text data for inversely evaluating the predicted image, and specifically, the preset text data can be consistent with the image quality level of the second image sample, so that the process of inputting text data by the user can be omitted.

[0247] In an example, in the first training and / or the second training, the output predicted image can be displayed so that the user can observe whether the predicted image meets the expectation, and if so, the user can specify whether the training is completed. In addition, during the first training, the predicted image output by the first preset model can be displayed, and the first training can be ended in response to a first preset operation performed on the predicted image; or during the second training, the predicted image output by the first preset model can be displayed, and the second training can be ended in response to a second preset operation performed on the predicted image; or during the first training and the second training, the output predicted image can be displayed so that the user can end the first training and the second training in a timely manner according to the human eye observation of the predicted image, thereby improving the model training efficiency.

[0248] The first preset operation can be an operation of clicking an "end control" of the first training, and the second preset operation can be an operation of clicking an "end control" of the second training; or the first preset operation and the second preset operation can be an operation of double-clicking or right-clicking a mouse in the operation interface, which will not be described herein.

[0249] In a case where the first preset operation is detected, the first training of the first preset model can be stopped, and the parameters of the first preset model can be fixed.

[0250] In another example, during the third training of the second preset model, a loss value corresponding to at least one training of the second preset model before the current time can also be displayed; and in response to a third preset operation performed on each of the displayed loss values, the third training is ended; wherein the gradient is determined by the predicted similarity and the category label.

[0251] In this example, in response to the start of the third training, a loss value used for gradient update at each time during the third training can be displayed, and the loss value can be determined according to the predicted similarity and the category label corresponding to the text data sample. In this way, the optimization process of the second preset model can be determined according to the displayed loss value, as shown in FIG. 6. Figure 11 As shown in FIG. 6, a loss value change trend graph of the second preset model during the third training is output, as shown in FIG. 6. Figure 11 As shown in FIG. 6, as the third training proceeds, the loss value becomes smaller and smaller. When the change curve of the loss value indicates that the loss value converges, the third training can be ended.

[0252] The third preset operation is as described above in the first preset operation and the second preset operation, and will not be described again here. In response to the third preset operation, the parameters of the second preset model can be fixed, and the input end of the second preset model can be connected to the output end of the first preset model. Specifically, the input end of the image encoder in the second preset model can be connected to the output end of the first preset model.

[0253] When this implementation is adopted, the training processes of the first preset model and the second preset model can be visualized, so that in the case of training the first preset model and the second preset model in stages, the user can actively control the starting time and the ending time of each training stage, thereby optimizing the model training process.

[0254] As described above, in the case of including the first preset model and the second preset model, the output end of the first preset model can be connected to the input end of the second preset model, and the two models constitute a target model, which can be referred to as a model to be trained. In practice, the target model can be trained in stages using the image sample pairs, the plurality of third image samples, and the text data samples corresponding to the third image samples, to obtain the trained first preset model and the trained second preset model. The trained first preset model can be used as an image inpainting model, and the trained second preset model can be used in a text and image spatial alignment task.

[0255] wherein, in the training of the target model, the following process is included:

[0256] The first stage of training (first training) process: input the first image sample in the image sample pair to the first preset model, calculate the pixel difference between the predicted image output by the first preset model and the second image sample corresponding to the first image sample, i.e. obtain the loss value, update the parameters of the first preset model according to the loss value, and no processing is performed on the second preset model in this process, i.e. the predicted image is not input to the second preset model for training.

[0257] The second stage of training (third training) process: fix the parameters of the first preset model, which mainly includes the following training methods:

[0258] input the first image sample and the text data sample corresponding to the first image sample to the first preset model and the second preset model, input the second image sample and the text data sample corresponding to the second image sample to the second preset model, and simultaneously input the predicted image output by the first preset model and the text data sample corresponding to the predicted image to the second preset model, the second preset model outputs the predicted similarity between the image sample and the text data sample, then according to the predicted similarity and the category label corresponding to the text data sample, the loss value corresponding to the second preset model is determined, and then the parameters of the second preset model are updated;

[0259] input the first image sample to the first preset model, input the predicted image output by the first preset model and the text data sample corresponding to the predicted image to the second preset model, the second preset model outputs the predicted similarity between the image sample and the text data sample, then according to the predicted similarity and the category label corresponding to the text data sample, the loss value corresponding to the second preset model is determined, and then the parameters of the second preset model are updated;

[0260] input a new image sample different from the first image sample, the second image sample and the predicted image, and the text data sample corresponding to the new image sample to the second preset model, the second preset model outputs the predicted similarity between the image sample and the text data sample, then according to the predicted similarity and the category label corresponding to the text data sample, the loss value corresponding to the second preset model is determined, and then the parameters of the second preset model are updated.

[0261] The first image sample, the second image sample, the predicted image sample and the new image sample described above are collectively referred to as third image samples in the third training.

[0262] After the second stage of training is completed, the third stage of training process (second training) is started:

[0263] With the parameters of the second preset model fixed, the first image sample is input into the first preset model, the predicted image output by the first preset model and the text data generated by the user's reverse evaluation of the predicted image are input into the second preset model, and the second preset model outputs the similarity between the predicted image and the text data.

[0264] Next, the loss value between the predicted image and the second image sample is calculated, and the parameters of the first preset model are updated based on the similarity and the loss value.

[0265] Once the training in the third stage is completed, the first preset model can be used as the image restoration model.

[0266] Below, refer to Figure 12 As shown, an exemplary schematic diagram of a model training method is illustrated, such as... Figure 12 As shown, taking portrait photo restoration as an example, it is necessary to train an image restoration model for portrait photo restoration. The training method of this model includes the following steps:

[0267] S11: Prepare multiple high-resolution portrait photos, take photos of the multiple portrait photos, and use the pre-processed images as second image samples. Then, blur the multiple portrait photos, such as by scratching or adding noise to the portrait photos. Take photos of the blurred portrait photos, and use the pre-processed images as first image samples. The first image samples and second image samples belonging to the same portrait photo are combined into an image sample pair.

[0268] In this example, there can be 1000 image sample pairs.

[0269] S12: Constructing a model, including constructing a first preset model and a second preset model; wherein, the first preset model may adopt the DDPM model (Diffusion Models Beat GANs on Image Synthesis) or CNN model in related technologies; the first preset model is used to repair the input image, and the repair includes: improving the image sharpness and eliminating scratches and noise; the second preset model may include an image encoder and a text encoder, as well as a similarity determination module connected to the image encoder and the text encoder respectively;

[0270] The second preset model can adopt a mature CLIP model structure, including two parts, a text encoder (Text Encoder) and an image encoder (Image Encoder); the image encoder is used to encode image feature information of the input image, and outputs an image feature vector; the text encoder is used to encode text feature information of the input text, and outputs a text feature vector.

[0271] The second preset model is connected to the output end of the first preset model, and in the subsequent training process, the model composed of the first preset model and the second preset model can be trained separately.

[0272] S13: First training, when the first training is performed, the connection between the first preset model and the second preset model is disconnected, in the first training, the first image sample in the plurality of image sample pairs is input to the first preset model, the predicted image after the first image sample is repaired by the first preset model is displayed, and a loss function is constructed according to the predicted image and the corresponding second image sample, so as to update the parameters of the first preset model; it should be noted that the second image sample and the predicted image used to construct the loss function are for the same first image sample; wherein the loss function can adopt the function described in the following formula (1) or formula (2):

[0273] Formula (1)

[0274] Formula (2)

[0275] In the above formula (1) and formula (2), C represents the number of channels (RGB image is 3 channels, then C=3, and gray image is single channel, then C=1), H represents the height of the image, and W represents the width of the image. Y is the true value image, x is the input image, and f(x) is the output image.

[0276] S14: In the first training process, the user can observe the repair optimization process of the first preset model for the first image sample through the displayed predicted image, for example, the higher the quality of the predicted image, the better the preliminary image repair capability of the first preset model, for example, if the image quality of the predicted image is low, the first image sample is continuously input to optimize the first preset model; wherein when it is detected that the user has performed a first preset operation on the displayed predicted image, such as clicking the corresponding control on the operation interface, the first training stops, at this time, the parameters of the first preset model are fixed, and the third training is started;

[0277] Wherein, it is assumed that the first image sample input to the first preset model is 340 when the first training stops;

[0278] In the first training, the predicted image corresponding to each first image sample can be saved, so as to be used as a third image sample in subsequent third training, and the text data sample is labeled for the third image sample.

[0279] S15: constructing new third image samples, which can be obtained by image acquisition on the old blurred photos and the new clear photos, and a preset number of predicted images at the end of the first training can be used as the third image samples, assuming that 40 predicted images input into the first preset model in the first training process are used as the third image samples;

[0280] The third image samples are assumed to be 500, which are less than the number of the first image samples in the plurality of image sample pairs;

[0281] The text data sample is labeled for each third image sample. When a user clicks a third image sample, an input area can be popped up on the operation interface, which is used to input the text data sample. The input text data sample can include the text for positive evaluation of the third image sample. For example, for the third image sample of the old photo shooting, the picture is relatively blurred, and the input text can be: the background is blurred; the edge has sawtooth; the noise is a little large; the eyes are unnatural; the hair is not clear; and the like. For the third image sample of the new photo shooting, the picture is relatively clear, and the input text can be: the background is clear; the noise is small; the hair is clear; and the like.

[0282] Of course, for the same third image sample, the text for negative evaluation of the third image sample can also be included. For example, for the third image sample of the old photo shooting, the picture is relatively blurred, and the input text can be: the background is clear; the noise is small; the hair is clear; and the like. For the third image sample of the new photo shooting, the picture is relatively clear, and the input text can be: the background is blurred; the edge has sawtooth; the noise is a little large; the eyes are unnatural; the hair is not clear; and the like.

[0283] In this example, each third image sample corresponds to the first category of text data samples and the second category of text data samples;

[0284] Each text data sample corresponding to the third image sample carries a category label, which is used to represent the real similarity between the text data sample and the image quality of the third image sample. For example, the category label of the first category of text data samples can be set to 1, representing that the text data sample is a positive evaluation, and the similarity is 1. For the second category of text data samples, the category label can be set to 0, representing that the text data sample is a negative evaluation, and the similarity is 0.

[0285] S16: starting the third training, such as Figure 8The third image sample and the text data sample corresponding to the third image sample are input into the second preset model, wherein the third image sample is input into an image encoder, and the text data sample is input into a text encoder, and the similarity determination module can determine the similarity between the image feature vector output by the image encoder and the text feature vector output by the text encoder;

[0286] Then, based on the similarity and the category label carried by the text data sample, a loss value for updating the parameters of the second preset model is determined; it should be noted that in the third training, the parameters of the first preset model are fixed and unchanged;

[0287] In the third training, the loss value of the second preset model in the training process can be displayed, so that the user can observe whether the third training needs to be completed according to the displayed loss value. If the third training needs to be completed, the corresponding control can be triggered to end the third training and start the second training.

[0288] S17: Start the second training. In this training process, the output end of the first preset model is connected to the input end of the second preset model;

[0289] S171: Input the first image sample in the 1000 image sample pairs into the first preset model. Since the third image sample used when the second preset model is trained includes 40 predicted images output by the first preset model in the first training, 40 first image samples corresponding to the 40 predicted images can be input into the first preset model first to use the text data samples labeled in the third training.

[0290] The quality difference between the predicted image output by the first preset model and the predicted image output in the first training is small, so the text data corresponding to the 40 predicted images can be used as text data. Specifically, when the 40 first image samples corresponding to the 40 predicted images are input into the first preset model, the text data samples corresponding to the 40 predicted images are input into the second preset model, and the predicted image output by the first preset model is also input into the second preset model, so that the second preset model can output the similarity. The loss value between the predicted image and the second image sample is calculated by formula (1) or formula (2). Then, according to the loss value and the similarity, the parameters of the first preset model are updated.

[0291] After the 40 first image samples corresponding to the 40 predicted images are input into the first preset model for training, the remaining 960 image sample pairs in the 1000 image sample pairs can be used to continue training the first preset model.

[0292] S172: Continue the training process:

[0293] The first image sample in the 960 image sample pairs is input to the first preset model, and a predicted image output by the first preset model can be input to the second preset model and displayed in the display area of the front-end operation interface;

[0294] The operation interface also has an input area parallel to the display area, and the user inputs text data corresponding to the displayed predicted image in the input area. In the second training, the input text data is the data of the reverse evaluation. For example, if the current screen human feedback is that the hair is not clear, the input text data is that the hair is clear. In this way, the image feature vector of the predicted image needs to be close to the text semantic vector of "clear hair", so as to stimulate the first preset model to train in the direction of clear hair. If the user does not input text in the input area, the default preset text of the reverse evaluation is used as the text data.

[0295] The text data input by the input area is input to the second preset model, and the second preset model outputs the similarity between the predicted image and the text data. The loss value between the predicted image and the second image sample is calculated through formula (1) or formula (2), and then the parameters of the first preset model are updated according to the loss value and the similarity.

[0296] It should be noted that in the second training, the parameters of the second preset model are fixed.

[0297] In the second training process, the user can determine the image restoration capability of the first preset model through the displayed predicted image. When the image quality of the predicted image is always at a high level, the user can trigger the second preset operation to end the second training. Of course, the second training can also be ended when the loss value and the similarity value tend to be close to the respective preset threshold values.

[0298] Based on the same inventive concept, the present disclosure also provides a model training platform. Referring to Figure 13 , a schematic diagram of the framework structure of a model training platform is shown, as Figure 13 , specifically can include a sample library and a training module; wherein:

[0299] The sample library is used to store a plurality of image sample pairs, and each image sample pair includes a first image sample and a second image sample of the same image. The image quality of the second image sample is higher than that of the first image sample.

[0300] The training module is used to respond to a training operation, and a plurality of image sample pairs are used as training samples to train the first preset model multiple times. The first preset model is used to improve the image quality of the first image sample. The training process includes:

[0301] acquire text data corresponding to the predicted image currently output by the first preset model; the text data is used to describe the image quality difference between the predicted image and the second image sample;

[0302] update the parameters of the first preset model based on the text data, the predicted image, and the second image sample.

[0303] The model training platform provided in the embodiment can include a sample library and a training module connected with the sample library. The first preset model can be deployed in the training module. The training module can extract a plurality of image sample pairs from the sample library in response to a training operation, input a first image sample in the image sample pair to the first preset model, and thus start training the first preset model.

[0304] As described in the above embodiment, in the process of training the first preset model, there can be at least one training mode as follows: acquiring text data corresponding to the predicted image currently output by the first preset model; and updating the parameters of the first preset model based on the text data, the predicted image, and the second image sample; wherein the text data includes data for evaluating the image quality of the predicted image.

[0305] Correspondingly, if the text data is data generated by positively evaluating the predicted image, the text data is consistent with the image quality of the predicted image, the target text opposite to the semantics of the text data can be determined, the similarity between the target text and the predicted image is determined, then the loss value between the predicted image and the second image sample is determined, and in determining the loss, the above formula (1) or formula (2) can be used. Thus, the parameters of the first preset model can be updated according to the loss value and the similarity.

[0306] The plurality of image sample pairs included in the sample library can be uploaded by a user in advance, and the acquisition process of the image sample pairs can refer to the description in the above model training method embodiment, which will not be described here.

[0307] Using such a training platform, the training module can use the plurality of image samples acquired from the sample library as training samples to train the first preset model. In the training process, the text data can be used to determine the to-be-optimized region in the predicted image that is different from the expected image quality. Thus, when updating the first preset model, the to-be-optimized region can be fed back to the first preset model to supervise the reinforcement learning of the first preset model on the repair of the to-be-optimized region, thereby guiding the first preset model to continuously optimize the weak link of image repair and helping to improve the repair quality of the model on the image.

[0308] As described in the above embodiments, the training of the first preset model can be performed in stages, including a first training stage, a second training stage and a third training stage. In the first training stage, the first preset model is trained using part of the image samples. In this training process, the parameters of the first preset model are updated based on the predicted image output by the first preset model and the second image sample. In the second training process, the parameters of the first preset model are updated based on the predicted image output by the first preset model, the second image sample and the text data corresponding to the predicted image. In the third training process, the second preset model is trained based on the third image sample and the text data sample.

[0309] For example, to enable intuitive visualization of the training stages and control the switching between the training stages during the staged training of the first preset model, the training platform can further provide an operation interface, which can include a control area that includes a plurality of controls. Different controls are used to start different training when triggered. Wherein:

[0310] The training includes first training and second training. The first training includes updating the parameters of the first preset model based on the predicted image output by the first preset model and the second image sample.

[0311] The second training includes updating the parameters of the first preset model obtained by the first training based on the text data, the predicted image and the second image sample.

[0312] In this embodiment, the training module can start the corresponding training stage in response to the triggering of the corresponding control in the operation interface. For example, in response to the triggering of the first control in the operation interface, the training module can obtain a plurality of image sample pairs from the sample library and start the first training of the first preset model using the obtained plurality of image sample pairs as training samples. Starting the first training can mean that the training module starts a first thread corresponding to the first training. When the first thread is running, the first image sample in the obtained image sample pair can be input to the first preset model, and the predicted image output by the first preset model and the second image sample can be used to calculate the loss function. Then, the parameters of the first preset model are updated according to the value of the loss function.

[0313] For example, in response to triggering the second control in the operation interface, the second thread corresponding to the second training can be started, and the first thread is kept running; wherein, when the second thread is running, the first image sample in the obtained image sample pair can be input to the first preset model, and the predicted image output by the first preset model can be displayed, and the text data input for the predicted image can be listened to; then, it is judged whether the text data is data generated by positively evaluating the predicted image, if yes, the text data is converted into target text (data for inversely evaluating the predicted image), and the similarity between the target text and the predicted image is calculated; if not, the similarity between the text data and the predicted image is calculated.

[0314] Since the first thread is kept running, the first thread will calculate the loss value between the predicted image and the second image sample, in this case, the second thread will feed back the similarity to the first thread, and the first thread will start updating the parameters of the first preset model according to the similarity and the loss value when receiving the similarity.

[0315] Of course, as mentioned above, the control can also include a control for triggering a third training, which includes: inputting a plurality of third image samples and corresponding at least two text data samples to the second preset model to train the second preset model; wherein, the second preset model after the third training is completed, is used to determine the similarity between the predicted image and the text data in the training of the first preset model.

[0316] Wherein, the control for triggering the third training can be called the third control, when the third control is triggered, the third thread corresponding to the third training can be started, when the third thread is started, the third image sample and the corresponding text data sample can be obtained from the sample library, the third image sample and the text data sample are input to the second preset model, and the loss value is determined according to the predicted similarity output by the second preset model and the category label corresponding to the text data sample, and the parameters of the second preset model are updated based on the loss value.

[0317] As mentioned above, the step of third training the second preset model can be performed in the process of training the first preset model; wherein, the plurality of third image samples include at least one of the following: the first image sample, the second image sample, and the predicted image output by the first preset model before the current time.

[0318] In one implementation, since the third image sample can include a predicted image output in the first training process, the first thread can save each predicted image output to the sample library and associate the predicted image with the first image sample; before the third training starts, in response to the labeling of the third image sample, the fourth thread for labeling can obtain the predicted image from the sample library and display the predicted image, and then obtain the text data sample input for the displayed predicted image, save the text data sample to the sample library after associating the text data sample with the predicted image.

[0319] In another implementation, in response to the data labeling function being enabled in the first training, the training module can instruct the first thread to display the predicted image output by the first preset model in the display interface after obtaining the predicted image, and then the fourth thread can be started in response to the input operation triggered on the display interface, obtain the text data sample input for the displayed predicted image, and save the text data sample to the sample library after associating the text data sample with the predicted image; and update the parameters of the first preset model according to the loss value between the predicted image and the second image sample.

[0320] In any of the above implementations, when the third thread is started, the predicted image and the corresponding text data sample can be input to the second preset model from the sample library for training.

[0321] As described above, the text data sample needs to be associated with the predicted image, so that the text data sample can be found based on the association in the second training for updating the parameters of the first preset model. Accordingly, the training platform can further include an association unit and an output unit, the association unit can be used to associate the text data sample input by the input area with the predicted image when the predicted image is displayed;

[0322] The output unit can be used to output the text data sample associated with the predicted image output by the first preset model as text data when the first preset model is trained for the second time.

[0323] Specifically, the output unit can listen to the predicted image output by the first preset model when the second thread is running, and find the text data sample associated with the predicted image from the sample library, and send the text data sample to the second thread, so that the second thread can input the text data sample output by the output unit to the second preset model.

[0324] For example, to enable the first preset model to visually visualize the training stage in the staged training process, the operation interface can further include a display area and an input area; wherein:

[0325] The display area is configured to display at least one of: a predicted image output by the first preset model, a third image sample, and a loss value in the third training process; the loss value is determined by a predicted similarity output by the second preset model and a category label; the category label is configured to indicate whether the description of the text data sample conforms to the image quality of the third image sample.

[0326] The input area is configured to allow a user to input text data and text data samples.

[0327] In the second training and the first training, the first thread can output the predicted image to the display interface for display. For example, the first thread can send the predicted image to a front-end interface rendering thread of the training platform, and the front-end interface rendering thread can render the predicted image to the display area. Correspondingly, the first thread can perform the operation of outputting the predicted image to the front-end interface rendering thread in the first training and the second training, thereby helping the user to intuitively observe the optimization degree of the first preset model in the first training and the second training.

[0328] In the third training, the third thread can send the loss between the predicted similarity and the category label to the front-end interface rendering thread of the training platform, and the front-end interface rendering thread can render the predicted image to the display area.

[0329] In an example, the front-end interface rendering thread can record the received loss values, and in response to a triggering operation on a loss value change curve on the front-end interface (display interface), the front-end interface rendering thread can generate a gradient change curve based on the plurality of loss values and the time corresponding to the plurality of loss values, and render the gradient change curve to the display area. In this way, the user can determine whether the second preset model converges by the gradient change curve.

[0330] In another example, the front-end interface rendering thread can display the received loss values in real time, for example, near the display area. In this way, the user can determine the optimization degree of the second preset model in the third training by the loss values displayed in real time.

[0331] In some examples, the third image sample used in the third training can correspond to two types of text data samples, including a first type of text data sample and a second type of text data sample. The first type of text data sample represents that the description of the text data sample conforms to the image quality of the third image sample, and the second type of text data sample represents that the description of the text data sample does not conform to the image quality of the third image sample.

[0332] Accordingly, in this embodiment, in response to the first text data input in the input area for the displayed third image sample, the first text data can be sent to the fifth thread, and the fifth thread will perform the following steps:

[0333] Determine the category corresponding to the first text data, wherein the category is used to indicate whether the description of the first text data matches the image quality of the third image sample;

[0334] Based on the first text data, second text data is generated; wherein the category of the second text data is different from the category of the first text data;

[0335] The first text data and the second text data are used as the text data samples corresponding to the third image sample;

[0336] The fifth thread can bind the first and second text data with the third image sample and store them in the sample library.

[0337] Accordingly, in the third training, the third thread can obtain the third image sample from the sample library, as well as the first text data and the second text data corresponding to the third image sample, and input the third image sample, the first text data and the second text data into the second preset model, so as to update the parameters of the second preset model according to the predicted similarity and category label corresponding to the two types of text data samples respectively.

[0338] Accordingly, the second preset model may include an image encoder, a text encoder, and a similarity determination module. The third thread can input the third image sample into the image encoder of the second preset model and input the text data sample corresponding to the third image sample into the text encoder of the second preset model. The similarity determination module determines the predicted similarity between the text data sample and the third sample image. Then, the third thread can update the parameters of the second preset model based on this predicted similarity and category label.

[0339] The following section, using a specific example, describes the model training process performed by the disclosed model training platform. Figure 14a - Figure 14d The diagram shows the changes in the user interface of the model training platform during the model training process.

[0340] S21: As Figure 14aAs shown, first, the first operation interface can be output in response to the user creating a model training task, and the setting bar required for creating a training task is given in the first operation interface. Including but not limited to: task name, training data set, network structure of the first preset model, network structure of the second preset model, loss calculation these options; When the setting is completed, click the "start training 1" control (first control) in the first operation interface, the first training can be started.

[0341] S22: As Figure 14b As shown, when the "start training 1" control in the first operation interface is triggered, the operation platform outputs the second operation interface, in which multiple image sample pairs in the sample library for first training can be selected, and the list of first image samples in the selected image sample pair can be displayed on the second operation interface, such as image 1, image 2-image 5, and the first image sample currently input to the first preset model for training can be highlighted in the list, such as black and bold representing the first image sample currently input to the first preset model for training.

[0342] Among them, in response to the triggering of the first control, the first thread starts running, and the multiple first image samples displayed in the list can be automatically input to the first preset model in order, and the first thread can calculate the loss value based on the predicted image output by the first preset model and the second image sample, and then update the parameters of the first preset model according to the loss value;

[0343] At the same time, in the first training, the second operation interface includes a display area and an input area, and the first image sample currently input to the first preset model for training and the predicted image output after repair can be displayed in the display area; Among them, in this example, the input area in the second operation interface can be locked to not support text labeling of the predicted image, that is, in the first training, the predicted image can not be labeled with text data samples;

[0344] When the user thinks that the first preset model has been trained to an effective state according to the displayed predicted image, the user can click the start training 2 button (third control) below, and then the third training is started.

[0345] Among them, in the second operation interface, the predicted image and the first image sample can be displayed in the display area; such as clicking the first image sample in the display area, then switching to display the predicted image, and clicking the predicted image to switch to display the first image sample.

[0346] Among them, in the first training process, the predicted image output by the first preset model can be saved to the sample library, so as to label the predicted image as a third image sample in the future.

[0347] S23: AsFigure 14c As shown, when the control (third control) of "start training 2" in the second operation interface is triggered, the third operation interface is output, and the third training is started.

[0348] In the third training, a list of third image samples can be displayed on the third operation interface. Specifically, in response to the third training, third image samples accurate for the third training can be obtained from the sample library, and a preset number of predicted images at the end of the first training can be obtained from the sample library, and these images can be displayed as third image samples in the list of the third operation interface, such as image 6, image 7-image 10. In the list, the current third image sample to be labeled can be highlighted, such as the black bold representing the current third image sample to be labeled. The third operation interface still includes a display area in which the selected third image sample can be displayed. The third operation interface still includes an input area. At this time, in the third training stage, the input area is unlocked, and text input can be allowed.

[0349] Wherein, the user can input the text expression in the input area, and the text expression that has not been input is set with a category label. At this time, the third image sample displayed in the display area and the text input in the input area are bound. Wherein, if one third image sample is selected, and the user observer thinks that there is no problem, it is considered that the third image sample is an image sample with higher image quality. Therefore, the user can not need to fill in the text in the input area, and the training platform can directly take the first preset text data as the text data sample of the first category of the third image sample, and / or take the second preset text data as the text data sample of the second category of the third image sample.

[0350] Then, the next third image sample can be directly clicked for labeling. When all the third image samples are labeled, the completion of labeling is clicked. At this time, the third training is really started.

[0351] At this time, the third thread starts to run, inputs the third image sample and the corresponding text data sample into the second preset model, and determines the loss value according to the prediction similarity output by the second preset model and the category label carried by the text data sample. The parameters of the second preset model are updated according to the loss value.

[0352] S24: as Figure 14c As shown, in the third operation interface, the loss value obtained in the third training can also be displayed in real time, such as Figure 14c "Training loss value dynamic display: 0.00012" in

[0353] Wherein, a fourth control can also be set in the third operation interface. The fourth control can be Figure 14cThe control of "viewing loss curve" is triggered, and the loss values corresponding to the second preset model in the third training process are displayed in the third operation interface, as shown in FIG. 8, and the current loss value record of the third training is dynamically displayed; Figure 11

[0354] The user can determine whether the third training needs to be ended according to the displayed loss value, and if so, the control (second control) of "starting training 3" can be clicked, so that the second training is started in response to the triggering of the second control.

[0355] S4: as Figure 14d shown, when the control of "starting training 3" in the third operation interface is triggered, the fourth operation interface is output, and the second training is started.

[0356] The second training is a mode of training while annotating text, and the fourth operation interface still includes a list, and the list includes a plurality of first image samples as training samples, such as image n1-image n5. The first image sample corresponding to the predicted image used for the third training can be arranged at the front, that is, the first preset number of first image samples arranged at the front in the list, and the predicted image corresponding to the first image sample is bound with text data samples in the sample library.

[0357] For the first preset number of first image samples, the selected first image sample can be input to the first preset model in response to the selection operation of the image identifier displayed in the list. The predicted image output by the first preset model for the first image sample is displayed in the display area of the third operation interface, and the text data sample corresponding to the predicted image obtained from the sample library is displayed in the input area. Then, the operation in the display area can be listened to. If the user clicks the input area and updates the text data, the user's newly input text data is sent to the second thread, and the second thread calculates the similarity based on the input text data and the predicted image and feeds back to the first thread. If the user does not click the input area and update the text data, the text data sample is sent to the second thread, and the second thread calculates the similarity based on the input text data sample and the predicted image and feeds back to the first thread.

[0358] For subsequent first image samples, the selected first image sample can be input to the first preset model in response to the selection operation of the image identifier displayed in the list. The predicted image output by the first preset model for the first image sample is displayed in the display area of the third operation interface, and the operation in the display area can be listened to. The text data input by the user in the input area is obtained, and the text data is sent to the second thread. The second thread calculates the similarity based on the input text data and the predicted image and feeds back to the first thread. ​

[0359] The first thread updates the parameters of the first preset model in the second training based on the similarity sent by the second thread and the loss value between the predicted image and the second image sample.

[0360] The first thread updates the parameters of the first preset model in the first training based on the loss value between the predicted image and the second image sample.

[0361] The third thread is not running when the first thread and the second thread are running to fix the parameters of the second preset model.

[0362] In summary, by using the model training platform provided by the present disclosure, through the operation interface, the training module, and the display area, the input area, and the plurality of controls arranged on the operation interface, the user can help the user to train the first preset model in stages, and observe the optimization degree of the image restoration model in the training process through the predicted image displayed in the display area. By inputting the evaluation text of the predicted image in the input area, the first preset model can be guided to update the parameters in the optimization direction indicated by the evaluation text during the training of the first preset model, so as to realize the model training from the human visual angle in the image restoration scene, make a new attempt and exploration for the model training in the image restoration field, and thus optimize the model training mechanism and contribute to improving the image quality of the first preset model.

[0363] Based on the same inventive concept, the present disclosure also provides an image restoration method. As shown in Figure 15 , a step flow diagram of an image restoration method is shown. As shown in Figure 15 , the specific steps can include the following steps:

[0364] Step S1501: obtaining a target image to be repaired;

[0365] Step S1502: inputting the target image into an image restoration model; wherein the image restoration model is a first preset model trained according to the model training method;

[0366] Step S1503: obtaining a repaired image output by the image restoration model; wherein the image quality of the repaired image is higher than that of the target image.

[0367] In this embodiment, the target image can be an image with poor image quality, such as old photos, movie film data with scratches, etc.

[0368] The image restoration model is the first preset model described in the above embodiments, and its training process can refer to the description of the above embodiments, which will not be repeated here. The restoration image output by the image restoration model can be obtained. Since the image restoration model is used for image restoration processing, the image quality of the restoration image is higher than that of the target image.

[0369] For example, the image restoration task of the image restoration model is clarity restoration, and the clarity of the restoration image is higher than that of the target image. For another example, the image restoration task of the image restoration model is scratch restoration, and the number of scratches of the restoration image is less than that of the target image. For another example, the image restoration task of the image restoration model is missing part restoration, and the missing image region in the target image is restored with image details. For another example, the image restoration task of the image restoration model is noise restoration, and the number of noise points in the restoration image is less than that in the target image.

[0370] The image restoration method of this embodiment uses the image restoration model trained according to the above model training method. In the training, the parameters of the first preset model are updated using the text data, the predicted image and the second image sample. The text data includes data for evaluating the image quality of the predicted image. Therefore, the training can be supervised not only based on the pixel difference between the predicted image output by the model and the second image sample, but also based on the difference in visual evaluation of the predicted image by human vision. Therefore, the model can acquire the ability to perceive human vision, so that the optimization direction of the first preset model can be guided based on human vision, thereby helping to improve the image restoration quality of the trained model, and thus improving the image restoration quality of the target image.

[0371] Based on the same inventive concept, the present disclosure also provides an image restoration device, which refers to Figure 16 As shown in FIG. 1, a structural schematic diagram of an image restoration device is shown. As shown in FIG. 1, the image restoration device can specifically include the following modules: Figure 16

[0372] The first acquisition module is configured to acquire a target image to be restored.

[0373] The input module is configured to input the target image into an image restoration model. The image restoration model is a first preset model trained according to the model training method.

[0374] The second acquisition module is configured to acquire a restoration image output by the image restoration model. The image quality of the restoration image is higher than that of the target image.

[0375] The description of the device embodiment can be found in the description of the above method embodiments, which will not be repeated here. ​

[0376] The embodiments of the present disclosure further provide a computer readable storage medium storing a computer program, which causes a processor to execute the model training method or the image inpainting method as described in the embodiments of the present disclosure.

[0377] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0378] Finally, it should be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, product or equipment. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, product or equipment including the element.

[0379] The above describes in detail the model training method and platform, the image inpainting method, the device, the equipment and the medium provided by the present disclosure. The principles and implementation manners of the present disclosure are described by applying specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present disclosure; at the same time, for those skilled in the art, according to the idea of the present disclosure, the specific implementation manner and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present disclosure.

[0380] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and the practice of the disclosed subject matter. The present disclosure is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not disclosed by the present disclosure. The specification and examples are only considered as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0381] It should be understood that the present disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is only limited by the appended claims.

[0382] As used in this description, the terms "one embodiment", "an embodiment” or "one or more embodiments” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase "in one embodiment” in various places in this description are not necessarily all referring to the same embodiment.

[0383] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0384] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The disclosure can be implemented by means of both hardware and software, and any combinations of these. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain task. The use of the term "first", "second", "third" etc. does not limit the number of these means. These terms can be construed to either refer to names of the means themselves or to the order of appearance of these means in the description or claims. The use of the terms "first", "second", "third" etc. does not limit the number of these means.

[0385] It should be noted that the above-mentioned embodiments illustrate rather than limit the disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. It is appreciated that some features of the above-described embodiments can be used to advantage without the use of other features. Dependent claims enumerating several features should be construed as applying individually to each and every dependent claim and not as a limitation on the separate use of these features in the respective dependent claims.

Claims

1. A model training method, wherein, The method comprises: obtaining a plurality of image sample pairs, the image sample pair comprising a first image sample and a second image sample of the same image; wherein the image quality of the second image sample is higher than that of the first image sample; training a first preset model using the plurality of image sample pairs as training samples, wherein the first preset model is used to improve the image quality of the first image sample, and the training process comprises: obtaining text data corresponding to a predicted image currently output by the first preset model; wherein the text data comprises data evaluating the image quality of the predicted image; updating parameters of the first preset model based on the text data, the predicted image and the second image sample; wherein the updating of the parameters of the first preset model based on the text data, the predicted image and the second image sample comprises: determining the similarity between a target text and the predicted image; wherein the target text is the text data or a text with a semantic opposite to the text data; determining a loss value based on the predicted image and the second image sample; updating the parameters of the first preset model based on the similarity and the loss value.

2. The method of claim 1, wherein, The determination of the similarity between the target text and the predicted image comprises: encoding the target text to obtain a text feature vector of the target text; encoding the predicted image to obtain an image feature vector of the predicted image; wherein the text feature vector and the image feature vector have consistent dimensions; determining the similarity based on the text feature vector and the image feature vector.

3. The method of claim 1, wherein, The training of the first preset model using the plurality of image sample pairs as training samples comprises: first training of the first preset model using part of the image sample pairs as training samples; wherein in the first training, the parameters of the first preset model are updated based on a predicted image output by the first preset model and the second image sample; second training of the first preset model obtained through the first training using part or all of the image sample pairs as training samples; wherein in the second training, the parameters of the first preset model obtained through the first training are updated based on the text data, the predicted image and the second image sample.

4. The method according to any of claims 1 to 3, wherein, The method further comprises: obtaining a plurality of third image samples and text data samples corresponding to the third image samples; wherein the text data samples are used to describe the image quality of the third image samples; third training of a second preset model based on the plurality of third image samples and the text data samples; wherein the second preset model is used to determine the similarity between the third image samples and the text data samples; The updating of the parameters of the first preset model based on the text data, the predicted image and the second image sample comprises: inputting the predicted image and the text data into the second preset model after the third training is completed; update parameters of the first preset model based on the similarity output by the second preset model, the predicted image and the second image sample.

5. The method of claim 4, wherein, The text data sample carries a category label, which is used to indicate whether the description of the text data sample conforms to the image quality of the third image sample; the second preset model is trained based on multiple third image samples and corresponding at least two text data samples, including: inputting the third image sample and the text data sample into the second preset model to obtain a predicted similarity between the third image sample and each text data sample; updating parameters of the second preset model based on the predicted similarity and the category label.

6. The method of claim 4, wherein, The third image sample corresponds to text data samples of a first category and text data samples of a second category, the first category representing that the description of the text data sample conforms to the image quality of the third image sample, and the second category representing that the description of the text data sample does not conform to the image quality of the third image sample.

7. The method of claim 4, wherein, The second preset model includes a text encoder and an image encoder, and a similarity determination module connected to the text encoder and the image encoder respectively; wherein: The text encoder is configured to perform text encoding on the text data sample to obtain a predicted text vector; The image encoder is configured to perform image encoding on the third image sample to obtain a predicted image vector; wherein the predicted image vector and the predicted text vector have consistent dimensions; The similarity determination module is configured to determine a predicted similarity between the predicted image vector and the predicted text vector.

8. The method of claim 4, wherein, In the process of training the first preset model, the step of training the second preset model is performed; the multiple third image samples include at least one of the following: the first image sample, the second image sample, and a predicted image output by the first preset model before the current time.

9. The method of any one of claims 5-8, wherein, The third training is performed in the gap of training the first preset model, the third image sample includes a predicted image output by the first preset model for the first image sample, and the second preset model is trained based on multiple third image samples and text data samples, including: inputting multiple first image samples into the first preset model; inputting a predicted image output by the first preset model and a text data sample corresponding to the predicted image into the second preset model to perform the third training on the second preset model; wherein in the third training, the parameters of the first preset model are fixed.

10. The method of any one of claims 5-8, wherein, After the third training, the training of the first preset model includes: inputting multiple first image samples into the first preset model; inputting a predicted image output by the first preset model and the text data into the second preset model; updating parameters of the first preset model based on the predicted image, the second image sample, and a predicted similarity output by the second preset model; Specifically, when updating the parameters of the first preset model, the parameters of the second preset model are fixed.

11. The method of claim 4, wherein, The acquisition of multiple third image samples and corresponding text data samples includes: Obtain the first text data corresponding to the third image sample; Determine the category corresponding to the first text data, wherein the category is used to indicate whether the description of the first text data matches the image quality of the third image sample; Based on the first text data, second text data is generated; wherein the category of the second text data is different from the category of the first text data; The first text data and the second text data are used as the text data samples corresponding to the third image sample.

12. The method of claim 4, wherein, The step of obtaining text data corresponding to the predicted image currently output by the first preset model includes: From the plurality of third image samples, determine whether there exists a target third image sample corresponding to the predicted image; wherein the target third image sample and the predicted image correspond to the same first image sample; If so, the text data sample corresponding to the target third image sample shall be used as the text data; If not, then obtain the text data input for the predicted image.

13. The method of claim 1, wherein, The step of obtaining text data corresponding to the predicted image currently output by the first preset model includes: Display the predicted image; Obtain the text data input for the predicted image.

14. The method of claim 13, wherein, After displaying the predicted image, the method further includes: Listen for input operations on the user interface that displays the predicted image; If the input operation is not detected, the preset text data will be used as the text data corresponding to the predicted image. The step of obtaining text data input for the predicted image includes: If the input operation is detected, the text data input for the predicted image is obtained.

15. The method of claim 1, wherein, The text data includes at least one term; the at least one term is used to describe the image quality of the predicted image in different image regions, and / or the image quality in different quality dimensions.

16. The method of any one of claims 3 or 13-15, wherein, The method further includes at least one of the following: In response to the first training, the predicted image output by the first preset model is displayed, and in response to the first preset operation performed on the predicted image, the first training ends; In response to the second training, the predicted image output by the first preset model is displayed, and in response to the second preset operation performed on the predicted image, the second training ends.

17. The method of claim 5, wherein, During the third training process on the second preset model, the method further includes: Display the loss value corresponding to at least one training iteration of the second preset model prior to the current time; wherein, the loss value is determined by the predicted similarity and the category label; The third training ends in response to a third preset operation performed on each of the displayed loss values.

18. An image inpainting method, wherein, The method includes: The target image to be repaired; The target image is input into the image restoration model; wherein the image restoration model is a first preset model trained by the model training method according to any one of claims 1-17; obtain a repaired image output by the image inpainting model; wherein the repaired image has a higher image quality than the target image.

19. An image inpainting apparatus, wherein, The device comprises: a first obtaining module configured to obtain a target image to be repaired; an input module configured to input the target image into an image inpainting model; wherein the image inpainting model is a first preset model trained according to the model training method of any one of claims 1-17; a second obtaining module configured to obtain a repaired image output by the image inpainting model; wherein the repaired image has a higher image quality than the target image.

20. An electronic device, comprising: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the model training method of any one of claims 1-17 or the image inpainting method of claim 18.

21. A computer readable storage medium, wherein, The computer program stored therein causes the processor to execute the model training method of any one of claims 1-17 or the image inpainting method of claim 18.

Images