Training method of image processing model, image restoration method and related equipment

Abstract

The invention discloses a training method of an image processing model, an image restoration method and related equipment, which are used for improving the restoration effect of partial parts in an object image and obtaining a high-quality restored image. The training method of the image processing model comprises the steps that key point detection is carried out on a first image of a sample object to obtain first key point data, and the first key point data comprises key points of a specified part of the sample object in the first image; based on the first key point data, performing mask processing on a first region of the first image to obtain a sample mask image, the first region being a region of the specified part in the first image; filling the sample mask image based on the first key point data and a reference image of the specified part through an image processing model to obtain a repaired image of the first image; and adjusting parameters of the image processing model based on the restored image and the first image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing model training method, an image restoration method and related equipment. Background Art

[0002] Image restoration is a key technology in the field of image processing, which is used to repair image areas that do not meet the requirements. For example, some parts of the image containing the object are blurred, which affects the downstream business processing effect. This requires the use of image restoration technology to repair such image areas. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing model training method, an image restoration method and related equipment.

[0004] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a method for training an image processing model, comprising:

[0006] Performing key point detection on a first image of the sample object to obtain first key point data, wherein the first key point data includes key points of a designated part of the sample object in the first image;

[0007] Based on the first key point data, mask processing is performed on a first area of ​​the first image to obtain a sample mask image, where the first area is an area of ​​the designated part in the first image;

[0008] Filling the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a repaired image of the first image;

[0009] Based on the restored image and the first image, parameters of the image processing model are adjusted.

[0010] In a second aspect, an embodiment of the present application provides an image restoration method, comprising:

[0011] Performing key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a designated part of the target object in the image;

[0012] Based on the target key point data, mask processing is performed on the target area of ​​the image to obtain a target mask image, where the target area is the area of ​​the designated part in the image;

[0013] The target mask image is filled based on the target key point data through an image processing model to obtain a repaired image of the image.

[0014] In a third aspect, an embodiment of the present application provides a training device for an image processing model, comprising:

[0015] A first detection module, configured to perform key point detection on a first image of a sample object to obtain first key point data, wherein the first key point data includes key points of a specified part of the sample object in the first image;

[0016] A first mask module, configured to perform mask processing on a first region of the first image based on the first key point data to obtain a sample mask image, wherein the first region is a region of the designated part in the first image;

[0017] A first restoration module, configured to fill the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a restored image of the first image;

[0018] A training module is used to adjust the parameters of the image processing model based on the restored image and the first image.

[0019] In a fourth aspect, an embodiment of the present application provides an image restoration device, comprising:

[0020] A second detection module is used to perform key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a specified part of the target object in the image;

[0021] A second mask module is used to perform mask processing on a target area of ​​the image based on the target key point data to obtain a target mask image, wherein the target area is an area of ​​the designated part in the image;

[0022] The second restoration module is used to fill the target mask image based on the target key point data through an image processing model to obtain a restored image of the image.

[0023] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0024] processor;

[0025] a memory for storing instructions executable by the processor;

[0026] Wherein, the processor is configured to execute the instructions to implement the training method of the image processing model provided in the first aspect; or, the processor is configured to execute the instructions to implement the image restoration method provided in the second aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the training method of the image processing model provided in the first aspect; or, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the image restoration method provided in the second aspect.

[0028] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the training method of the image processing model provided in the first aspect or the image restoration method provided in the second aspect.

[0029] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0030] By performing key point detection on the first image of the sample object, first key point data is obtained, which includes the key points of the designated part of the sample object in the first image; then, based on the first key point data, the area of ​​the designated part in the first image is masked, so that the area of ​​the designated part is removed, and a sample mask image is obtained; further, since the first key point data can objectively and accurately reflect the position distribution of the designated part, the reference image of the designated part can clearly reflect the characteristics of the designated part, the sample mask image is filled based on the image processing model, the first key point data and the reference image of the designated part, and the image processing model is trained based on the obtained repaired image and the first image, so that the image processing model can use the position distribution and characteristics of the designated part to learn how to fill the removed area in the first image, and ensure that the filling result meets the quality requirements. In this way, the training effect of the image processing model is improved, so that the trained image processing model can repair the area of ​​the designated part in the input image in a targeted manner to obtain a high-quality repaired image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1A flowchart of a method for training an image processing model provided for one embodiment of the present application;

[0033] Figure 2 A flowchart of a method for training an image processing model provided in another embodiment of the present application;

[0034] Figure 3 A schematic diagram of a flow chart of an image restoration method provided for one embodiment of the present application;

[0035] Figure 4 A schematic flow chart of an image restoration method provided for another embodiment of the present application;

[0036] Figure 5 A schematic diagram of the structure of a training device for an image processing model provided in one embodiment of the present application;

[0037] Figure 6 A schematic diagram of the structure of an image restoration device provided in one embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0040] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0041] The inventive concept of the image restoration scheme proposed in the embodiment of the present application is: first, the image area of ​​the unclear part in the object image is removed, and then the removed image area is filled with the reference image of the part, so that the image restoration can be performed in a targeted manner to obtain a high-quality restored image. The image area of ​​the part in the restored image meets the quality requirements, providing reliable data support for downstream business processing.

[0042] Based on the above invention concept, the embodiment of the present application proposes a training method for an image processing model, by performing key point detection on a first image of a sample object, obtaining first key point data, which includes key points of a specified part of the sample object in the first image; then, masking the area of ​​the specified part in the first image based on the first key point data, so that the area of ​​the specified part is removed, and obtaining a sample mask image; further, since the first key point data can objectively and accurately reflect the position distribution of the specified part, the reference image of the specified part can clearly reflect the characteristics of the specified part, the sample mask image is filled based on the image processing model, the first key point data and the reference image of the specified part, and the image processing model is trained based on the obtained repaired image and the first image, so that the image processing model can use the position distribution and characteristics of the specified part to learn how to fill the removed area in the first image, and ensure that the filling result meets the quality requirements. In this way, the training effect of the image processing model is improved, so that the trained image processing model can repair the area of ​​the specified part in the input image in a targeted manner to obtain a high-quality repaired image.

[0043] Based on the image processing model trained by the above-mentioned training method, an embodiment of the present application further proposes an image restoration method, which performs key point detection on the image of the target object to obtain the key points of the specified part of the target object in the image, i.e., target key point data; then, based on the target key point data, the area of ​​the specified part of the target object in the image is masked so that the area of ​​the specified part is removed to obtain a target mask image; further, since the target key point data can objectively and accurately reflect the position distribution of the specified part, the image processing model uses the position distribution information reflected by the target key point data to fill the removed area in a targeted manner, thereby ensuring that the restored image obtained by filling meets the quality requirements, thereby improving the restoration quality of the target object image.

[0044] It should be understood that the training method of the image processing model and the image restoration method provided in the embodiments of the present application can be performed by an electronic device. The so-called electronic device here may include terminal devices, such as smart phones, tablet computers, laptops, desktop computers, intelligent voice interaction devices, smart home appliances, smart watches, vehicle terminals, aircraft, etc.; or, the electronic device may also include a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The electronic device is independent of the back-end server.

[0045] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0046] Please refer to Figure 1, is a flow chart of a method for training an image processing model provided by an embodiment of the present application, the method comprising the following steps:

[0047] S102: Perform key point detection on a first image of the sample object to obtain first key point data.

[0048] The first key point data includes key points of a designated part of the sample object in the first image.

[0049] The designated part refers to one or more parts of the target object. Depending on the application scenario, the target object may be different, and the designated part may also be different. For example, in a digital human scenario, the first image is a digital human image, and the target object is a digital human. Since the teeth of a digital human often have jitter artifacts or are missing, the designated part includes the teeth of the digital human. For another example, in a face recognition scenario, the first image is a face image, the target object is a face, and the designated part is the eyes, lips, nose, and other parts of the face that reflect the characteristics of the face.

[0050] The first key point data is used to describe the position distribution of the designated part of the sample object in the first image, specifically including but not limited to the position coordinates of the key points of the designated part in the first image, etc. In the above S102, the first key point data can be obtained by performing key point detection on the first image using various key point detection technologies, which is not limited in the embodiment of the present application.

[0051] In one embodiment, the first key point data can be obtained by detecting the first image through a detection network. For example, the detection network can adopt a resnet18 network, etc., which is not limited in the embodiment of the present application. In practical applications, the detection network can be a model independent of the image processing model, or it can also be a part of the image processing model.

[0052] S104: Perform mask processing on the first region of the first image based on the first key point data to obtain a sample mask image.

[0053] The first region is a region of the designated part in the first image.

[0054] Since the first key point data objectively and accurately reflects the position distribution of the designated part in the first image, the area of ​​the designated part in the first image can be accurately located based on the first key point data, and the area can be accurately removed by masking the area.

[0055] In the above S104, various masking techniques may be used to perform masking processing on the first region, which is not limited in the embodiment of the present application. For example, the pixel values ​​of the pixels in the first region are set to 0.

[0056] S106, filling the sample mask image based on the first key point data and the reference image of the designated part through the image processing model to obtain a repaired image of the first image.

[0057] The reference image of the designated part refers to an image that contains the designated part and has an image quality (such as clarity, angle, etc.) of the area where the designated part is located that meets the requirements.

[0058] Since the first key point data can objectively and accurately reflect the position distribution of the designated part in the first image, and the reference image of the designated part can clearly reflect the characteristics of the designated part, the sample mask image is filled based on the image processing model, the first key point data and the reference image of the designated part, so that the image processing model can use the position distribution and characteristics of the designated part to learn how to fill the removed first area, thereby ensuring that the area of ​​the designated part in the repaired image is clearly visible.

[0059] In one embodiment, the above S106 includes the following steps:

[0060] S161, extracting features from the reference image to obtain texture features.

[0061] For example, the feature is the feature of the specified part in the reference image.

[0062] Texture features reflect the slowly changing or periodically changing surface structural organization and arrangement properties of the surface of a specified part, which can intuitively and accurately reflect the characteristics of the specified part and help reconstruct clearly visible image data of the specified part.

[0063] For example, Figure 2 As shown, the image processing model includes a repair network, which is used to repair the sample mask image. After the repair network performs convolution processing on the reference image, the convolution processing result is encoded using a multilayer perceptron (MLP) to obtain texture features. As an example, the repair network performs convolution processing on the reference image, which can be achieved through multiple 3*3 convolution layers.

[0064] S162, extracting features from the sample mask image to obtain a feature image.

[0065] The feature image contains the features of each pixel in the sample mask image.

[0066] For example, Figure 2As shown in the figure, after the restoration network performs convolution processing on the sample mask image, the Prelu function is used for activation processing to obtain the feature image of the sample mask image. As an example, the restoration network performs convolution processing on the reference image, which can be achieved through multiple layers of 3*3 convolution layers, with a convolution step size of 2 and a convolution kernel size of 64.

[0067] S163, based on the first key point data, fusing the texture feature and the feature image to obtain a restored image of the first image.

[0068] By fusing texture features and feature images, the texture features can be used to fill in the area of ​​the specified part in the feature image, so that the area of ​​the specified part in the repaired image is clearly visible and meets the quality requirements.

[0069] In order to avoid problems such as information loss during the filling process and to more accurately fill the first area removed in the sample mask image, in the above 163, based on the first key point data, a target sub-feature image of the feature image is determined, wherein the target sub-feature image is the area of ​​the specified part in the feature image; the target sub-feature image is fused with the texture feature to obtain a first fused feature image; and decoding is performed based on the first fused feature image to obtain a repaired image.

[0070] The target sub-feature image includes the features of the pixels of the specified part in the feature image. By fusing the target sub-feature image with the texture features, such pixels can be filled.

[0071] As an example of the decoding process of the first fused feature image, the target sub-feature image may be fused with the texture feature on the basis of the feature image and then decoded to obtain a restored image.

[0072] As another example, in order to avoid the problem of missing information during the fusion process, the first fused feature image and the feature image may be spliced ​​to obtain a second fused feature image, and the second fused feature image may be decoded to obtain a repaired image.

[0073] For example, Figure 2As shown, the repair network uses MLP to encode the first key point data to obtain a key point vector. Then, the repair network obtains the position distribution information of the specified part in the feature image from the key point vector, and then locates the target sub-feature image in the feature image, and fuses the target sub-feature image with the texture feature to obtain the first fused feature image. The first fused feature image is a latent vector, also known as a latent code. Finally, the first fused feature image is spliced ​​with the feature image by means of jump connection, that is, the fusion of the image and the texture feature is realized to obtain the second fused feature image; finally, the repair network decodes the second fused feature image by convolution processing on the second fused feature image to obtain a repaired image. As an example, the repair network performs convolution processing on the second fused feature image, which can be achieved through multiple layers of 3*3 convolution layers, the convolution step size is 1 / 2, and the convolution kernel size is 64.

[0074] The embodiment of the present application shows some implementation methods of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0075] S108: Adjust parameters of the image processing model based on the restored image and the first image.

[0076] The difference between the repaired image and the first image reflects the effect of the image processing model on repairing the first image. Training the image processing model based on this can enhance the repair effect of the image processing model, so that the trained image processing model can not only repair the area of ​​the specified part in the input image in a targeted manner, but also reduce the impact of the repair process on the texture features of other areas outside the area, thereby obtaining a high-quality repaired image.

[0077] In one embodiment, the above S108 includes the following steps:

[0078] S181, determining the loss of the image processing model based on the restored image and the first image.

[0079] As an example, in the case where the first key point data is obtained by performing key point detection on the first image through an image processing model, for example, the first key point data is obtained by performing key point detection through a detection network of the image processing model, the above S181 includes the following steps: determining the first loss of the image processing model based on the repaired image and the first image; determining the second loss of the image processing model based on the first key point data and the key point labels of the first image; determining the loss of the image processing model based on the first loss and the second loss.

[0080] The first loss represents the difference between the restored image and the first image, which reflects the restoration effect of the image processing model. In order to ensure that the trained image processing model can restore the texture features of the non-specified area in the image, the first loss can be determined based on the sample mask image and the restored image. Exemplarily, the first loss can be determined by the following formula (1).

[0081] The key point label of the first image is obtained by annotating the first region of the first image, which reflects the real key point of the specified part in the first image. The second loss represents the difference between the first key point data and the key point label, which reflects the key point detection accuracy of the image processing model. The second loss can be determined by the following formula (2).

[0082] The loss of the image processing model can be obtained by weighted summing the first loss and the second loss, as shown in the following formula (3). In this way, the influence of each link in the whole process from key point detection to image restoration on the image restoration result is fully considered. The restoration loss obtained can more accurately reflect the processing effect of the image processing model. Based on this, the image processing model can be trained to enhance the restoration ability of the image processing model and output a higher quality restoration image.

[0083] Lrec=MSE(x',y') (1)

[0084] L1=MSE(lmd(out),y) (2)

[0085] Loss=Lrec+α*L1 (3)

[0086] Among them, Lrec represents the first loss, x' represents the repaired image, y' represents the sample mask image, and MSE represents the mean square error; L1 represents the second loss, lmd(out) represents the first key point data, and y represents the key point label; Loss represents the loss of the image processing model, and α represents the weight, which can be set according to actual needs. As an example, considering that the difficulty of repair is less than the difficulty of key point recognition, α can be set to a value greater than 1 to increase the proportion of the second loss.

[0087] As another example, when the detection network is independent of the image processing model, the above S181 includes the following steps: determining a first loss based on the repaired image and the first image; and determining the first loss as the loss of the image processing model.

[0088] S182, adjusting parameters of the image processing model based on the loss.

[0089] As an example, a back-propagation algorithm may be used to adjust parameters of a restoration network of an image processing model based on the restoration loss.

[0090] As another example, when the image processing model also includes a detection network, a back-propagation algorithm is used to adjust parameters of the restoration network and parameters of the detection network based on the restoration loss, thereby achieving end-to-end training from restoration to key point detection.

[0091] In another example, when the image processing model also includes a detection network, the parameters of the detection network and the parameters of the repair network can be updated separately by step-by-step training. Specifically, before the above S102, it also includes: performing key point detection on the second image of the sample object through the detection network to obtain second key point data, the second key point data including the key points of the specified part of the sample object in the second image; adjusting the parameters of the detection network based on the second key point data and the key point labels of the second image. Accordingly, in the above S182, the parameters of the repair network are adjusted based on the loss.

[0092] Exemplarily, before adjusting the parameters of the repair network, the parameters of the detection network are frozen.

[0093] Based on the second key point data and the key point labels of the second image, the loss of the detection network is determined; then, based on the loss of the detection network, the parameters of the detection network are updated. By performing multiple training on the detection network, the pre-training of the detection network is completed.

[0094] In the embodiment of the present application, the second image can be randomly extracted from the sample image set. The number of second images can be multiple. In this case, the loss corresponding to each second image can be determined based on the above method; then, the sum of the losses corresponding to all second images is used as the loss of the detection network, and based on the loss, the back propagation algorithm is used to update the parameters of the detection network.

[0095] Exemplarily, the loss corresponding to each second image can be determined by the following formula (4).

[0096] Lm=BCE(x i ,y i ) / n, i∈n (4)

[0097] Among them, Lm represents the loss corresponding to the second image, BCE represents the sampling mean error, and x i Indicates the position coordinates of the i-th key point in the second key point data, y i represents the position label of the i-th key point in the key point label, and n represents the number of key points contained in the second key point data.

[0098] In the above implementation, before performing key point detection on the first image, the detection network is pre-trained, and then the pre-trained detection network is used to execute the above S102 to S106. Finally, in S108, the parameters of the detection network are frozen, and only the parameters of the repair network are updated. This can not only ensure the accuracy of the first key point data to ensure the training effect of the image processing model, but also reduce the amount of calculation in the training process and accelerate the convergence speed of the image processing model, thereby improving the model training efficiency while ensuring the model training effect.

[0099] The embodiment of the present application shows some implementation methods of the above S108. Of course, it should be understood that the above S108 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0100] It is worth noting that the above S102 to S108 is only a training process for the image processing model. In practical applications, S102 to S108 can be executed multiple times until the image processing model meets the preset training stop condition. Among them, the training stop condition can be set according to actual needs, such as the number of training times reaches a preset threshold, the restoration loss of the image processing model converges, etc., and the embodiments of the present application do not limit this.

[0101] In addition, the first image can be randomly extracted from a sample image set, and the sample image set includes multiple images containing sample objects. In practical applications, the number of first images can be multiple. In this case, steps S102 to S106 can be performed for each first image. Further, in the above S108, based on the difference between each first image and its restored image, the loss corresponding to each first image is determined; then, the sum of the losses corresponding to all first images is used as the loss of the image processing model, and the image processing model is trained based on the loss.

[0102] The training method of one or more image processing models provided in the embodiment of the present application is to obtain first key point data by performing key point detection on the first image of the sample object, which includes the key points of the designated part of the sample object in the first image; then, based on the first key point data, the area of ​​the designated part in the first image is masked so that the area of ​​the designated part is removed to obtain a sample mask image; further, since the first key point data can objectively and accurately reflect the position distribution of the designated part, the reference image of the designated part can clearly reflect the characteristics of the designated part, the sample mask image is filled based on the image processing model, the first key point data and the reference image of the designated part, and the image processing model is trained based on the obtained repaired image and the first image, so that the image processing model can use the position distribution and characteristics of the designated part to learn how to fill the removed area in the first image, and ensure that the filling result meets the quality requirements. In this way, the training effect of the image processing model is improved, so that the trained image processing model can repair the area of ​​the designated part in the input image in a targeted manner to obtain a high-quality repaired image.

[0103] Based on the image processing model trained by the above training method, the present application embodiment also provides an image restoration method. Figure 3 , is a flow chart of an image restoration method provided by an embodiment of the present application, the method comprising the following steps:

[0104] S302, performing key point detection on the image of the target object to obtain target key point data.

[0105] The target key point data includes the key points of the specified parts of the target object in the image.

[0106] The target key point data is used to describe the position distribution of the specified part in the image, including but not limited to the position coordinates of the key point.

[0107] The specific implementation of S302 is the same as Figure 1 The specific implementation of S102 in the illustrated embodiment is similar and will not be described in detail.

[0108] S304: performing mask processing on the target image area of ​​the image based on the target key point data to obtain a target mask image.

[0109] The target area is the area of ​​the specified part in the image.

[0110] Since the target key point data objectively and accurately reflects the position distribution of the specified part, the area of ​​the specified part in the image can be accurately located based on the target key point data. By masking the target image area, the target area can be accurately removed.

[0111] The specific implementation of S304 is the same as Figure 1 The specific implementation of S104 in the illustrated embodiment is similar and will not be described in detail.

[0112] S306, filling the target mask image based on the target key point data through the image processing model to obtain a repaired image of the image.

[0113] The specific implementation of S306 is the same as Figure 1 The specific implementation of S106 in the illustrated embodiment is similar and will not be described in detail.

[0114] For example, Figure 4 As shown in FIG. 1 , the image processing model includes a detection network and a repair network. After the image is detected by the detection network for key points, the target key point data is obtained. Then, the repair network performs convolution processing on the target mask image and uses the Prelu function for activation processing to obtain the feature image of the target mask image. In addition, the repair network also uses MLP to encode the target key point data to obtain the key point vector. Furthermore, since the repair network has mastered the texture features of the specified part with the required quality in the training stage, at this stage, the repair network obtains the position distribution information of the specified part from the key point vector, and then locates the target sub-feature image of the specified part in the feature image, and fuses the target sub-feature image with the mastered texture features to obtain the first fused feature image. Finally, the first fused feature image is spliced ​​with the feature image by means of jump connection, that is, the fusion of the image and the texture features is realized, and the second fused feature image is obtained; finally, the repair network decodes the second fused feature image by convolution processing on the second fused feature image to obtain the repaired image.

[0115] One or more embodiments of the present application provide an image restoration method, which performs key point detection on an image of a target object to obtain key points of a specified part of the target object in the image, i.e., target key point data; then, based on the target key point data, a mask is performed on an area in the image of the specified part of the target object, so that the area of ​​the specified part is removed to obtain a target mask image; further, since the target key point data can objectively and accurately reflect the position distribution of the specified part, the image processing model uses the position distribution information reflected by the target key point data to fill in the removed area in a targeted manner, thereby ensuring that the restored image obtained by filling meets the quality requirements, thereby improving the restoration quality of the target object image.

[0116] The training method of the image processing model and the image restoration method provided in the embodiment of the present application can be applied to various business scenarios with image restoration requirements. The following takes the digital human scene as an example to illustrate the training method of the image processing model and the image restoration method provided in the embodiment of the present application.

[0117] It should be understood that the application of the above method to the digital human scenario is only an exemplary description and should not be understood as a limitation on the application scenario of the above method.

[0118] In the digital human scene, in the digital human video generated based on the general digital human algorithm in the related technology, there is a problem of jitter artifacts or missing teeth of the digital human, which requires repairing the area of ​​the digital human's teeth in the video image.

[0119] Based on this, the training method of the image processing model provided in the embodiment of the present application obtains a first image of a digital human, and performs key point detection on the first image to obtain the key points of the digital human's teeth in the first image, that is, the first key point data; then, based on the first key point data, a mask process is performed on the area of ​​the teeth in the first image, so that the area is removed to obtain a sample mask image that does not contain teeth; further, through the image processing model, based on the first key point data and a reference image of the teeth, the sample mask image is filled to obtain a repaired image of the first image, and compared with the first image, the teeth of the digital human are clearly visible in the repaired image, which effectively overcomes the problem of jitter artifacts or missing teeth; finally, based on the repaired image and the first sample object image, the image processing model is trained, so that the trained image processing model can repair the area of ​​the teeth of any digital human in the image in a targeted manner to obtain a digital human image with clear teeth and high quality.

[0120] Based on the image processing model trained by the above-mentioned training method, the image restoration method provided in the embodiment of the present application receives a digital human video, and performs frame processing on the digital human video to obtain multiple video frames. Then, for each video frame, if the area of ​​the teeth in the video frame does not meet the quality requirements, such as the clarity does not meet the requirements, key point detection is performed on the video frame to obtain the key points of the teeth in the video frame, that is, the target key point data. Further, based on the target key point data, the area of ​​the teeth in the video frame is masked to obtain a target mask image. Finally, through the image processing model, the target mask image is filled based on the target key point data to obtain a repaired video frame. Compared with the original video frame, the digital human's teeth are clearly visible in the repaired video frame, which effectively overcomes the problem of jitter artifacts or missing teeth.

[0121] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] Based on the same inventive concept, the present application also provides a training device for an image processing model. Figure 5 , is a structural schematic diagram of a training device 500 for an image processing model provided in an embodiment of the present application, the device 500 includes: a first detection module 510, a first mask module 520, a first repair module 530 and a training module 540.

[0123] The first detection module 510 is used to perform key point detection on the first image of the sample object to obtain first key point data, where the first key point data includes key points of a specified part of the sample object in the first image.

[0124] The first mask module 520 is used to perform mask processing on a first area of ​​the first image based on the first key point data to obtain a sample mask image, where the first area is an area of ​​the designated part in the first image.

[0125] The first restoration module 530 is used to fill the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a restored image of the first image.

[0126] The training module 540 is used to adjust the parameters of the image processing model based on the restored image and the first image.

[0127] In another embodiment, the first repair module is used to:

[0128] Extracting features from the reference image to obtain texture features, where the texture features are features of the designated part in the reference image;

[0129] Performing feature extraction on the sample mask image to obtain a feature image;

[0130] Based on the first key point data, the texture feature and the feature image are fused to obtain a restored image of the first image.

[0131] In another embodiment, when the first restoration module performs a fusion process on the texture feature and the feature image based on the first key point data to obtain a restoration image of the first image, the first restoration module performs the following steps:

[0132] Based on the first key point data, determining a target sub-feature image of the feature image, wherein the target sub-feature image is a region of the designated part in the feature image;

[0133] Fusing the target sub-feature image with the texture feature to obtain a first fused feature image;

[0134] Decoding is performed based on the first fused feature image to obtain the repaired image.

[0135] In another embodiment, when the first restoration module decodes based on the first fused feature image to obtain the restoration image, the first restoration module performs the following steps:

[0136] splicing the first fused feature image with the feature image to obtain a second fused feature image;

[0137] The second fused feature image is decoded to obtain the repaired image.

[0138] In another embodiment, the training module is used to:

[0139] Determining a loss of the image processing model based on the inpainted image and the first image;

[0140] Based on the loss, parameters of the image processing model are adjusted.

[0141] In another embodiment, the first key point data is obtained by performing key point detection on the first image through the image processing model;

[0142] The training module performs the following steps when determining the loss of the image processing model based on the repaired image and the first image:

[0143] Determining a first loss of the image processing model based on the inpainted image and the sample mask image;

[0144] determining a second loss of the image processing model based on the first key point data and the key point labels of the first image;

[0145] Based on the first loss and the second loss, a loss of the image processing model is determined.

[0146] In another embodiment, the first key point data is obtained by performing key point detection on the first image through the detection network of the image processing model; the repaired image is obtained by filling the sample mask image through the repair network of the image processing model;

[0147] The training module is also used to: perform key point detection on the second image of the sample object through the detection network to obtain second key point data, where the second key point data includes key points of the designated part in the second image;

[0148] Adjusting parameters of the detection network based on the second key point data and the key point labels of the second image;

[0149] When the training module adjusts the parameters of the image processing model based on the loss, the training module performs the following steps:

[0150] Based on the loss, parameters of the restoration network are adjusted.

[0151] Obviously, the image processing model training device 500 provided in the embodiment of the present application can be used as the above Figure 1 The execution subject of the training method of the image processing model shown in FIG. Figure 1 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0152] Based on the same inventive concept, the present application also provides an image restoration device. Figure 6 , is a schematic structural diagram of an image restoration device 600 provided in one embodiment of the present application, the device 600 includes:

[0153] The second detection module 610 is used to perform key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a specified part of the target object in the image.

[0154] The second mask module 620 is used to perform mask processing on the target area of ​​the image based on the target key point data to obtain a target mask image, where the target area is the area of ​​the designated part in the image.

[0155] The second restoration module 630 is used to fill the target mask image based on the target key point data through an image processing model to obtain a restored image of the image.

[0156] Obviously, the image restoration device 600 provided in the embodiment of the present application can be used as the above Figure 3 The execution subject of the image restoration method shown in FIG. Figure 3 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0157] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0158] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0159] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0160] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a training device for the image processing model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0161] Performing key point detection on a first image of the sample object to obtain first key point data, wherein the first key point data includes key points of a designated part of the sample object in the first image;

[0162] Based on the first key point data, mask processing is performed on a first area of ​​the first image to obtain a sample mask image, where the first area is an area of ​​the designated part in the first image;

[0163] Filling the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a repaired image of the first image;

[0164] Based on the restored image and the first image, parameters of the image processing model are adjusted.

[0165] Alternatively, the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an image restoration device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0166] Performing key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a designated part of the target object in the image;

[0167] Based on the target key point data, mask processing is performed on the target area of ​​the image to obtain a target mask image, where the target area is the area of ​​the designated part in the image;

[0168] The target mask image is filled based on the target key point data through an image processing model to obtain a repaired image of the image.

[0169] The above application Figure 1 The method performed by the training device of the image processing model disclosed in the embodiment shown, or the above-mentioned method of the present application Figure 3The method performed by the image restoration device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0170] The electronic device may also perform Figure 1 The method and the training device of the image processing model are implemented in Figure 1 , Figure 3 Alternatively, the electronic device may also perform the functions of the embodiments shown. Figure 3 The method is implemented in the image restoration device Figure 3 , Figure 4 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0171] Of course, in addition to software implementation, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0172] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0173] Performing key point detection on a first image of the sample object to obtain first key point data, wherein the first key point data includes key points of a designated part of the sample object in the first image;

[0174] Based on the first key point data, mask processing is performed on a first area of ​​the first image to obtain a sample mask image, where the first area is an area of ​​the designated part in the first image;

[0175] Filling the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a repaired image of the first image;

[0176] Based on the restored image and the first image, parameters of the image processing model are adjusted.

[0177] Alternatively, when the instruction is executed by a portable electronic device including a plurality of application programs, the portable electronic device can execute Figure 3 The method of the embodiment shown is specifically used to perform the following operations:

[0178] Performing key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a designated part of the target object in the image;

[0179] Based on the target key point data, mask processing is performed on the target area of ​​the image to obtain a target mask image, where the target area is the area of ​​the designated part in the image;

[0180] The target mask image is filled based on the target key point data through an image processing model to obtain a repaired image of the image.

[0181] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute a training method for an image processing model as provided in an embodiment of the present application or part or all of the steps in an image restoration method as provided in an embodiment of the present application.

[0182] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0183] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0184] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0185] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0186] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A training method for an image processing model, characterized in that: include: Performing key point detection on a first image of the sample object to obtain first key point data, wherein the first key point data includes key points of a designated part of the sample object in the first image; Based on the first key point data, mask processing is performed on a first area of ​​the first image to obtain a sample mask image, where the first area is an area of ​​the designated part in the first image; Filling the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a repaired image of the first image; Based on the restored image and the first image, parameters of the image processing model are adjusted.

2. The method according to claim 1, characterized in that The step of filling the sample mask image based on the first key point data and the reference image of the designated part to obtain a repaired image of the first image includes: Extracting features from the reference image to obtain texture features, where the texture features are features of the designated part in the reference image; Performing feature extraction on the sample mask image to obtain a feature image; Based on the first key point data, the texture feature and the feature image are fused to obtain a restored image of the first image.

3. The method according to claim 2, characterized in that The fusing process of the texture feature and the feature image based on the first key point data to obtain a restored image of the first image includes: Based on the first key point data, determining a target sub-feature image of the feature image, wherein the target sub-feature image is a region of the designated part in the feature image; Fusing the target sub-feature image with the texture feature to obtain a first fused feature image; Decoding is performed based on the first fused feature image to obtain the repaired image.

4. The method according to claim 3, characterized in that The decoding based on the first fused feature image to obtain the repaired image includes: splicing the first fused feature image with the feature image to obtain a second fused feature image; The second fused feature image is decoded to obtain the repaired image.

5. The method according to claim 1, characterized in that The adjusting the parameters of the image processing model based on the repaired image and the first image includes: Determining a loss of the image processing model based on the inpainted image and the first image; Based on the loss, parameters of the image processing model are adjusted.

6. The method according to claim 5, characterized in that The first key point data is obtained by performing key point detection on the first image through the image processing model; The step of determining the loss of the image processing model based on the restored image and the first image includes: Determining a first loss of the image processing model based on the inpainted image and the sample mask image; determining a second loss of the image processing model based on the first key point data and the key point labels of the first image; Based on the first loss and the second loss, a loss of the image processing model is determined.

7. The method according to claim 5, characterized in that The first key point data is obtained by performing key point detection on the first image through the detection network of the image processing model; the repaired image is obtained by filling the sample mask image through the repair network of the image processing model; Before performing key point detection on the first image of the sample object to obtain first key point data, the method further includes: Performing key point detection on the second image of the sample object by the detection network to obtain second key point data, wherein the second key point data includes key points of the designated part in the second image; Adjusting parameters of the detection network based on the second key point data and the key point labels of the second image; The adjusting the parameters of the image processing model based on the loss includes: adjusting the parameters of the restoration network based on the loss.

8. An image restoration method, characterized in that: include: Performing key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a designated part of the target object in the image; Based on the target key point data, mask processing is performed on the target area of ​​the image to obtain a target mask image, where the target area is the area of ​​the designated part in the image; The target mask image is filled based on the target key point data through an image processing model to obtain a repaired image of the image.

9. A training device for an image processing model, characterized in that: include: A first detection module, configured to perform key point detection on a first image of a sample object to obtain first key point data, wherein the first key point data includes key points of a specified part of the sample object in the first image; A first mask module, configured to perform mask processing on a first region of the first image based on the first key point data to obtain a sample mask image, wherein the first region is a region of the designated part in the first image; A first restoration module, configured to fill the sample mask image based on the first key point data and the reference image of the designated part through an image processing model to obtain a restored image of the first image; A training module is used to adjust the parameters of the image processing model based on the restored image and the first image.

10. An image restoration device, characterized in that: include: A second detection module is used to perform key point detection on the image of the target object to obtain target key point data, wherein the target key point data includes key points of a specified part of the target object in the image; A second mask module is used to perform mask processing on a target area of ​​the image based on the target key point data to obtain a target mask image, wherein the target area is an area of ​​the designated part in the image; The second restoration module is used to fill the target mask image based on the target key point data through an image processing model to obtain a restored image of the image.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the image processing model as described in any one of claims 1 to 7; or, the processor is configured to execute the instructions to implement the image restoration method as described in claim 8.

12. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image processing model as described in any one of claims 1 to 7; or, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image restoration method as described in claim 8.

13. A computer program product, characterized in that The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the training method of the image processing model as described in any one of claims 1 to 7 or the image restoration method as described in claim 8.

Images