A face image repairing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a face image repairing method and device, electronic equipment and storage medium. The method comprises: obtaining a face image to be repaired; cropping a first face image from the face image to be repaired by a face detection algorithm; analyzing the first face image by a face analysis model to generate a binary face mask image; performing face repairing processing on the first face image by a face repairing model to obtain a second face image; synthesizing a third face image according to the first face image, the second face image and the binary face mask image; and replacing the first face image in the face image to be repaired with the third face image to form a repaired face image. The method analyzes and performs face repairing processing on the face image by the face analysis model and the face repairing model respectively to obtain the binary face mask image and the second face image. Different regions of the face image are selectively repaired according to the binary face mask image, so that the repaired face image is more natural.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method, apparatus, electronic device, and storage medium for facial image restoration. Background Technology

[0002] With the rapid development of the video industry, people have increasingly higher demands for video image quality. There are many image enhancement techniques in the field of computer vision, such as deblurring, denoising, and deartening. However, these algorithms are often global enhancement algorithms, only able to handle a specific type of distortion, resulting in limited and unfocused enhancement effects.

[0003] On the other hand, facial areas in video footage are often regions of interest (ROIs). Therefore, the need for face restoration in video footage to enhance image quality has arisen. Existing face restoration algorithms often enhance the image quality of the area surrounding the face based on face detection. Since non-face areas in the image are not enhanced, a significant difference appears between the enhanced and unenhanced images at the region boundaries, creating a sense of incongruity and a poor viewing experience. This phenomenon is known as the boundary problem in face restoration. Therefore, how to solve the boundary problem in face restoration and provide viewers with a more natural and smooth viewing experience has become a pressing technical problem in the industry. Summary of the Invention

[0004] This disclosure provides a face image restoration method, apparatus, electronic device, and storage medium. The method introduces a face analysis model, constructs a binary face mask based on the face region, and then performs differentiated face restoration for different regions in the face image, solving the boundary problem in face restoration and bringing a more natural and smooth face restoration effect.

[0005] According to a first aspect of the present disclosure, a method for face image restoration is provided, comprising: acquiring a face image to be restored; cropping a first face image from the face image to be restored using a face detection algorithm; parsing the first face image using a pre-trained face parsing model to generate a binarized face mask image; the binarized face mask image including at least a target restoration region and a non-target restoration region; performing face restoration processing on the first face image using a pre-trained face restoration model to obtain a second face image; synthesizing a third face image based on the first face image, the second face image, and the binarized face mask image; in the third face image, the region corresponding to the target restoration region uses the second face image, and the region corresponding to the non-target restoration region uses the first face image; replacing the first face image in the face image to be restored with the third face image to form a restored face image.

[0006] In some exemplary embodiments of this disclosure, the step of parsing the first face image using a pre-trained face parsing model to generate a binarized face mask image includes: parsing the first face image using the pre-trained face parsing model, and marking each pixel in the first face image with a corresponding face region value; each face region value corresponds to a pre-defined face region category; according to the pre-defined correspondence between the face region category and the target restoration region, setting the pixels corresponding to the face region category as the target restoration region or a non-target restoration region; performing binarization processing on the first face image, setting the pixels corresponding to the target restoration region as first pixel values, and setting the pixels corresponding to the non-target restoration regions as second pixel values, thereby forming the binarized face mask image.

[0007] In some exemplary embodiments of this disclosure, the training method of the face parsing model includes: acquiring a face parsing training dataset; the face parsing training dataset includes at least: a face image and first face region annotation information; parsing the face image using the face parsing model to be trained to obtain second face region annotation information; using the distance function between the first face region annotation information and the second face region annotation information as the loss function of the face parsing model to be trained; and training the face parsing model to be trained using the face parsing training dataset to obtain the face parsing model.

[0008] In some exemplary embodiments of this disclosure, the method further includes: randomly occluding the face image to obtain an occluded face image; and parsing the face image using a face parsing model to be trained to obtain second face region annotation information, which includes: parsing the occluded face image using the face parsing model to be trained to obtain the second face region annotation information.

[0009] In some exemplary embodiments of this disclosure, obtaining the face parsing training dataset includes: obtaining the face parsing training dataset; the face parsing training dataset includes at least: a first face parsing training dataset and a second face parsing training dataset; training at least two data annotation models using the first face parsing training dataset; the data annotation models are used to parse the face image to obtain face region annotation information; parsing the face image in the second face parsing training dataset using the at least two data annotation models to obtain third face region annotation information respectively; updating the first face region annotation information in the second face parsing training dataset based on the intersection of the third face region annotation information obtained by each of the data annotation models.

[0010] In some exemplary embodiments of this disclosure, the training method of the face restoration model includes: acquiring a face restoration training dataset; the face restoration training dataset includes at least a corresponding first face restoration image and a second face restoration image; performing face restoration processing on the first face restoration image using the face restoration model to be trained to obtain a third face restoration image; the face restoration model to be trained is constructed based on an encoder-decoder model framework; an encoding model is used to convert the first face restoration image into a feature vector; a decoding model is used to convert the feature vector into the third face restoration image; a loss function of the face restoration model to be trained is constructed based on the second face restoration image corresponding to the first face restoration image and the third face restoration image; an adversarial judgment model is used to judge the image authenticity of the third face restoration image through a pre-trained adversarial judgment model to obtain adversarial judgment information; and the face restoration model to be trained is trained using the face restoration training dataset based on the loss function and the adversarial judgment information to obtain the face restoration model.

[0011] In some exemplary embodiments of this disclosure, cropping a first face image from the face image to be repaired using a face detection algorithm includes: determining the position coordinates of a face rectangle in the face image to be repaired using the face detection algorithm; and cropping the first face image from the face image to be repaired based on the position coordinates of the face rectangle.

[0012] According to a second aspect of the present disclosure, a face image restoration apparatus is provided, comprising: a face image acquisition module configured to acquire a face image to be restored; a face detection module configured to crop a first face image from the face image to be restored using a face detection algorithm; and a face parsing module configured to parse the first face image using a pre-trained face parsing model to generate a binarized face mask image; wherein the binarized face mask image includes at least a target restoration region and a non-target restoration region; and a face restoration module configured to use a pre-trained face parsing model to generate a binarized face mask image. A face restoration model performs face restoration processing on the first face image to obtain a second face image; a face image synthesis module is configured to synthesize a third face image based on the first face image, the second face image, and a binarized face mask image; in the third face image, the region corresponding to the target restoration region uses the second face image, and the region corresponding to the non-target restoration region uses the first face image; a face image output module is configured to replace the first face image in the face image to be restored with the third face image to form a restored face image.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement any of the facial image restoration methods described above.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the facial image restoration methods described herein.

[0015] The face image restoration method provided in this disclosure analyzes and restores a face image using a face analysis model and a face restoration model, respectively, to obtain a binarized face mask and a restored face image. Based on the binarized face mask, different regions in the face image are selected for restoration or preservation of the original image, thereby solving the boundary problem in face restoration and making the restored face image more natural.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] Figure 1 A schematic diagram of an exemplary system architecture to which the methods of embodiments of this disclosure can be applied is shown.

[0019] Figure 2 This is a flowchart illustrating a face image restoration method according to an exemplary embodiment.

[0020] Figure 3 This is a flowchart illustrating the generation of a binary face mask, as shown in the example.

[0021] Figure 4 This is a schematic diagram illustrating the generation of a binarized face mask, as shown in the example.

[0022] Figure 5 This is a flowchart illustrating a face analysis model training method.

[0023] Figure 6 This is a flowchart illustrating the process of obtaining a face analysis training dataset, as shown in the example.

[0024] Figure 7 This is a flowchart illustrating a face restoration model training method.

[0025] Figure 8 This is a block diagram illustrating a face image restoration apparatus according to an exemplary embodiment.

[0026] Figure 9 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure, according to an exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0029] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0030] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0031] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0032] Figure 1 A schematic diagram of an exemplary system architecture to which the methods of embodiments of this disclosure can be applied is shown.

[0033] like Figure 1 As shown, the system architecture may include server 101, network 102, terminal device 103, terminal device 104, and terminal device 105. Network 102 serves as the medium for providing a communication link between terminal device 103, terminal device 104, or terminal device 105 and server 101. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0034] Server 101 can be a server that provides various services, such as a back-end management server that supports the devices operated by users using terminal devices 103, 104, or 105. The back-end management server can analyze and process received requests and other data, and feed back the processing results to terminal devices 103, 104, or 105.

[0035] Terminal devices 103, 104, and 105 can be smartphones, tablets, laptops, desktop computers, smart speakers, wearable smart devices, virtual reality devices, augmented reality devices, etc., but are not limited to these.

[0036] It should be understood that Figure 1 The number of terminal devices 103, 104, 105, network 102, and server 101 in the diagram is merely illustrative. Server 101 can be a single physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0037] The steps of the method in the exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings and examples.

[0038] Figure 2 This is a flowchart illustrating a face image restoration method according to an exemplary embodiment. Figure 2 The method provided in the embodiments can be executed by any electronic device, such as the one described above. Figure 1 Terminal devices in or Figure 1 The server in, or Figure 1 The terminal devices and servers in the process are executed together, but this disclosure does not limit this.

[0039] In step S210, the face image to be repaired is obtained.

[0040] In this embodiment of the disclosure, an image of a face to be repaired containing at least one face is acquired. This face includes, but is not limited to, a frontal face, a three-quarter view face, a side view face, a top-down view face, and a bottom-up view face. The image of the face to be repaired can be a frame or keyframe from a video, or it can be a single image of the face to be repaired.

[0041] In step S220, a first face image is cropped from the face image to be repaired using a face detection algorithm.

[0042] In this embodiment of the disclosure, since the face may only be located in a local area of ​​the image to be repaired, processing the entire image would affect both processing efficiency and processing results. Therefore, it is necessary to identify the location of the face to be repaired in the image using a face detection algorithm, and then crop out the face to be repaired and its surrounding local image, which is the first face image. This first face image is typically a rectangular bounding box region surrounding the face to be repaired. This step S220 may specifically include the following steps.

[0043] In step S221, the position coordinates of the face rectangle in the face image to be repaired are determined by the face detection algorithm.

[0044] This face detection algorithm can be denoted as φ. FaceDetection The face image to be repaired can be denoted as I.src The coordinates B of the bounding box of the face in the image to be repaired. face It can be expressed by the following formula.

[0045] φ FaceDetection (I src ) = B face

[0046] Wherein, position coordinate B face It can usually be represented by two sets of coordinates, namely B face = [x1, y1, x2, y2]. Where the coordinates [x1, y1] and [x2, y2] are the coordinates of the two diagonally opposite vertices of the face image rectangle, such as the top left vertex and the bottom right vertex.

[0047] In step S222, the first face image is cropped from the face image to be repaired based on the position coordinates of the face rectangle.

[0048] Based on the location coordinates B face From the face image to be repaired I src The first face image is cropped from the middle. face .

[0049] In this embodiment, the face detection algorithm can be a pre-built model based on a neural network (such as RetinaNet) and trained using a specific data training set. Alternatively, it can be an existing face detection algorithm, such as MTCNN or CenterFace. Any algorithm that can determine the face location from the image of the face to be repaired and crop out the surrounding local image can be considered within the scope of protection of this disclosure.

[0050] In step S230, the first face image is parsed using a pre-trained face parsing model to generate a binarized face mask image; the binarized face mask image includes at least a target restoration region and a non-target restoration region.

[0051] In this embodiment of the disclosure, the first face image I cropped in step S220 is... face The process is divided into two paths. One path uses a pre-trained face parsing model φ. FaceParsing For the first face image I face The data is analyzed to generate a binary face mask image I. mask It can be expressed by the following formula.

[0052] φ FaceParsing (I face ) = I mask

[0053] The face analysis model is a pre-built model based on a neural network (such as U2-Net) and trained using a specific training dataset. This trained model is used to analyze face images and identify the regions within the image containing different facial features. These include, for example, the eyes, eyebrows, nose, mouth, ears, and hair. Furthermore, it can also include common accessories and environmental backgrounds surrounding the face, such as the background, hat, earrings, and glasses.

[0054] Based on the recognition of different regions in a face image using a face analysis model, these regions can be further categorized into target restoration regions and non-target restoration regions. Then, based on this division, a binarized face mask is generated. A mask is an image that hides or occludes parts of a given region.

[0055] It should be noted that, depending on the actual application needs, the binarized face mask may include not only the target restoration area and the non-target restoration area, but also other area settings, which should also be considered within the protection scope of this disclosure.

[0056] In step S240, the first face image is processed by a pre-trained face restoration model to obtain a second face image.

[0057] In this embodiment of the disclosure, the first face image I cropped in step S220 is... face The process is divided into two paths. One path uses a pre-trained face restoration model φ. FaceRestoration For the first face image I face Face restoration processing was performed to obtain the second face image I. FaceRestoration It can be expressed by the following formula.

[0058] φ FaceRestoration (I face ) = I FaceRestoration

[0059] In an exemplary embodiment, the face restoration model may be a model pre-built based on a neural network and trained using a specific data training set. The trained face restoration model is used to perform face restoration processing on face images to improve the image quality.

[0060] In an exemplary embodiment, the face restoration model may consist of multiple image enhancement units. Each image enhancement unit performs specialized image enhancement on the face image to achieve comprehensive face restoration processing and improve the image quality. Each image enhancement unit employs different image enhancement techniques to enhance the face image, such as deblurring, denoising, and deartening. It should be noted that the choice of image enhancement units and the processing intensity of each unit can be flexibly configured according to actual needs, and all such configurations should be considered within the scope of this disclosure.

[0061] It should be noted that there are many technical means in the art for enhancing the image quality of facial images, and only two exemplary implementation methods are given above. Those skilled in the art can use other alternative technical means to perform facial restoration processing on the first facial image in this step to improve the relevant image quality and obtain a restored second facial image, which should also be considered within the scope of protection of this disclosure.

[0062] In step S250, a third face image is synthesized based on the first face image, the second face image, and the binarized face mask image; in the third face image, the region corresponding to the target repair region uses the second face image, and the region corresponding to the non-target repair region uses the first face image.

[0063] In this embodiment of the disclosure, the first face image I has been obtained based on the foregoing steps. face Second face image I FaceRestoration And binarized face mask image I mask In step S250, the three images described above are synthesized to generate a third face image I. FaceRestoration In this synthesis process, a second face image I is used for the region corresponding to the target repair area. FaceRestoration The pixel values ​​are used to determine the region corresponding to the non-target repair area, and the first face image I is used. face The pixel value.

[0064] In an exemplary embodiment, the binary face mask image I mask In this method, the pixel value of the target restoration area is set to 255, and the pixel value of the non-target restoration area is set to 0. This synthesized third-party face image I... FaceReconstruction It can be expressed by the following formula.

[0065] I FaceReconstruction =I FaceRestoration (I Mask ==255)+I face (I Mask ==0)

[0066] In step S260, the first face image in the face image to be repaired is replaced with the third face image to form a repaired face image.

[0067] In this embodiment of the disclosure, the position coordinates B of the face rectangle in the face image to be repaired, determined in step S220, are used as the basis. face The original first face image I in the face image to be repaired face Replace with a third-party face image I FaceReconstruction This results in the creation of a restored facial image I. out .

[0068] In this embodiment, the face image restoration method, for a face image cropped from the face image to be restored, firstly, analyzes the face image using a face analysis model to obtain a binarized face mask; secondly, it performs face restoration processing on the face image using a face restoration model to obtain a second face image. By selectively restoring different regions of the face image based on the binarized face mask, the boundary problem in face restoration is solved, resulting in a more natural face restoration result.

[0069] Figure 3 This is a flowchart illustrating the generation of a binary face mask, as shown in the example. Figure 3 As shown, the aforementioned step S230 may also include the following steps.

[0070] In step S310, the first face image is parsed using the pre-trained face parsing model, and each pixel in the first face image is marked with a corresponding face region value; each face region value corresponds to a pre-defined face region category.

[0071] In this embodiment of the disclosure, multiple face region categories are determined before the face parsing model is trained. After training, the face parsing model analyzes the face image, determines the face region category to which each pixel in the face image belongs, and marks each pixel as a face region value corresponding to that face region category. Each face region value corresponds one-to-one with a face region category.

[0072] In an exemplary embodiment, the face image is pre-defined into 16 face region categories, including: background, facial skin, right eyebrow, left eyebrow, right eye, left eye, glasses, right ear, left ear, earring, nose, mouth, upper lip, lower lip, hair, and hat. Each of these 16 face region categories corresponds to a face region value from 0 to 15. The correspondence between face region values ​​and face region categories is shown in Table 1 below.

[0073] 0 1 2 3 4 5 6 7 background Facial skin right eyebrow Left eyebrow right eye Left eye Glasses right ear 8 9 10 11 12 13 14 15 left ear earrings nose oral cavity upper lip lower lip hair hat

[0074] Table 1

[0075] During the training process, the face analysis model identifies and judges the 16 face regions mentioned above in the face image using the training data set, and labels each pixel in the face image with the corresponding face region value based on the judgment result.

[0076] It should be noted that, depending on the actual application needs, the above-mentioned face region categories may be enriched or simplified, and all such simplifications should be considered within the scope of protection of this disclosure.

[0077] In step S320, according to the pre-defined correspondence between the face region category and the target repair region, the pixels corresponding to the face region category are set as the target repair region or a non-target repair region.

[0078] In this embodiment of the disclosure, based on the aforementioned identification and judgment of various face region categories in the face image, the correspondence between different face region categories and target restoration regions or non-target restoration regions is further determined. Based on this, it is determined whether each pixel in the face image belongs to the target restoration region or a non-target restoration region.

[0079] In an exemplary embodiment, as described above, the face image is pre-defined into 16 face region categories. As needed, the background, hair, and hat are designated as non-target restoration regions, while the remaining face region categories are designated as target restoration regions. Based on this, pixels with face region values ​​of 0, 14, and 15 in step S310 are designated as non-target restoration regions, while pixels with face region values ​​of 1 to 13 in step S310 are designated as target restoration regions.

[0080] In step S330, the first face image is binarized, the pixels corresponding to the target repair area are set as the first pixel value, and the pixels corresponding to the non-target repair area are set as the second pixel value, thus forming the binarized face mask image.

[0081] In this embodiment of the disclosure, based on setting the pixels corresponding to the face region category as the target restoration region or the non-target restoration region in step S320, the first face image is binarized, and the pixels corresponding to the target restoration region are set as the first pixel value, and the pixels corresponding to the non-target restoration region are set as the second pixel value, thereby forming the binarized face mask image.

[0082] In an exemplary embodiment, pixels with face region values ​​of 1 to 13 in step S310 are set as target repair regions, and their first pixel value is set to 255; pixels with face region values ​​of 0, 14, and 15 in step S310 are set as non-target repair regions, and their second pixel value is set to 0.

[0083] Figure 4This is a schematic diagram illustrating the generation of a binarized face mask, as shown in the example. Figure 4 As shown, the left image is the first face image; the middle image is the face image after step S310, which parses the face image and marks the face region values ​​for different face region categories; the right image is the binarized face mask image generated after step S330, which binarizes the first face image.

[0084] In this embodiment, the process of the face parsing model recognizing and judging face regions (step S310) and the correspondence between face region categories and target restoration regions are divided into two processes, instead of directly judging target restoration regions or non-target restoration regions based on the face parsing model. This allows for more flexible settings for target and non-target restoration regions. Users can flexibly configure the correspondence between face region categories and target restoration regions based on actual application needs without retraining the face parsing model. This significantly improves the application flexibility of this embodiment.

[0085] Figure 5 This is a flowchart illustrating a face recognition model training method. For example... Figure 5 As shown, the training method for the face analysis model may include the following steps.

[0086] In step S510, a face analysis training dataset is obtained; the face analysis training dataset includes at least: face images and first face region annotation information.

[0087] In this embodiment, a face analysis training dataset can be obtained from publicly available sources. The training data in this dataset should include at least: a face image and first face region annotation information. The face image should be as similar as possible to the aforementioned first face image. The first face region annotation information corresponds to the annotation of different regions of the face image. The division of face regions should correspond to the aforementioned pre-defined face region categories. Based on this training data, the face analysis model is trained to recognize and determine the pre-defined face region categories based on face images.

[0088] In step S520, the face image is analyzed by the face analysis model to be trained to obtain the second face region annotation information.

[0089] In this embodiment of the disclosure, the face parsing model to be trained is a model built based on a neural network (such as a U2-Net network). By parsing the face images in the aforementioned face parsing training dataset, second face region annotation information is obtained. This second face region annotation information is also based on the face region annotation information corresponding to the aforementioned pre-defined face region categories.

[0090] In step S530, the distance function between the first face region annotation information and the second face region annotation information is used as the loss function of the face parsing model to be trained.

[0091] In this embodiment of the disclosure, a loss function for the face parsing model to be trained is constructed based on the distance function between the first face region annotation information and the second face region annotation information.

[0092] In step S540, the face parsing model to be trained is trained using the face parsing training dataset to obtain the face parsing model.

[0093] In this embodiment of the disclosure, the face parsing training dataset and the constructed loss function are used to train the face parsing model to be trained until the training indicators reach the preset requirements, thereby obtaining the face parsing model.

[0094] In this embodiment of the disclosure, the face image restoration method obtains a face analysis model through model training, which can identify different regions of the face and calculate a loss function based on the first face region annotation information with the label, thereby improving the recognition accuracy of the face analysis model.

[0095] Furthermore, existing face recognition models cannot effectively distinguish between occluded faces. If a face is occluded, the occluded area will still be identified as a facial feature rather than as background. To address this technical problem, the face recognition model training method in this disclosure may further include the following steps.

[0096] In step S515, the face image is randomly occluded to obtain an occluded face image.

[0097] In this embodiment of the disclosure, random object occlusion is introduced into the face images in the face parsing training dataset to obtain occluded face images.

[0098] In step S520, the occluded face image is analyzed by the face analysis model to be trained to obtain the second face region annotation information.

[0099] In this embodiment of the disclosure, the face parsing model to be trained does not directly parse the face images in the face parsing training dataset, but parses the occluded face image obtained after processing in step S515 to obtain the second face region annotation information.

[0100] The subsequent processing steps are the same as those in the aforementioned embodiments, and will not be repeated here.

[0101] In this embodiment, random object occlusion is introduced into the face images in the face parsing training dataset, and the face parsing model to be trained then parses the occluded face images. The loss function is still constructed based on the original first face region annotation information and the second face region annotation information in the face parsing training dataset. The face parsing model trained using the above method, in addition to face parsing capabilities, also possesses anti-occlusion ability, enabling more accurate identification of face region categories in face images.

[0102] Furthermore, the aforementioned training method for the face parsing model requires that the annotation information of the first face region in the face parsing training dataset corresponds to a pre-defined face region category. However, face parsing training datasets obtained through public channels may not directly meet the relevant face region annotation requirements. This results in a limited number of face parsing training datasets available for training, affecting the training effect of the final face parsing model. To expand the available face parsing training dataset, this disclosure further provides a method for obtaining a face parsing training dataset.

[0103] Figure 6 This is a flowchart illustrating the process of obtaining a face analysis training dataset, as shown in the example. Figure 6 As shown, the method for obtaining the face analysis training dataset may include the following steps.

[0104] In step S610, the face analysis training dataset is obtained; the face analysis training dataset includes at least: a first face analysis training dataset and a second face analysis training dataset.

[0105] In this embodiment of the disclosure, multiple sets of face analysis training datasets can be obtained from different public channels. Among these face analysis training datasets, a portion that matches a pre-defined face region category well constitutes the first face analysis training dataset; the portion that does not match a pre-defined face region category well constitutes the second face analysis training dataset. Here, the first and second face analysis training datasets do not refer to a specific face analysis training dataset, but rather to two datasets categorized based on their matching degree with the pre-defined face region category.

[0106] In step S620, at least two data annotation models are trained using the first face parsing training dataset; the data annotation models are used to parse the face image to obtain face region annotation information.

[0107] In this embodiment, multiple different models can be constructed based on different network structures. Each constructed model is trained using the first face parsing training dataset to obtain multiple data-annotated models. These data-annotated models are essentially also face parsing models, similarly parsing face images to obtain face region annotation information. However, here they are considered to be constructed based on different network structures to expand the available face parsing training dataset.

[0108] Here, the face region annotation information obtained by parsing through these multiple data annotation models also corresponds to the pre-defined face region categories.

[0109] In step S630, the face images in the second face parsing training dataset are parsed using the at least two data annotation models to obtain third face region annotation information.

[0110] In this embodiment of the disclosure, multiple data annotation models trained as described above are used to parse face images in the second face parsing training dataset to obtain third face region annotation information. Since the model construction and training of these multiple data annotation models correspond to pre-defined face region categories, the parsed third face region annotation information also corresponds to the pre-defined face region categories.

[0111] In step S640, the first face region annotation information in the second face parsing training dataset is updated based on the intersection of the third face region annotation information obtained from each of the data annotation models.

[0112] In this embodiment, the intersection of the third face region annotation information obtained from each data annotation model is taken to obtain face region annotation information for each face image in the second face parsing training dataset, and this face region annotation information corresponds to a pre-defined face region category. Therefore, the first face region annotation information in the second face parsing training dataset is updated based on the face region annotation information obtained from the intersection.

[0113] Through the embodiments of this disclosure, multiple data annotation models can be trained first on a first face parsing training dataset with good data matching. Then, the trained data annotation models are used to perform face parsing on a second face parsing training dataset to update the face region annotation information in the second face parsing training dataset. This completes the face region annotation information in the second face parsing training dataset, which initially had poor data matching, thus matching it with pre-defined face region categories. This second face parsing training dataset then becomes a usable dataset. This method can significantly expand the size of the usable face parsing training dataset, providing a foundation for training face parsing models.

[0114] Figure 7 This is a flowchart illustrating a face restoration model training method. For example... Figure 7 As shown, the training method for the face restoration model may include the following steps.

[0115] In step S710, a face restoration training dataset is obtained; the face restoration training dataset includes at least a corresponding first face restoration image and a second face restoration image.

[0116] In this embodiment, a face restoration training dataset can be obtained from publicly available sources. This dataset includes at least a corresponding first and second face restoration image. The first and second face restoration images should, as far as possible, be face images similar to the aforementioned first face image. Specifically, the first and second face restoration images are corresponding face images with the same identity information. The first face restoration image is a low-quality face image, used as the input image for model training. The second face restoration image is a high-quality face image, used as the label image for model training.

[0117] In step S720, the first face restoration image is processed by the face restoration model to be trained to obtain a third face restoration image; the face restoration model to be trained is built based on an encoder-decoder model framework; the encoder model is used to convert the first face restoration image into a feature vector; the decoder model is used to convert the feature vector into the third face restoration image.

[0118] In this embodiment, the face restoration model to be trained is a model built based on an encoder-decoder model framework. The encoder-decoder model framework includes an encoder model and a decoder model. The encoder model converts the first face restoration image into a fixed-dimensional feature vector. The decoder model converts this feature vector into a third face restoration image. This third face restoration image is the face restoration image generated after being processed by the face restoration model to be trained.

[0119] In an exemplary embodiment, we can pre-train the encoding or decoding model. For example, we can first pre-train the decoding model, using randomly sampled noise as input, a generated face image as output, and a high-quality face image as a label. Through this pre-training process, the decoding model learns the style and texture characteristics of the face in advance, such as the relative positional relationships of facial features and the fine texture of hair strands—such as prior information. In an exemplary embodiment, we use the StyleGANv2 model to construct the pre-trained decoding model.

[0120] By pre-training the decoding model, we first enable it to output high-quality face images. Then, based on the pre-trained decoding model, we perform overall training on the face restoration model to be trained. This allows us to set a higher learning rate for the encoding model and a lower learning rate for the decoding model. With this setup, the encoding model can learn the high-level speech information of low-quality face images more quickly, while the decoding model, already capable of generating high-quality face images, only needs to learn to generate the corresponding face image based on the feature vectors passed from the encoding model. This significantly improves overall training efficiency and effectiveness.

[0121] In step S730, a loss function for the face restoration model to be trained is constructed based on the second face restoration image and the third face restoration image corresponding to the first face restoration image.

[0122] In this embodiment of the disclosure, a loss function for the face restoration model to be trained is constructed based on the distance function between the second face restoration image and the third face restoration image.

[0123] In step S740, the image authenticity of the third face restoration image is judged by a pre-trained adversarial judgment model to obtain adversarial judgment information.

[0124] In this embodiment, to ensure the generated face image is realistic and natural, an adversarial judgment model is introduced. This adversarial judgment model is pre-trained and generated. It does not generate the face image itself, but rather uses training data to judge the authenticity of the image. The adversarial judgment model is used to judge the authenticity of a third-party face restoration image, obtaining adversarial judgment information. This information determines whether the input face image is real or fake. By introducing this adversarial judgment information, the trained face image becomes visually difficult to distinguish from a fake one, thus ensuring the naturalness of the face enhancement effect.

[0125] In an exemplary embodiment, the adversarial judgment model can also be incorporated into the aforementioned pre-training process of the decoding model, which will not be repeated here.

[0126] In step S750, based on the loss function and adversarial judgment information, the face restoration model to be trained is trained using the face restoration training dataset to obtain the face restoration model.

[0127] In this embodiment of the disclosure, based on the loss function and adversarial judgment information, the face restoration training dataset is used to train the face parsing model to be trained until the training indicators reach the preset requirements, thereby obtaining the face restoration model.

[0128] In this embodiment, the face image restoration method obtains a face restoration model through model training. This face restoration model is built on an encoder-decoder framework, and the encoder and decoder models can be trained separately, thereby greatly improving the overall training efficiency and effect. Furthermore, by introducing a dedicated adversarial judgment model to determine image authenticity during training, the resulting face images are visually difficult to distinguish from fake images, thus ensuring the naturalness of the face enhancement effect.

[0129] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0130] Figure 8 This is a block diagram illustrating a face image restoration apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The device 800 may include: a face image acquisition module 810, a face detection module 820, a face parsing module 830, a face restoration module 840, a face image synthesis module 850, and a face image output module 860.

[0131] The face image acquisition module 810 is configured to acquire the face image to be repaired.

[0132] The face detection module 820 is configured to crop out a first face image from the face image to be repaired using a face detection algorithm.

[0133] The face parsing module 830 is configured to parse the first face image using a pre-trained face parsing model to generate a binarized face mask image; the binarized face mask image includes at least a target restoration region and a non-target restoration region.

[0134] The face restoration module 840 is configured to perform face restoration processing on the first face image using a pre-trained face restoration model to obtain a second face image.

[0135] The face image synthesis module 850 is configured to synthesize a third face image based on the first face image, the second face image, and the binarized face mask image; in the third face image, the region corresponding to the target restoration region uses the second face image, and the region corresponding to the non-target restoration region uses the first face image.

[0136] The face image output module 860 is configured to replace the first face image in the face image to be repaired with the third face image to form a repaired face image.

[0137] In some exemplary embodiments of this disclosure, the face parsing module 830 is configured to parse the first face image using a pre-trained face parsing model, and to mark the face region value corresponding to each pixel in the first face image; each face region value corresponds to a pre-defined face region category; according to the pre-defined correspondence between the face region category and the target restoration region, the pixels corresponding to the face region category are set as the target restoration region or a non-target restoration region; the first face image is binarized, and the pixels corresponding to the target restoration region are set as first pixel values, and the pixels corresponding to the non-target restoration region are set as second pixel values, thereby forming the binarized face mask image.

[0138] In some exemplary embodiments of this disclosure, the face parsing module 830 is configured as a training method for the face parsing model, comprising: acquiring a face parsing training dataset; the face parsing training dataset comprising at least: a face image and first face region annotation information; parsing the face image using the face parsing model to be trained to obtain second face region annotation information; using a distance function between the first face region annotation information and the second face region annotation information as a loss function of the face parsing model to be trained; and training the face parsing model to be trained using the face parsing training dataset to obtain the face parsing model.

[0139] In some exemplary embodiments of this disclosure, the face parsing module 830 is configured as a training method for the face parsing model, including: randomly occluding the face image with an object to obtain an occluded face image; and parsing the face image using the face parsing model to be trained to obtain second face region annotation information, including: parsing the occluded face image using the face parsing model to be trained to obtain the second face region annotation information.

[0140] In some exemplary embodiments of this disclosure, the face parsing module 830 is configured to acquire the face parsing training dataset, including: acquiring the face parsing training dataset; the face parsing training dataset includes at least: a first face parsing training dataset and a second face parsing training dataset; training at least two data annotation models using the first face parsing training dataset; the data annotation models are used to parse the face image to obtain face region annotation information; parsing the face image in the second face parsing training dataset using the at least two data annotation models to obtain third face region annotation information respectively; and updating the first face region annotation information in the second face parsing training dataset based on the intersection of the third face region annotation information obtained by each of the data annotation models.

[0141] In some exemplary embodiments of this disclosure, the face restoration module 840 is configured as a training method for the face restoration model, comprising: acquiring a face restoration training dataset; the face restoration training dataset including at least a corresponding first face restoration image and a second face restoration image; performing face restoration processing on the first face restoration image using the face restoration model to be trained to obtain a third face restoration image; the face restoration model to be trained is constructed based on an encoder-decoder model framework; an encoding model is used to convert the first face restoration image into a feature vector; a decoding model is used to convert the feature vector into the third face restoration image; constructing a loss function for the face restoration model to be trained based on the second face restoration image corresponding to the first face restoration image and the third face restoration image; performing image authenticity judgment on the third face restoration image using a pre-trained adversarial judgment model to obtain adversarial judgment information; and training the face restoration model to be trained using the face restoration training dataset based on the loss function and the adversarial judgment information to obtain the face restoration model.

[0142] In some exemplary embodiments of this disclosure, the face detection module 820 is configured to determine the position coordinates of a face rectangle in the face image to be repaired using the face detection algorithm; and to crop out the first face image from the face image to be repaired based on the position coordinates of the face rectangle.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0144] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0145] like Figure 9 As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), and a display unit 940.

[0146] The storage unit stores program code, which can be executed by the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can perform actions such as... Figure 2 The steps shown.

[0147] For example, electronic devices can achieve such Figure 2 The steps shown.

[0148] Storage unit 920 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 921 and / or cache memory 922, and may further include a read-only memory (ROM) 923.

[0149] The storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0150] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0151] Electronic device 900 can also communicate with one or more external devices 970 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0152] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0153] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0154] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the methods described above.

[0155] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0156] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for restoring facial images, characterized in that, include: Obtain the image of the face to be repaired; A first face image is cropped from the face image to be repaired using a face detection algorithm; The first face image is parsed using a pre-trained face parsing model to generate a binarized face mask image; the binarized face mask image includes at least a target restoration region and a non-target restoration region. The first face image is processed by a pre-trained face restoration model to obtain the second face image. A third face image is synthesized based on the first face image, the second face image, and the binarized face mask image; in the third face image, the region corresponding to the target restoration region uses the second face image, and the region corresponding to the non-target restoration region uses the first face image; The first face image in the face image to be repaired is replaced with the third face image to form the repaired face image; The step of parsing the first face image using a pre-trained face parsing model to generate a binarized face mask image includes: The first face image is analyzed by the pre-trained face parsing model, and the face region value corresponding to each pixel in the first face image is marked; each face region value corresponds to a pre-defined face region category. Based on the pre-defined correspondence between the face region category and the target restoration region, the pixels corresponding to the face region category are set as the target restoration region or a non-target restoration region; the correspondence between the face region category and the target restoration region can be flexibly configured according to application requirements. The first face image is binarized, and the pixels corresponding to the target restoration area are set as the first pixel value, and the pixels corresponding to the non-target restoration area are set as the second pixel value, thus forming the binarized face mask image.

2. The method according to claim 1, characterized in that, The training method for the face analysis model includes: Obtain a face analysis training dataset; the face analysis training dataset includes at least: face images and first face region annotation information; The face image is analyzed by the face analysis model to be trained to obtain the second face region annotation information; The distance function between the first face region annotation information and the second face region annotation information is used as the loss function of the face parsing model to be trained; The face analysis model is trained using the face analysis training dataset to obtain the face analysis model.

3. The method according to claim 2, characterized in that, Also includes: The face image is randomly occluded to obtain an occluded face image; The step of parsing the face image using a face parsing model to be trained to obtain second face region annotation information includes: The occluded face image is analyzed using the face analysis model to be trained, and the annotation information of the second face region is obtained.

4. The method according to claim 2, characterized in that, The process of obtaining the face analysis training dataset includes: Obtain the face analysis training dataset; the face analysis training dataset includes at least: a first face analysis training dataset and a second face analysis training dataset; At least two data annotation models are trained using the first face parsing training dataset; the data annotation models are used to parse the face image to obtain face region annotation information; The face images in the second face parsing training dataset are parsed using the at least two data annotation models to obtain the third face region annotation information. Based on the intersection of the third face region annotation information obtained from each of the data annotation models, the first face region annotation information in the second face parsing training dataset is updated.

5. The method according to claim 1, characterized in that, The training method for the face restoration model includes: Obtain a face restoration training dataset; the face restoration training dataset includes at least a corresponding first face restoration image and a second face restoration image; The first face restoration image is processed by a face restoration model to be trained to obtain a third face restoration image; the face restoration model to be trained is built based on an encoder-decoder model framework; the encoder model is used to convert the first face restoration image into a feature vector; the decoder model is used to convert the feature vector into the third face restoration image; The loss function of the face restoration model to be trained is constructed based on the second face restoration image and the third face restoration image corresponding to the first face restoration image; The image authenticity of the third face restoration image is judged by a pre-trained adversarial judgment model to obtain adversarial judgment information; Based on the loss function and adversarial judgment information, the face restoration model is trained using the face restoration training dataset to obtain the face restoration model.

6. The method according to claim 1, characterized in that, The step of cropping a first face image from the face image to be repaired using a face detection algorithm includes: The face detection algorithm is used to determine the position coordinates of the face bounding box in the face image to be repaired. Based on the position coordinates of the face rectangle, the first face image is cropped from the face image to be repaired.

7. A facial image restoration device, characterized in that, include: The face image acquisition module is configured to acquire the face image to be repaired; The face detection module is configured to crop out a first face image from the face image to be repaired using a face detection algorithm; The face parsing module is configured to parse the first face image using a pre-trained face parsing model to generate a binarized face mask image; the binarized face mask image includes at least a target restoration region and a non-target restoration region. The face restoration module is configured to perform face restoration processing on the first face image using a pre-trained face restoration model to obtain a second face image; The face image synthesis module is configured to synthesize a third face image based on the first face image, the second face image, and the binarized face mask image; in the third face image, the region corresponding to the target restoration region uses the second face image, and the region corresponding to the non-target restoration region uses the first face image; The face image output module is configured to replace the first face image in the face image to be repaired with the third face image to form a repaired face image; The face parsing module is configured to parse the first face image using a pre-trained face parsing model, and to mark the face region value corresponding to each pixel in the first face image; each face region value corresponds to a pre-defined face region category; according to the pre-defined correspondence between the face region category and the target restoration region, the pixels corresponding to the face region category are set as the target restoration region or a non-target restoration region. The correspondence between the face region category and the target restoration region can be flexibly configured according to application requirements; The first face image is binarized, and the pixels corresponding to the target restoration area are set as the first pixel value, and the pixels corresponding to the non-target restoration area are set as the second pixel value, thus forming the binarized face mask image.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the executable instructions to implement the face image restoration method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the face image restoration method as described in any one of claims 1 to 6.

Images