Image processing method and model training method

Abstract

The embodiment of the invention provides an image processing method and a model training method. The image processing method comprises the steps that a face image of a digital person to be generated and a template face image are acquired, and the template face image comprises five sense organ template information and facial contour template information; based on the five-sense-organ template information, acquiring a facial five-sense-organ mask in the facial image of the to-be-generated digital human; based on the face contour template information and the face image of the to-be-generated digital person, determining a face contour of the to-be-generated digital person; and rendering the mask of the five sense organs of the human face to the facial contour of the to-be-generated digital human, and generating a human face image of the digital human. Through the image processing method provided by the invention, the efficiency of the digital human generation process and the personalized features of the digital human generation result are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to an image processing method and a model training method. BACKGROUND

[0002] A digital human refers to a virtual image generated by computer technology, which has visual, voice, behavior and other aspects of human characteristics. With the continuous development of computer technology, people's application of digital humans in daily life is also becoming more and more diverse.

[0003] In related technologies, in the process of generating a digital human, a corresponding digital human is usually generated relying on a video uploaded by a user, or a preset digital human template is adjusted to generate a digital human with a single image. However, in the manner of related technologies, the generation efficiency is low, and the generated digital human image is highly homogeneous, which is difficult to match the real image of the user. SUMMARY

[0004] Therefore, embodiments of the present specification provide an image processing method and a model training method. One or more embodiments of the present specification also relate to an image processing apparatus, a model training apparatus, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of embodiments of the present specification, an image processing method is provided, comprising: obtaining a face image of a digital human to be generated and a template face image, wherein the template face image includes facial feature template information and facial contour template information; based on the facial feature template information, obtaining a facial feature mask in the face image of the digital human to be generated; based on the facial contour template information and the face image of the digital human to be generated, determining a facial contour of the digital human to be generated; rendering the facial feature mask to the facial contour of the digital human to be generated to generate a face image of a digital human.

[0006] According to a second aspect of embodiments of the present specification, a model training method is provided, comprising: obtaining sample data, wherein the sample data includes sample facial feature information and sample facial feature parameters; inputting the sample facial feature information into a facial feature parameter determination model to obtain predicted facial feature parameters output by the facial feature parameter determination model; calculating a model loss value according to the predicted facial feature parameters and the sample facial feature parameters; The model parameters of the five-feature parameter determination model are adjusted according to the model loss value, and the five-feature parameter determination model is continuously trained until a model training stop condition is reached, wherein the five-feature parameter determination model is used to determine the five-feature parameters of the digital human to be generated, and the five-feature parameters are used to render a face feature mask to the facial contour of the digital human to be generated.

[0007] According to a third aspect of the embodiments of the present specification, an image processing method applied to a cloud-side device is provided, comprising: receiving a face image of a digital human to be generated sent by an end-side device, and obtaining a template face image, wherein the template face image comprises five-feature template information and facial contour template information; obtaining a face feature mask in the face image of the digital human to be generated based on the five-feature template information; determining a facial contour of the digital human to be generated based on the facial contour template information and the face image of the digital human to be generated; rendering the face feature mask to the facial contour of the digital human to be generated to generate a face image of a digital human; sending the face image to the end-side device.

[0008] According to a fourth aspect of the embodiments of the present specification, a task platform is provided, comprising a request interface and a response unit; The request interface is configured to receive a face image of a digital human to be generated sent by an end-side device. The response unit is configured to obtain a template face image, wherein the template face image comprises five-feature template information and facial contour template information; obtain a face feature mask in the face image of the digital human to be generated based on the five-feature template information; determine a facial contour of the digital human to be generated based on the facial contour template information and the face image of the digital human to be generated; and render the face feature mask to the facial contour of the digital human to be generated to generate a face image of a digital human.

[0009] According to a fifth aspect of the embodiments of the present specification, an image processing device is provided, comprising: The obtaining unit is configured to obtain a face image of a digital human to be generated and a template face image, wherein the template face image comprises five-feature template information and facial contour template information; The processing unit is configured to obtain a face feature mask in the face image of the digital human to be generated based on the five-feature template information; and determine a facial contour of the digital human to be generated based on the facial contour template information and the face image of the digital human to be generated. The generating unit is configured to render the facial feature mask to the face contour of the digital human to be generated, and generate a human face image of the digital human.

[0010] According to a sixth aspect of the embodiments of the present specification, a model training apparatus is provided, comprising: The obtaining unit is configured to obtain sample data, wherein the sample data comprises sample feature information and sample feature parameters. The prediction unit is configured to input the sample feature information into a feature parameter determination model to obtain predicted feature parameters output by the feature parameter determination model. The calculation unit is configured to calculate a model loss value according to the predicted feature parameters and the sample feature parameters. The adjustment unit is configured to adjust model parameters of the feature parameter determination model according to the model loss value, and continue to train the feature parameter determination model until a model training stop condition is reached, wherein the feature parameter determination model is used to determine feature parameters of a digital human to be generated, and the feature parameters are used to render a facial feature mask to a face contour of the digital human to be generated.

[0011] According to a seventh aspect of the embodiments of the present specification, a computing device is provided, comprising: a memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which realize the steps of the above method when executed by the processor.

[0012] According to an eighth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, which realize the steps of the above method when executed by the processor.

[0013] According to a ninth aspect of the embodiments of the present specification, a computer program product is provided, comprising computer programs / instructions, which realize the steps of the above method when executed by the processor.

[0014] According to the image processing method provided by the present specification, the facial image of the digital human to be generated and the feature template information and the face contour information in the template facial image are used, so that the individual information of the digital human to be generated is more accurately captured. On this basis, in the process of determining the face contour according to the face contour information in the template facial image and the facial image of the digital human to be generated, the generated digital human image is similar to the digital human to be generated in overall structure, thereby improving the individualization of the digital human generation method. In addition, the method of generating the facial image of the digital human corresponding to the digital human to be generated according to the facial image of the digital human to be generated is simple in operation, high in digital human generation efficiency, and improves the satisfaction and experience of users. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flow chart of an image processing method is shown according to an embodiment of the present specification; Figure 2 A schematic diagram of a model training method of a facial feature parameter determination model is shown according to an embodiment of the present specification; Figure 3 A schematic diagram of a sample data acquisition method is shown according to an embodiment of the present specification; Figure 4 A flow chart of a model training method is shown according to an embodiment of the present specification; Figure 5 A schematic diagram of a processing process of an image processing method is shown according to an embodiment of the present specification; Figure 6 A schematic diagram of an image processing apparatus is shown according to an embodiment of the present specification; Figure 7 A schematic diagram of a model training apparatus is shown according to an embodiment of the present specification; Figure 8 An architecture diagram of an image processing system is shown according to an embodiment of the present specification; Figure 9 A structural block diagram of a computing device is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it is understood that the scope of the present specification is not limited to the details below.

[0017] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0018] It should be understood that, although the terms first, second, etc. can be employed in describing various information in one or more embodiments of the present specification, the information should not be limited to such terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of one or more embodiments of the present specification, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0019] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in the relevant region, and provide corresponding operation portal for user to choose authorization or refusal.

[0020] Firstly, the nomenclature involved in one or more embodiments of the present specification is explained.

[0021] Mask: the outside of the frame. The mask involved in the present specification is a layer mask. For example, the facial feature mask can be understood as a facial feature layer mask set for different feature parts in the facial features.

[0022] Digital person: can be understood as a virtual character.

[0023] The image processing method provided by the present specification is applied to the scene of generating a virtual character. For example, it is applied to the scene of interacting according to the virtual character.

[0024] With the continuous development of computer technology, in the process of generating a virtual character, it is usually dependent on manual setting. For example, according to the hand-drawn multiple layered materials by the relevant technical personnel, and then through the live2D rendering technology to split, bind and debug, so as to generate a digital person image. In this process, the operation process is time-consuming, high threshold and highly repetitive, which is difficult to respond to the efficient and personalized customization demand in the application process of the digital person.

[0025] In view of the above situation, in the related technology, a video of a user can also be obtained, and a virtual character of the user can be generated according to the video content. However, by using this method, since a video with a long time is needed in the early stage, the processing time is long and the processing efficiency is low. Or select from the existing digital person image template. However, by using this method, the digital person image is highly homogeneous, and the motion of the generated digital person is limited by the template, which makes it difficult to make personalized adjustment, and thus leads to the single digital person image.

[0026] In view of this, in the present specification, an image processing method is provided, and the present specification also relates to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0027] In a specific embodiment provided in the present specification, an image processing method comprises: obtaining a face image of a digital person to be generated and a template face image, wherein the template face image comprises facial feature template information and facial contour template information; obtaining a face feature mask in the face image of the digital person to be generated based on the facial feature template information; determining a facial contour of the digital person to be generated based on the facial contour template information and the face image of the digital person to be generated; rendering the face feature mask to the facial contour of the digital person to be generated to generate a face image of a digital person.

[0028] In a specific embodiment provided in the present specification, a face image containing clear facial contour and facial feature information is obtained, and a template face image containing facial feature template information and facial contour information is obtained. According to the facial feature template information containing different facial features in the template face image, a face feature mask corresponding to different facial features is obtained in the face image of the digital person to be generated. In addition, according to the facial contour template information in the template face image and the face image of the digital person to be generated, the facial contour of the digital person to be generated is determined. On the basis of the face feature mask and the facial contour of the digital person to be generated, a face image of a digital person is generated.

[0029] According to a specific embodiment provided in the present specification, the face image of the digital person to be generated and the facial feature template information and facial contour information in the template face image are used to more accurately capture the individual information of the digital person to be generated, and to improve the individualization of the digital person generation method. Moreover, the way of generating a face image of a digital person corresponding to the face image of the digital person to be generated is simple in operation and high in efficiency of generating a digital person.

[0030] For ease of understanding, the above-mentioned image method is explained and described in the present specification in the manner as shown in Figure 1 .

[0031] Referring to Figure 1 , Figure 1 a flowchart of an image processing method according to an embodiment of the present specification is shown, which comprises steps 102-108.

[0032] Step 102: obtaining a face image of a digital person to be generated and a template face image.

[0033] The template face image includes facial feature template information and face contour template information. The facial feature template information can be understood as including at least position information of facial features in the template face image, and the face contour template information can be understood as including at least size information of a face contour in the template face image.

[0034] The face image can be understood as an image containing a clear face contour and facial feature information.

[0035] The face image of the digital human to be generated can be understood as an image corresponding to the real face image of a user who has a demand for generating a digital human image and is determined by adaptive photographing of the user.

[0036] The template face image can be understood as a pre-set standard face image, and the facial features in the template face image are uniformly and symmetrically distributed.

[0037] The digital human to be generated can be understood as a user who has a demand for generating a virtual human image and uploads a face image for generating a virtual image.

[0038] The facial features can be understood as eye, nose, mouth, eyebrow, and ear.

[0039] In an embodiment of the present specification, a real face image uploaded by a user through photographing is used as the face image of the digital human to be generated. A pre-generated face image containing a standard and symmetric facial feature distribution mode is used as the template face image.

[0040] It should be understood that the acquisition method of the real face image uploaded by the user through photographing in the present specification is a specific example in the present specification. The acquisition method of the face image of the digital human to be generated in the present specification is not limited to this. Similarly, the acquisition method of the template face image in the present specification is also not limited.

[0041] In an embodiment provided in the present specification, the face image of the digital human to be generated is acquired, including: An initial face image is acquired, and left eye position information and right eye position information in the facial features of the initial face image are determined; The initial face image is adjusted so that the left eye position information and the right eye position information are in the same horizontal position, and the adjusted initial face image is used as the face image of the digital human to be generated.

[0042] The initial face image can be understood as an unprocessed face image uploaded by a user. The left eye position information can be understood as information representing the left eye in the initial face image. Specifically, it can be understood as information representing the pixel position and pixel coordinates of the left eye in the face in the form of a pixel point. The right eye position information can be understood as information representing the right eye in the initial face image. Specifically, it can be understood as information representing the pixel position and pixel coordinates of the right eye in the form of a pixel point.

[0043] In a specific embodiment of the present specification, due to the difference in shooting angle or the difference in shooting method, the initial face image uploaded by the user is skewed. However, the template face image used in the present specification is a standard face image with uniform and symmetrical facial features. Therefore, in order to make the digital human image corresponding to the subsequent generated user face image more accurate, whether the initial face image is skewed is determined according to the position information of each part of the facial features. On this basis, in order to facilitate the determination, the left eye position information and the right eye position information in the facial features are used as the basis for determination, and whether the initial face image is skewed is determined by determining whether the horizontal positions of the pixel points representing the horizontal positions in the left eye position information and the right eye position information are the same horizontal coordinate. In the case where the initial face image is skewed, the left eye position information and the right eye position information are adjusted to obtain a corrected initial face image, and the corrected initial face image is used as the face image of the digital human to be generated.

[0044] For example, the horizontal positions of the pixel points corresponding to the left eye pupils and the horizontal positions of the pixel points corresponding to the right eye pupils in the facial features of the obtained initial face image are determined. When the horizontal positions of the pixel points corresponding to the left eye pupils and the horizontal positions of the pixel points corresponding to the right eye pupils are not in the same horizontal position, it is considered that the initial face image is skewed. On this basis, the horizontal positions of the pixel points corresponding to the left eye and the horizontal positions of the pixel points corresponding to the right eye are rotated to make the horizontal positions of the left eye and the right eye in the adjusted initial face image in the same horizontal position.

[0045] According to the specific embodiment provided in the present specification, by determining the horizontal positions of the left eye and the right eye in the initial face image, the initial face image is adjusted to eliminate the misplacement of the subsequent generated digital human caused by the skew of the initial face image, and the naturalness and accuracy of the subsequent digital human image generation are improved.

[0046] Step 104: Based on the facial feature template information, a face feature mask is obtained in the face image of the digital human to be generated.

[0047] The facial feature mask can be understood as a mask image corresponding to each part of the facial features, for example, a mask image corresponding to the eyes, a mask image corresponding to the mouth, and the like. In the mask image corresponding to each part of the facial features, there is no mutual influence between the masks corresponding to other parts of the facial features when adjusting a certain part of the facial features. For example, when adjusting the mask image corresponding to the eyes in the facial features, there is no influence on the mask image corresponding to the nose in the facial features. The facial feature mask can also be understood as a mask image corresponding to each part of the facial features, for example, when adjusting each part of the facial features, different parts are adjusted in the same mask image.

[0048] In an embodiment of the present specification, the key points corresponding to each part of the facial features in the template facial image are determined, so that the contour corresponding to each part of the facial features in the template facial image can be obtained according to the facial feature key points. On this basis, in order to facilitate the accuracy of subsequent segmentation of each part of the facial features in the facial image of the digital person to be generated, according to the facial feature template information corresponding to each part of the facial features in the template facial image, the corresponding facial feature part is obtained in the facial image of the digital person to be generated. The facial feature mask is obtained.

[0049] In an embodiment provided in the present specification, based on the facial feature template information, the facial feature mask in the facial image of the digital person to be generated is obtained, including S1042-S1044: S1042, aligning the facial features of the facial image of the digital person to be generated with the facial features of the template facial image.

[0050] In an embodiment of the present specification, since there is a deviation in the proportion of the facial features and the arrangement position of each part of the facial features in the facial image of the digital person to be generated and the template facial image. Therefore, in order to facilitate the subsequent generation of a digital person that matches the facial feature proportion and arrangement position of the digital person to be generated according to the template facial image, the facial features of the facial image of the digital person to be generated are aligned with the facial features of the template facial image by aligning the facial feature key points. For example, the eyes in the facial image of the digital person to be generated are aligned with the eyes in the template facial image.

[0051] In an embodiment provided in the present specification, aligning the facial features of the facial image of the digital person to be generated with the facial features of the template facial image includes: Determining a facial feature part to be processed.

[0052] The facial feature part to be processed is any part of the facial features.

[0053] obtain first key point information of the to-be-processed facial feature part in a face image of the digital human to be generated, and obtain second key point information of the to-be-processed facial feature part in a template face image.

[0054] The first key point information can be understood as position information of each key point in the key point set of the to-be-processed facial feature part in the face image of the digital human to be generated. The second key point information can be understood as position information of each key point in the key point set of the same facial feature part as the to-be-processed facial feature part in the template face image.

[0055] determine a transformation relationship between the first key point information and the second key point information.

[0056] The transformation relationship can be understood as a translation relationship and / or a scaling relationship between the first key point information and the second key point information.

[0057] adjust the face image of the digital human to be generated according to the transformation relationship, so that the to-be-processed facial feature part in the face image of the digital human to be generated is aligned with the to-be-processed facial feature part in the template face image.

[0058] For example, the to-be-processed facial feature part is an eye, and the above content is exemplarily explained.

[0059] For example, a first key point set constituting an eye is determined in the face image of the digital human to be generated, and first key point information corresponding to the first key point set is determined. The first key point information can represent the position and size of the eye in the face image of the digital human to be generated. A second key point set constituting an eye is also determined in the template face image, and second key point information corresponding to the second key point set is determined. The second key point information can represent the position and size of the eye in the template face image. The offset between the position of the eye in the first key point information and the position of the eye in the second key point information is calculated, and the scaling between the size of the eye in the first key point information and the size of the eye in the second key point information is calculated. The first key point information is adjusted according to the offset and the scaling, so that the adjusted first key point information corresponds to the second key point information. On this basis, the eye part in the face image of the digital human to be generated is aligned with the eye part in the template face image.

[0060] For other parts of the facial feature part, the processing manner described above for the eye part is consistent, and this will not be described one by one in this specification.

[0061] According to the specific implementation provided in the specification, the change relationship is calculated for the key points of any of the five facial parts to be processed, so that the face of the digital person to be generated can be accurately marked on the corresponding position of the template face image, thereby improving the accuracy of subsequent digital person generation.

[0062] In S1044, based on the five facial part template information, the face five facial part mask is obtained by intercepting the face five facial part in the face image of the five facial part aligned digital person to be generated.

[0063] In a specific implementation of the specification, in the face image of the five facial part aligned digital person to be generated, the five facial part positions have been aligned with the five facial part positions in the template face image (for example, the position and proportion of the eyes in the face image of the digital person to be generated are consistent with the position and proportion of the eyes in the template face image). On this basis, in order to enable independent adjustment of the five facial part positions, the five facial part positions are determined by taking the five facial part template information in the template face image as a reference and combining the features of the five facial part positions in the face image of the digital person to be generated. Based on the intercepting boundary, the face five facial part mask corresponding to the digital person to be generated is intercepted. The obtained face five facial part mask includes the five facial part masks obtained by individual interception of the five facial part positions (for example, the face five facial part mask includes at least a mask corresponding to the nose, a mask corresponding to the eyes, and a mask corresponding to the mouth, etc.).

[0064] According to the implementation, in the aligned face image of the digital person to be generated, the five facial part template information is used for interception, thereby avoiding the offset between the five facial part positions in the template face image and the five facial part positions in the actual face image due to errors in the alignment process.

[0065] In a specific implementation provided in the specification, in the face image of the five facial part aligned digital person to be generated, the face five facial part mask corresponding to different five facial part positions in the face five facial part is obtained according to the five facial part template information.

[0066] For ease of understanding, the following examples are used in the specification to explain the way of obtaining the face five facial part mask.

[0067] Taking the face five facial part mask corresponding to the eye position as an example, the template information about the eye position in the template face is used, such as the position, proportion, and angle of the eye position in the template face image. According to the above information, the face five facial part mask of the eye position is intercepted in the face image of the digital person to be generated aligned with the eye position.

[0068] The face five facial part mask of other positions in the five facial part position is intercepted in the same way as the face five facial part mask of the eye position, and thus multiple face five facial part masks corresponding to different five facial part positions are obtained.

[0069] According to the specific implementation provided in the specification, by aligning the five features of the to-be-generated digital human with the five features of the template image, the personalized five features of the to-be-generated digital human are mapped into the standard coordinate system corresponding to the template face image, so as to ensure that the positions, proportions, and angles of the five features meet the digital human face structure specification. In addition, for each five feature part, the face feature mask generated according to the five feature template information is used to separate each part of the five features from other parts of the face on the basis of determining the corresponding area of each part of the five features in the face image of the to-be-generated digital human. This can avoid misoperation on other parts during subsequent adjustment of any part of the five feature parts, and improve the stability of the adjustment of the five feature parts.

[0070] Step 106: determining the face contour of the to-be-generated digital human based on the face contour template information and the face image of the to-be-generated digital human.

[0071] In an embodiment of the specification, the face contour corresponding to the face image of the to-be-generated digital human is determined according to the size of the face contour included in the face contour template information of the template face image and the positions of the key points constituting the face contour.

[0072] For example, the size of the face contour corresponding to the template face image is determined, and the size of the face contour corresponding to the face image of the to-be-generated digital human is determined. The size of the face contour corresponding to the face image of the to-be-generated digital human is adjusted so that the size of the face contour corresponding to the template face image is greater than or equal to the size of the face contour corresponding to the face image of the to-be-generated digital human. On this basis, the face contour of the to-be-generated digital human is cut from the face image of the to-be-generated digital human according to the size of the face contour corresponding to the template face image, so that the cut face contour of the to-be-generated digital human includes the complete face contour of the to-be-generated digital human.

[0073] Step 108: rendering the face feature mask to the face contour of the to-be-generated digital human to generate the face image of the digital human.

[0074] In an embodiment of the specification, as known from the above, the face feature mask is obtained based on the face image of the to-be-generated digital human and the template face image. To obtain a complete face image of the digital human, each part of the five features in the face feature mask is mapped to the face contour of the to-be-generated digital human to obtain a face image that meets the face features of the to-be-generated digital human.

[0075] In an embodiment provided in the specification, rendering the face feature mask to the face contour of the to-be-generated digital human includes S1082-S1086: S1082, determining the five feature parameters of the to-be-generated digital human.

[0076] In one embodiment of this specification, to facilitate rendering of a facial feature mask onto the facial contour of a digital human to be generated, the facial feature parameters of the digital human to be generated are determined to facilitate determining the offset and scaling of the facial feature mask relative to the facial features of the template face image. This facilitates subsequent rendering of the facial feature mask onto the facial contour of the digital human to be generated based on the offset and scaling.

[0077] The facial feature parameters of the digital human to be generated are used to determine the position of the facial feature mask in the facial contour of the digital human to be generated.

[0078] In a specific implementation provided in this specification, determining the facial features parameters of the digital human to be generated includes: Determining facial features information of the digital person to be generated from the facial contour of the digital person to be generated; The facial feature information of the digital person to be generated is input into a facial feature parameter determination model to obtain the facial feature parameters of the digital person to be generated output by the facial feature parameter determination model.

[0079] For example, in the facial contour of the digital person to be generated, the position information and size information of each facial feature of the digital person to be generated are determined, and the position information and size information of each facial feature are input into the facial feature parameter determination model, so as to obtain the offset and scaling amount of each facial feature of the digital person to be generated compared with the corresponding facial feature in the template face image based on the facial feature parameter determination model.

[0080] According to the specific implementation method provided in this specification, the facial features of the digital person to be generated are input into the facial features parameter determination model to obtain the facial features of the digital person to be generated. Compared with the method of manually labeling the facial features, the facial features parameters obtained by using the facial features parameter determination model are more accurate.

[0081] In a specific embodiment provided in this specification, see Figure 2 . Figure 2 A schematic diagram of a model training method for determining facial features parameters provided by an embodiment of this specification is shown. Figure 2 As shown, the method includes steps 202 to 208: Step 202: Obtain sample data.

[0082] The sample data includes sample facial feature information and sample facial feature parameters.

[0083] In an embodiment of the present disclosure, sample facial feature information and sample facial feature parameters are obtained, and a sample pair is obtained according to the correspondence between the sample facial feature information and the sample facial feature parameters, and the sample pair is taken as sample data.

[0084] In an embodiment provided by the present disclosure, the sample data is obtained in the following manner: obtaining a template face image; The template face image includes initial facial feature information and initial facial feature parameters corresponding to the initial facial feature information.

[0085] randomly adjusting the initial facial feature parameters to obtain sample facial feature parameters, and obtaining sample facial feature information corresponding to the sample facial feature parameters; In an embodiment of the present disclosure, any facial feature parameter in the sample facial feature parameters is randomly adjusted (for example, the left eye is horizontally offset by 30 mm), the initial facial feature parameters are re-determined according to the adjusted facial feature parameters to obtain sample facial feature information, and the facial feature parameters and the sample facial feature information are taken as a sample pair.

[0086] The sample facial feature parameters and the sample facial feature data are saved to obtain the sample data.

[0087] For ease of understanding, the above-mentioned manner of obtaining sample data is explained in combination with the content shown in Figure 3

[0088] Figure 3 FIG. 1 shows a schematic diagram of a sample data obtaining method according to an embodiment of the present disclosure.

[0089] As shown in Figure 3 , the first left image shown in Figure 3 is taken as a template face image, and the initial facial feature parameters corresponding to the face image in the first left image can be understood as 0.

[0090] In an example, the left eye parameter corresponding to the left eye of the template face image is adjusted on the basis of the first left image, for example, the left eye horizontal parameter (left_eye x ) of the template face image is left shifted by 30 (-30) to obtain the second left image. In the second left image, since the other facial feature parameters (face_param others ) of the template face image except the left eye parameter are not adjusted, the other facial feature parameters (face_param others =0) of the template face image displayed in the second left image are not adjusted. On this basis, the facial feature key points corresponding to the adjusted template face image (the second left image) are obtained, the data corresponding to the facial feature key points are taken as facial feature data, and the facial feature data and the facial feature parameters are taken as sample data. ​

[0091] In another example, based on the left image, the mouth parameters corresponding to the mouth of the template face image are adjusted. For example, the mouth size corresponding to the mouth is scaled. For example, in the left third image, the mouth size is scaled by 30 (mouth z =30). And keep the other facial features parameters (face_param others ) unchanged (face_param others =0). On this basis, the corresponding facial features key points are obtained for the adjusted template face image (the third image on the left), and the data corresponding to the facial features key points are used as facial features data, and the facial features data and facial features parameters are used as sample data.

[0092] In another example, based on the three left images, the mouth position of the template face image is offset. As shown in the right image, the vertical position of the mouth is shifted down 30 degrees compared to the original mouth position. y =-30). And keep the other facial features parameters (face_param) in the template face image except the size and position of the mouth. others ) unchanged (face_param others =0). On this basis, the corresponding facial features key points are obtained for the adjusted template face image (the first image on the right), and the data corresponding to the facial features key points are used as facial features data, and the facial features data and facial features parameters are used as sample data.

[0093] Step 204: Input the sample facial feature information into a facial feature parameter determination model to obtain predicted facial feature parameters output by the facial feature parameter determination model.

[0094] In a specific embodiment of the present specification, taking the facial feature parameter determination model as a multi-layer perceptron as an example, sample facial feature information is used as input and predicted facial feature parameters are output.

[0095] Step 206: Calculate the model loss value based on the predicted facial feature parameters and the sample facial feature parameters.

[0096] In one embodiment of the present specification, to determine the accuracy of the facial feature parameter prediction model, the predicted facial feature parameters are compared with the sample facial feature parameters, and the loss value is determined based on the difference. The greater the difference, the greater the loss value.

[0097] Step 208: Adjust the model parameters of the facial feature parameter determination model according to the model loss value, and continue to train the facial feature parameter determination model until the model training stop condition is reached.

[0098] The model parameters can be understood as weights, biases, and the like of the model. The model parameters are adjusted to improve the prediction ability of the model.

[0099] In an embodiment of the present specification, according to the loss value, the influence degree of the loss on each parameter of the model is calculated by the back propagation algorithm, and then the model parameters of the model are adjusted, so that the predicted facial feature parameters output by the adjusted model parameters are consistent with the sample facial feature parameters.

[0100] For ease of understanding, the training process of the facial feature parameter determination model described above is exemplarily illustrated in the present specification in combination with the following examples.

[0101] For example, the sample facial feature parameters and sample facial feature information in the obtained sample data are normalized and cleaned, and the sample data is randomly divided into a training set and a validation set. The sample facial feature parameters are input into the multilayer perceptron, and the multilayer perceptron is used to train the sample facial feature information to predict the sample facial feature parameters. The predicted sample facial feature parameters are compared with the sample facial feature parameters to calculate the loss value of the multilayer perceptron. According to the loss value, the parameters of the multilayer perceptron are adjusted so that the error between the facial feature parameters predicted by the adjusted multilayer perceptron and the sample facial feature parameters is less than a set error, and then the training stop condition of the multilayer perceptron is reached. The adjusted multilayer perceptron is used as a facial feature parameter determination model.

[0102] S1084, based on the position of the facial feature mask in the face contour of the to-be-generated digital person, the facial features in the face contour of the to-be-generated digital person are shielded, and a to-be-combined face image is obtained.

[0103] In an embodiment of the present specification, to avoid the interference of redundant information in the face contour of the to-be-generated digital person, and to facilitate subsequent accurate positioning of the to-be-combined digital person to each part of the facial feature, the facial feature in the face contour of the to-be-generated digital person is intercepted according to the facial feature parameter, or the facial feature is covered in the form of a face texture map, and then a to-be-combined face image is obtained.

[0104] For example, there are facial feature bases in the face contour of the to-be-generated digital person. On this basis, if the facial feature shielding is not performed, there is a risk of generating double facial features in the subsequent process of generating a digital person according to the to-be-combined face image.

[0105] In an embodiment of the present specification, shielding the facial feature in the face contour of the to-be-generated digital person based on the position of the facial feature mask in the face contour of the to-be-generated digital person comprises: determine a position of the to-be-processed facial feature part based on a position of the facial feature mask in the face contour of the to-be-generated digital person; In some embodiments, the to-be-processed facial feature part is any one of the facial features. determine reference masking information corresponding to the to-be-processed facial feature part; mask the to-be-processed facial feature part according to the reference masking information based on the position of the to-be-processed facial feature part.

[0106] In some embodiments, the reference masking information can be understood as reference information for masking different parts of the facial features. For example, when the to-be-processed facial feature part is the mouth, the reference masking information can be the skin texture corresponding to the mouth template in the facial feature template information. At this time, the mouth in the to-be-processed facial feature part is masked according to the skin texture corresponding to the mouth template. For another example, when the to-be-processed facial feature part is the nose, the reference masking information can be the skin texture corresponding to the nose template in the facial feature template information. At this time, the nose in the to-be-processed facial feature part is masked according to the skin texture corresponding to the nose template.

[0107] In some embodiments, the position of the to-be-processed facial feature part is determined, and the position of the contour of the to-be-processed facial feature part in the face contour is determined according to the position of the facial feature mask in the face contour of the to-be-generated digital person. For example, the position of the key point corresponding to the mouth in the face contour of the to-be-generated digital person is determined according to the position of the mouth in the facial feature mask in the face contour of the to-be-generated digital person. The mouth is masked by the position of the mouth in the face contour (e.g., the mouth is intercepted so that the position corresponding to the mouth is a transparent layer, or the mouth is filled with skin texture, etc.). The facial features of the to-be-generated digital person are processed in the above-mentioned manner in turn until the facial features of the to-be-generated digital person are completely masked in the above-mentioned manner as shown by the mouth. The part in the face contour that does not contain the facial features is covered with black pixel points, and then the to-be-combined human face image is obtained.

[0108] For ease of understanding, the above-mentioned facial feature masking is explained and described in the following examples in the present specification.

[0109] In an example, the to-be-processed facial feature part is the eye. According to the masking information corresponding to the eye in the reference masking information, the eye is masked so that the display content of the eye in the masked facial features is the skin texture, and other facial feature parts except the eye are removed, and then the to-be-combined human face image corresponding to the eye part is obtained. For other facial feature parts in the to-be-combined human face image except the eye, the same method as the eye part is used to determine in the to-be-combined human face image, and then the to-be-combined human face image corresponding to each part of the facial features is obtained.

[0110] According to the specific implementation provided in the specification, by shielding the facial feature to be processed and accurately positioning the facial feature to be processed, the facial feature to be processed is separated from other facial features, thereby avoiding the influence of other non-processed facial features during the generation of the digital human, improving the accuracy of the generation of the digital human, and making it more in line with the facial features of the digital human to be generated.

[0111] S1086, render the facial feature mask to the face image to be completed.

[0112] In an embodiment of the specification, the facial feature mask corresponding to each part of the facial feature is mapped to the face image to be completed. For example, according to the position of each part of the facial feature in the face image to be completed, the facial feature mask is mapped according to the corresponding position.

[0113] In an embodiment of the specification, rendering the facial feature mask to the face image to be completed includes: determining a facial feature part to be rendered in the face image to be completed; wherein the facial feature part to be rendered is any part of the facial feature; based on the facial feature part to be rendered, determining a facial feature part to be processed mask corresponding to the facial feature part to be rendered in the facial feature mask; rendering the facial feature part to be processed mask to the facial feature part to be rendered in the face image to be completed.

[0114] In an embodiment of the specification, in the face image to be completed, any part of the facial feature is determined as a facial feature part to be rendered. In the facial feature mask, a facial feature part to be processed mask corresponding to the facial feature part to be rendered is determined, and the facial feature part to be processed mask is overlapped with the facial feature part corresponding to the facial feature part to be processed in the face image to be completed. On this basis, the facial feature part to be processed mask is rendered to the face image to be completed.

[0115] The above rendering operation is performed for any facial feature part, thereby realizing the rendering of the facial feature mask to the face image to be completed.

[0116] In an embodiment of the specification, after the facial feature mask is rendered to the face contour of the digital human to be generated, the method further includes: repairing or filling the face contour of the digital human to be generated.

[0117] In an embodiment of the present disclosure, after the facial feature mask is rendered to the face contour of the to-be-generated digital person, in order to avoid errors in the above process, which makes the face contour of the to-be-generated digital person difficult to fit the face contour of the template face image, and further causes flaws in the process of dynamically driving the to-be-generated digital person. Therefore, the face contour of the to-be-generated digital person is repaired or filled in, so that the face contour of the to-be-generated digital person is more natural and fits the face contour of the template face image.

[0118] In an embodiment of the present disclosure, the face contour of the to-be-generated digital person is repaired or filled in, for example, it can be understood as that the face contour of the to-be-generated digital person is filled in according to the face texture of the to-be-generated digital person. The implementation mode of repairing or filling in the face contour of the to-be-generated digital person in the present disclosure is not limited.

[0119] According to the embodiment provided in the present disclosure, according to the obtained facial feature mask, the obtained digital person is repaired or filled in, so that the obtained digital person is more consistent with the face of the to-be-generated digital person, and further avoids exposing flaws in the process of dynamically driving the generated digital person.

[0120] In an embodiment of the present disclosure, the face image of the to-be-generated digital person further includes the hair of the to-be-generated digital person; Before generating the face image of the digital person, it includes: Segmenting the hair of the face image of the to-be-generated digital person to obtain a hair mask; Rendering the hair mask to the face image to generate a face image of the digital person containing hair.

[0121] In an embodiment of the present disclosure, when the face image of the to-be-generated digital person includes hair on the basis of determining the face contour of the to-be-generated digital person, the hair in the face image of the to-be-generated digital person is segmented to obtain a hair mask. On this basis, the hair mask is rendered to the face contour of the to-be-generated digital person to obtain a face contour of the to-be-generated digital person containing a hair layer.

[0122] According to the embodiment provided in the present disclosure, by obtaining the hair mask of the to-be-generated digital person, and rendering the hair mask and the facial feature mask to the face contour of the to-be-generated digital person, the image of the finally generated digital person is consistent with the image corresponding to the to-be-generated digital person, and further realizes the personalized generation of the to-be-generated digital person.

[0123] In the image processing method described above in the specification, the initial face image of the digital person to be generated is corrected according to the template face image, so that the generated digital person is more consistent with the template face image. On this basis, the facial feature mask and the facial contour are extracted from the face image of the digital person to be generated by using the facial feature template information and the facial contour information of the template face image, so that the same template face image is used as the basis in the subsequent generation of the digital person, and the template face image does not need to be set separately, thereby improving the generation efficiency of the template face image. In addition, since the digital person is generated according to the face image of the digital person to be generated, the generated digital person is consistent with the facial features of the digital person to be generated, thereby realizing personalized customization of the generated digital person, and obtaining a digital person that is accurately consistent with the face image of the digital person to be generated.

[0124] In a specific embodiment of the specification, the model training method is described by combining the model training method provided in the specification with the application of the model training method in determining the facial feature parameters of the digital person to be generated. Figure 4 The model training method is described by taking the application of the model training method provided in the specification in determining the facial feature parameters of the digital person to be generated as an example. Figure 4 A flowchart of a model training method provided by an embodiment of the specification is shown, which includes steps 402-408.

[0125] Step 402: Obtain sample data, wherein the sample data includes sample facial feature information and sample facial feature parameters.

[0126] In a specific embodiment of the specification, the sample data is obtained in the following manner: Obtain a template face image, wherein the template face image includes initial facial feature information and initial facial feature parameters corresponding to the initial facial feature information; Randomly adjust the initial facial feature parameters to obtain sample facial feature parameters, and obtain sample facial feature information corresponding to the sample facial feature parameters; Save the sample facial feature parameters and the sample facial feature data to obtain the sample data.

[0127] Step 404: Input the sample facial feature information into the facial feature parameter determination model to obtain predicted facial feature parameters output by the facial feature parameter determination model.

[0128] Step 406: Calculate a model loss value according to the predicted facial feature parameters and the sample facial feature parameters.

[0129] Step 408: Adjust the model parameters of the facial feature parameter determination model according to the model loss value, and continue to train the facial feature parameter determination model until a model training stop condition is reached.

[0130] The five-want parameter determination model is used for determining five-want parameters of the digital person to be generated, and the five-want parameters are used for rendering a human face five-want mask to a facial contour of the digital person to be generated.

[0131] In a specific embodiment provided in the specification, sample data containing sample five-want information and sample five-want parameters are acquired. The sample five-want information is taken as an input of a five-want parameter determination model, and predicted five-want parameters output by the five-want parameter determination model are obtained. A loss value of the model is calculated according to the predicted five-want parameters and the sample five-want parameters, the five-want parameter determination model is adjusted according to the loss value, and the model training stopping condition is reached. At this time, the five-want parameters of the digital person to be generated are obtained according to the five-want parameter determination model reaching the model training stopping condition, and the human face five-want mask is rendered to the facial contour of the digital person to be generated by using the five-want parameters of the digital person to be generated.

[0132] To facilitate understanding of the image processing method provided in the specification, the following describes the image processing method with reference to the accompanying drawings. Figure 5 The image processing method is further described by taking the application of the image processing method in a server as an example. In the image processing method, Figure 5 A processing process schematic diagram of an image processing method provided in an embodiment of the specification is shown, and specifically includes the following contents.

[0133] In the image processing method, Figure 5 In the image processing method, a face image of a user is acquired, and key point information corresponding to a part where a five-want is located in the face image of the user is determined. The key point information corresponding to the part where the eye is located in the five-want is rotated to realize face correction. The face image of the user is adjusted so that the key point corresponding to the left eye and the key point corresponding to the right eye are horizontal, and the adjusted face image is taken as a face image of a digital person to be generated.

[0134] In the face image of the digital person to be generated, the key points corresponding to each part of the five-want are detected respectively, and a part of the key points corresponding to each part of the five-want is selected as a mark point (i.e., a first key point) of the part from the key points corresponding to each part of the five-want. In the template face image, the mark points (i.e., second key points) corresponding to the same parts of the five-want in the face image of the digital person to be generated are determined. The first key point information corresponding to the first key point is compared with the second key point information corresponding to the second key point. For the key points corresponding to any part of the five-want, the scaling and the translation of the first key point information in the face image of the digital person to be generated compared with the second key point information are calculated by least square, so that the key point information corresponding to the same part of the five-want is aligned in the face image of the digital person to be generated and the template face image. In order to facilitate subsequent interception of the face image of the digital person to be generated by using the second key point information in the template face image.

[0135] In the template face image, for the facial feature template information corresponding to each part of the facial features, the facial feature part is intercepted from the aligned face image of the digital human to be generated, and the facial features are migrated to the texture, and then the facial feature mask corresponding to the facial feature part is obtained.

[0136] In addition, by corresponding the face contour key points of the face contour of the digital human to be generated and the face contour template information in the template face image, the face contour of the digital human to be generated is migrated to the texture. That is, the face contour of the digital human to be generated is migrated to the template face image and to the bottom layer of the template face image, so that the face contour of the digital human to be generated without facial features is present in the current template face image, that is, the texture does not contain facial feature rendering.

[0137] In the case that the face contour of the digital human to be generated without facial features is present in the template face image, the facial feature parameter determination model (for example, a multi-layer perception machine) processes the facial feature parameters to obtain the facial feature parameters, and the facial features in the face contour of the digital human to be generated are masked (for example, the facial features on the face contour are smeared), to obtain the completed face image. The facial feature mask corresponding to the facial feature part is rendered to the completed face image (for example, the facial features and the face contour texture are combined), and the facial features of the rendered completed face image are smeared to avoid the generated digital human from exposing defects in the process of dynamic driving. The corresponding hair mask in the face image of the digital human to be generated is obtained, combined with the rendered facial features of the completed face image, and finally rendered to generate the face image of the digital human.

[0138] For the generated face image of the digital human, adjustment is performed, such as color migration of the facial feature parts and the exposed skin of the generated digital human by the facial skin color, to improve the skin color consistency of the finally obtained face image of the digital human and the face image of the digital human to be generated, and then the digital human consistent with the face image of the digital human to be generated is obtained.

[0139] For the above content, the process of obtaining the facial feature parameters by the facial feature parameter determination model refers to the training method shown in the above Figure 2 of the present specification, which will not be described herein again.

[0140] According to the image processing method provided in the present specification, the way of generating the corresponding digital human by the obtained face image improves the efficiency of generating the digital human compared with the way of generating the digital human by obtaining the video. In addition, the digital human is generated according to the face image of the digital human to be generated, so that the image of the generated digital human is consistent with the actual image of the digital human to be generated, and the personalized expression of the generated digital human is realized.

[0141] Corresponding to the above method embodiments, the present specification also provides image processing device embodiments,Figure 6 A structural diagram of an image processing device provided by one embodiment of the present specification is shown. As shown in the figure, the device comprises: Figure 6 An acquisition unit 602 configured to acquire a face image of a to-be-generated digital person, and a template face image, wherein the template face image comprises eye and nose template information and face contour template information; A processing unit 604 configured to acquire a face feature mask in the face image of the to-be-generated digital person based on the eye and nose template information; and determine a face contour of the to-be-generated digital person based on the face contour template information and the face image of the to-be-generated digital person. A generation unit 606 configured to render the face feature mask to the face contour of the to-be-generated digital person to generate a face image of a digital person.

[0142] Optionally, the generation unit 606 is further configured to: determine eye and nose parameters of the to-be-generated digital person, wherein the eye and nose parameters of the to-be-generated digital person are used to determine the position of the face feature mask in the face contour of the to-be-generated digital person; mask the eye and nose in the face contour of the to-be-generated digital person based on the position of the face feature mask in the face contour of the to-be-generated digital person to obtain a to-be-combined face image; render the face feature mask to the to-be-combined face image.

[0143] Optionally, the generation unit 606 is further configured to: determine eye and nose information of the to-be-generated digital person in the face contour of the to-be-generated digital person; input the eye and nose information of the to-be-generated digital person into an eye and nose parameter determination model to obtain the eye and nose parameters of the to-be-generated digital person output by the eye and nose parameter determination model.

[0144] Optionally, the device further comprises a training module configured to: acquire sample data, wherein the sample data comprises sample eye and nose information and sample eye and nose parameters; input the sample eye and nose information into an eye and nose parameter determination model to obtain predicted eye and nose parameters output by the eye and nose parameter determination model; calculate a model loss value according to the predicted eye and nose parameters and the sample eye and nose parameters; adjust model parameters of the eye and nose parameter determination model according to the model loss value, and continue to train the eye and nose parameter determination model until a model training stop condition is reached.

[0145] ​Optionally, the training module is further configured to: obtain a template face image, wherein the template face image comprises initial facial feature information and initial facial feature parameters corresponding to the initial facial feature information; randomly adjust the initial facial feature parameters to obtain sample facial feature parameters and obtain sample facial feature information corresponding to the sample facial feature parameters; save the sample facial feature parameters and the sample facial feature data to obtain the sample data.

[0146] Optionally, the generation unit 606 is further configured to: determine a to-be-processed facial feature part, and determine a position of the to-be-processed facial feature part based on a position of the face feature mask in a face contour of the to-be-generated digital person, wherein the to-be-processed facial feature part is any part of the facial features; determine reference masking information corresponding to the to-be-processed facial feature part; mask the to-be-processed facial feature part according to the reference masking information based on the position of the to-be-processed facial feature part.

[0147] Optionally, the generation unit 606 is further configured to: determine a to-be-rendered facial feature part in the to-be-combined face image, wherein the to-be-rendered facial feature part is any part of the facial features; determine a to-be-processed facial feature part mask corresponding to the to-be-rendered facial feature part in the face feature mask based on the to-be-rendered facial feature part; render the to-be-processed facial feature part mask to the to-be-rendered facial feature part in the to-be-combined face image.

[0148] Optionally, the processing unit 604 is further configured to: align facial features of the face image of the to-be-generated digital person with facial features of the template face image; obtain a face feature mask by intercepting face features in the face image of the to-be-generated digital person after alignment of the facial features based on the facial feature template information.

[0149] Optionally, the processing unit 604 is further configured to: determine a to-be-processed facial feature part, wherein the to-be-processed facial feature part is any part of the facial features; obtain first key point information of the to-be-processed facial feature part in the face image of the to-be-generated digital person, and obtain second key point information of the to-be-processed facial feature part in the template face image; determine a transformation relationship between the first key point information and the second key point information; Adjust a face image of the digital human to be generated according to the transformation relationship, so that a to-be-processed facial feature part in the face image of the digital human to be generated is aligned with a to-be-processed facial feature part in the template face image.

[0150] Optionally, the face image of the digital human to be generated further includes hair of the digital human to be generated. The generation unit 606 is further configured to: perform hair segmentation on the face image of the digital human to be generated to obtain a hair mask; render the hair mask to the face image to generate a face image of the digital human including hair.

[0151] Optionally, the acquisition unit 602 is further configured to: acquire an initial face image and determine left eye position information and right eye position information in facial features of the initial face image; adjust the initial face image so that the left eye position information and the right eye position information are at the same horizontal position, and use the adjusted initial face image as the face image of the digital human to be generated.

[0152] The above is a schematic scheme of the image processing device of the embodiment. It should be noted that the technical scheme of the image processing device belongs to the same concept as the technical scheme of the image processing method described above, and the details of the technical scheme of the image processing device that are not described in detail can be referred to the description of the technical scheme of the image processing method.

[0153] Corresponding to the method embodiments described above, the present specification also provides model training device embodiments, Figure 7 a structural schematic diagram of a model training device is shown. As shown in the figure, Figure 7 the device includes: An acquisition unit 702 is configured to acquire sample data, wherein the sample data includes sample facial feature information and sample facial feature parameters. A prediction unit 704 is configured to input the sample facial feature information into a facial feature parameter determination model to obtain predicted facial feature parameters output by the facial feature parameter determination model. A calculation unit 706 is configured to calculate a model loss value according to the predicted facial feature parameters and the sample facial feature parameters. An adjustment unit 708 is configured to adjust model parameters of the facial feature parameter determination model according to the model loss value, and continue to train the facial feature parameter determination model until a model training stop condition is reached, wherein the facial feature parameter determination model is used to determine facial feature parameters of a digital human to be generated, and the facial feature parameters are used to render a face feature mask to a facial contour of the digital human to be generated.

[0154] The acquisition unit 702 is further configured to: acquire a template face image, wherein the template face image includes initial facial feature information and initial facial feature parameters corresponding to the initial facial feature information; randomly adjust the initial facial feature parameters to obtain sample facial feature parameters and obtain sample facial feature information corresponding to the sample facial feature parameters; save the sample facial feature parameters and the sample facial feature data to obtain the sample data.

[0155] Referring to Figure 8 , Figure 8 An architecture diagram of an image processing system is shown, which can include a client 100 and a server 200; The client 100 is configured to send a face image of a digital human to be generated to the server 200; The server 200 is configured to acquire a template face image, wherein the template face image includes facial feature template information and face contour template information; Based on the facial feature template information, a face facial feature mask is acquired in the face image of the digital human to be generated; Based on the face contour template information and the face image of the digital human to be generated, a face contour of the digital human to be generated is determined; The face facial feature mask is rendered to the face contour of the digital human to be generated to generate a face image of a digital human; and the face image of the digital human is sent to the client 100; The client 100 is further configured to receive the face image of the digital human sent by the server 200.

[0156] The image processing system can include a plurality of clients 100 and a server 200, wherein the client 100 can be referred to as an end-side device, and the server 200 can be referred to as a cloud-side device. The plurality of clients 100 can establish a communication connection through the server 200. In an image processing scenario, the server 200 is used to provide image processing services between the plurality of clients 100. The plurality of clients 100 can respectively act as a sending end or a receiving end to realize communication through the server 200.

[0157] A user can interact with the server 200 through the client 100 to receive data sent by other clients 100 or send data to other clients 100, etc. In an image processing scenario, the user can publish a data stream to the server 200 through the client 100. The server 200 generates a face image of a digital human according to the data stream and pushes the generated face image of the digital human to other clients that establish a communication connection.

[0158] The client 100 and the server 200 establish a connection through a network. The network provides a medium for a communication link between the client 100 and the server 200. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like. The data transmitted by the client 100 can need to be encoded, transcoded, compressed, and the like before being published to the server 200.

[0159] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5) application, or a light application (also known as a small program, a lightweight application), or a cloud application, and the like. The client 100 can be developed based on a software development kit (SDK) provided by the server 200 for the corresponding service, such as an RTC (Real Time Communication) SDK, and the like. The client 100 can be deployed in a computing device, and needs to rely on the device or some APP in the device, and the like. The computing device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the computing device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

[0160] The server 200 can include servers that provide various services, such as servers that provide communication services for multiple clients, servers that provide support for models used on clients for background training, servers that process data sent by clients, and the like. It should be noted that the server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server of cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms, and the like. The server can also be an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.

[0161] It is worth noting that the image processing method provided in the embodiments of the present specification is generally executed by the server, but in other embodiments of the present specification, the client can also have similar functions as the server, so as to execute the image processing method provided in the embodiments of the present specification. In other embodiments, the image processing method provided in the embodiments of the present specification can also be executed by the client and the server together.

[0162] Figure 9 A structural block diagram of a computing device according to an embodiment of the present application is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 through a bus 930, and a database 950 is used to save data.

[0163] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 940 can include one or more of any type of network interface (e.g., a network interface card (NIC)) such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, etc.

[0164] In an embodiment of the present application, the above-mentioned components of the computing device 900 and other components not shown in the figure can be connected to each other, for example, through a bus. It should be understood that Figure 9 the components shown in the figure are only for the purpose of example, and are not a limitation on the scope of the present application. Those skilled in the art can add or replace other components as needed. Figure 9 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the present application. Those skilled in the art can add or replace other components as needed.

[0165] The computing device 900 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 can also be a mobile or stationary server.

[0166] The processor 920 is configured to execute computer program / instructions, which, when executed by the processor, implement the steps of the image processing method.

[0167] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the image processing method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the image processing method.

[0168] An embodiment of the present specification also provides a computer readable storage medium storing computer program / instructions, which, when executed by a processor, implement the steps of the image processing method.

[0169] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments. In particular, the computer readable storage medium embodiment is basically similar to the image processing method embodiment, and thus the description is relatively simple, and the relevant parts can be referred to the description of the image processing method embodiment.

[0170] An embodiment of the present specification also provides a computer program product comprising computer program / instructions, which, when executed by a processor, implement the steps of the image processing method.

[0171] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the image processing method belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the image processing method.

[0172] The above detailed description has shown, described, and pointed out novel features of the application. The foregoing can be implemented in other embodiments or carried out in alternative ways. For example, the order of the steps recited in the claims can be changed or omitted. It is understood that various modifications can be made without departing from the scope of the application, and it is intended to cover all such modifications that fall within the scope of the claims. Further, it is intended that changes in the meaning or scope of any claims or element thereof be made available to the extent fully encompassed by the doctrine of equivalents.

[0173] The computer readable medium can include any entity or apparatus that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0174] It should be noted that the above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0175] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0176] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. Alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Acquire a face image of a digital human to be generated and a template face image, wherein the template face image includes facial features template information and facial contour template information; Based on the facial features template information, obtaining a facial features mask in the facial image of the digital human to be generated; Determining the facial contour of the digital human to be generated based on the facial contour template information and the facial image of the digital human to be generated; The facial features mask is rendered onto the facial contour of the digital human to be generated to generate a facial image of the digital human.

2. The method according to claim 1, rendering the facial features mask to the facial contour of the digital human to be generated, comprising: Determining facial feature parameters of the digital human to be generated, wherein the facial feature parameters of the digital human to be generated are used to determine the position of the facial feature mask in the facial contour of the digital human to be generated; Based on the position of the facial features mask in the facial outline of the digital person to be generated, masking the facial features in the facial outline of the digital person to be generated, to obtain a facial image to be synthesized; Rendering the facial features mask to the face image to be synthesized.

3. The method according to claim 2, wherein determining the facial features parameters of the digital human to be generated comprises: Determining facial features information of the digital person to be generated from the facial contour of the digital person to be generated; The facial feature information of the digital person to be generated is input into a facial feature parameter determination model to obtain the facial feature parameters of the digital person to be generated output by the facial feature parameter determination model.

4. The method according to claim 3, wherein the facial features parameter determination model is obtained by training through the following steps: Get sample data, where The sample data includes sample facial features information and sample facial features parameters; Inputting the sample facial features information into a facial features parameter determination model to obtain predicted facial features parameters output by the facial features parameter determination model; Calculating a model loss value according to the predicted facial feature parameters and the sample facial feature parameters; Adjust the model parameters of the facial feature parameter determination model according to the model loss value, and continue to train the facial feature parameter determination model until the model training stop condition is reached.

5. The method according to claim 4, wherein the sample data is obtained by: Get the template face image, where The template face image includes initial facial feature information and initial facial feature parameters corresponding to the initial facial feature information; Randomly adjusting the initial facial feature parameters to obtain sample facial feature parameters, and obtaining sample facial feature information corresponding to the sample facial feature parameters; The sample facial feature parameters and the sample facial feature data are saved to obtain the sample data.

6. The method according to claim 2, wherein the facial features of the digital human to be generated are masked based on the position of the facial features mask in the facial contour of the digital human to be generated, comprising: Determining facial features to be processed, and determining the position of the facial features to be processed based on the position of the facial features mask in the facial contour of the digital human to be generated, wherein the facial features to be processed are any of the facial features; Determining reference masking information corresponding to the facial features to be processed; Based on the position of the facial features to be processed, the facial features to be processed are masked according to the reference masking information.

7. The method according to claim 2, wherein rendering the facial feature mask to the face image to be synthesized comprises: In the face image to be synthesized, determining a facial feature portion to be rendered, wherein the facial feature portion to be rendered is any one of the facial features; Based on the facial features to be rendered, determining a facial features mask to be processed corresponding to the facial features to be rendered in the facial features mask; The facial features to be processed are masked and rendered to the facial features to be rendered in the face image to be synthesized.

8. The method according to any one of claims 1 to 7, wherein, based on the facial feature template information, obtaining a facial feature mask in the facial image of the digital human to be generated comprises: Aligning the facial features of the face image of the digital human to be generated with the facial features of the template face image; Based on the facial feature template information, the facial features of the digital human to be generated are captured in the facial image after the facial features are aligned to obtain a facial feature mask.

9. The method according to claim 8, aligning the facial features of the facial image of the to-be-generated digital human with the facial features of the template facial image, comprising: Determining a facial feature part to be processed, wherein the facial feature part to be processed is any part of the facial features; Acquire first key point information of the facial features to be processed in the face image of the digital human to be generated, and acquire second key point information of the facial features to be processed in the template face image; Determining a transformation relationship between the first key point information and the second key point information; The facial image of the digital human to be generated is adjusted according to the transformation relationship so that the facial features to be processed in the facial image of the digital human to be generated are aligned with the facial features to be processed in the template facial image.

10. The method according to any one of claims 1 to 7, wherein the facial image of the digital human to be generated further includes the hair of the digital human to be generated; Before generating a digital human face image, include: Performing hair segmentation on the facial image of the digital human to be generated to obtain a hair mask; The hair mask is rendered onto the face image to generate a face image of a digital human including hair.

11. The method according to any one of claims 1 to 7, wherein obtaining a facial image of a digital human to be generated comprises: Obtaining an initial facial image, and determining left eye position information and right eye position information among the facial features of the initial facial image; The initial face image is adjusted so that the left eye position information and the right eye position information are at the same horizontal position, and the adjusted initial face image is used as the face image of the digital human to be generated.

12. A model training method comprising: Acquiring sample data, wherein the sample data includes sample facial feature information and sample facial feature parameters; Inputting the sample facial features information into a facial features parameter determination model to obtain predicted facial features parameters output by the facial features parameter determination model; Calculating a model loss value according to the predicted facial feature parameters and the sample facial feature parameters; Adjust the model parameters of the facial feature parameter determination model according to the model loss value, and continue to train the facial feature parameter determination model until the model training stop condition is reached, wherein the facial feature parameter determination model is used to determine the facial feature parameters of the digital human to be generated, and the facial feature parameters are used to render the facial feature mask to the facial contour of the digital human to be generated.

13. The method according to claim 12, wherein the sample data is obtained by: Get the template face image, where The template face image includes initial facial feature information and initial facial feature parameters corresponding to the initial facial feature information; Randomly adjusting the initial facial feature parameters to obtain sample facial feature parameters, and obtaining sample facial feature information corresponding to the sample facial feature parameters; The sample facial feature parameters and the sample facial feature data are saved to obtain the sample data.

14. An image processing method, applied to a cloud-side device, comprising: Receiving a facial image of a digital human to be generated sent by a terminal device, and obtaining a template facial image, wherein the template facial image includes facial features template information and facial contour template information; Based on the facial features template information, obtaining a facial features mask in the facial image of the digital human to be generated; Determining the facial contour of the digital human to be generated based on the facial contour template information and the facial image of the digital human to be generated; Rendering the facial features mask to the facial contour of the digital human to be generated to generate a facial image of the digital human; The facial image of the digital human is sent to the terminal device.

15. A task platform comprising a request interface and a response unit; The request interface is used to receive the facial image of the digital human to be generated sent by the terminal side device; The response unit is configured to obtain a template face image, wherein the template face image includes facial features template information and facial contour template information; Based on the facial features template information, a facial features mask is obtained from the facial image of the digital human to be generated; based on the facial contour template information and the facial image of the digital human to be generated, the facial contour of the digital human to be generated is determined; and the facial features mask is rendered to the facial contour of the digital human to be generated to generate the facial image of the digital human.

16. An image processing apparatus, comprising: An acquisition unit is configured to acquire a face image of a digital human to be generated and a template face image, wherein the template face image includes facial features template information and facial contour template information; a processing unit configured to obtain a facial feature mask from the facial image of the digital human to be generated based on the facial feature template information; and determine a facial contour of the digital human to be generated based on the facial contour template information and the facial image of the digital human to be generated; The generating unit is configured to render the facial features mask to the facial contour of the digital human to be generated, so as to generate a facial image of the digital human.

17. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 13 are implemented.

18. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the method according to any one of claims 1 to 13.

19. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images