Expression transfer method and apparatus, electronic device and computer readable storage medium

Abstract

An expression transfer method, comprising: acquiring a first facial image containing a target facial expression (S10); inputting the first facial image into an expression feature extraction model to obtain a target expression feature vector outputted by the expression feature extraction model and used for representing the target facial expression, wherein the expression feature extraction model is obtained by training a first network model on the basis of a plurality of sample triples and annotation information corresponding to each sample triple, each sample triple comprises three first sample images, and the annotation information is used for describing facial expressions of the three first sample images in the corresponding sample triple (S20); and inputting the target expression feature vector into an expression transfer model to obtain a second facial image of a target virtual character having the target facial expression outputted by the expression transfer model (S30). According to the expression transfer method provided by the present disclosure, the expression transfer effect can be improved.

Description

Expression migration method, device, electronic device and computer-readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202410172138.3, filed on February 6, 2024, entitled “Expression Migration Method, Device, Electronic Device and Computer-Readable Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to an expression migration method, device, electronic device, and computer-readable storage medium. Background Art

[0004] With the continuous development of image processing technology, facial expression and motion capture technology has emerged. Facial expression and motion capture technology refers to the process of using sensors, cameras, and other imaging equipment to record a person's facial expressions and movements and convert them into a series of expression parameters. These expression parameters can then be used to restore facial expressions, and based on these expression parameters, facial images of virtual characters with corresponding facial expressions can be created.

[0005] Related technologies require users to wear a specific helmet in a specific scenario, record their facial expressions through the helmet, and then use specific software to interpret the recorded facial expression data, transfer the user's facial expressions to the face of a virtual character, and finally output a facial image of the virtual character with the user's facial expression. For example, if the user's facial expression is recorded as laughing, the final output is a facial image of the virtual character with a laughing expression.

[0006] However, due to the limitations of the accuracy and recording effect of the above-mentioned specific software, the above-mentioned related technologies result in a significant difference between the facial expression migration effect to the virtual character's face and the facial expression of the person himself. It is necessary to further manually correct and repair the facial expressions in the virtual character image, which requires a lot of manpower and time costs. Summary of the Invention

[0007] According to one aspect of the present disclosure, a method for expression transfer is provided, which includes: obtaining a first facial image containing a target facial expression; inputting the first facial image into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression; wherein the expression feature extraction model is obtained by training a first network model based on multiple sample triplets and annotation information corresponding to each sample triplet, each sample triplet including three first sample images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet; inputting the target expression feature vector into the expression transfer model to obtain a second facial image of a target virtual character having the target facial expression output by the expression transfer model.

[0008] According to one aspect of the present disclosure, there is provided an expression migration device, which includes: an acquisition module for acquiring a first facial image containing a target facial expression; a feature extraction module for inputting the first facial image into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression; wherein the expression feature extraction model is obtained by training a first network model based on multiple sample triplets and annotation information corresponding to each sample triplet, each sample triplet includes three first sample images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet; a processing module for inputting the expression feature vector into the expression migration model to obtain a second facial image of a target virtual character having the target facial expression output by the expression migration model.

[0009] According to one aspect of the present disclosure, an electronic device is provided, which includes: a memory and a processor, and the memory and the processor are coupled; the memory is used to store one or more computer instructions; and the processor is used to execute one or more computer instructions to implement the expression migration method of any one of the above-mentioned first aspects.

[0010] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the instructions are executed by a processor to implement the expression migration method of any one of the above-mentioned first aspects.

[0011] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the expression migration method of any one of the above-mentioned first aspects when executed by a processor.

[0012] Compared with the related art, the present disclosure has the following advantages:

[0013] The expression transfer method provided by the present disclosure first obtains an arbitrary first facial image containing a target facial expression. Subsequently, the first facial image is input into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression. The first facial image used in the present disclosure can be recorded by an ordinary shooting device, without the need to use specific equipment such as a specific helmet to capture the expression of a real person, and is not subject to any scene restrictions. This greatly improves the diversity and concurrency of facial expression image acquisition and avoids a large amount of labor and time costs. The first facial image is input into the expression feature extraction model to obtain a target expression feature vector for characterizing the target facial expression in the first facial image. The expression feature extraction model is obtained by training a first network model based on multiple sample triplets and the annotation information corresponding to each sample triplet. Each sample triplet includes three facial images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet. The expression feature extraction model disclosed in the present invention can extract any fine-grained, continuous expression features. Moreover, since the annotation information corresponding to the sample triples only contains expression information and does not contain any irrelevant information (such as identity information, background, light, etc.), the expression feature vector extracted by the expression feature extraction model is only related to the expression, thereby achieving the decoupling of the expression feature vector from irrelevant information such as identity information, which can accurately perceive the fine-grained expressions between different identities and realize the expression transfer from any object to any virtual character. Finally, the target expression feature vector is input into the expression transfer model to obtain the second facial image of the target virtual character with the target facial expression output by the expression transfer model. The present invention can greatly improve the effect of expression transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a schematic diagram of a flow chart of an expression transfer method provided by one embodiment of the present disclosure;

[0015] FIG2 is a schematic diagram of an expression feature space visualization provided by one embodiment of the present disclosure;

[0016] FIG3 is a schematic diagram of a process of expression migration provided by one embodiment of the present disclosure;

[0017] FIG4 is a flowchart of an algorithm iteration based on human-machine collaboration and data closed loop provided by one embodiment of the present disclosure;

[0018] FIG5 is a schematic structural diagram of an expression transfer device provided by one embodiment of the present disclosure;

[0019] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure.

[0020] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0021] To make the purposes, advantages, and features of the present disclosure more clear, the present disclosure is described clearly and completely below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure. However, the described embodiments are only some of the embodiments of the present disclosure, not all of them. All other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of the present disclosure.

[0022] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance, or a specific order or precedence. For those skilled in the art, the specific meanings of the above terms in the present disclosure can be understood in specific circumstances. In addition, in the description of the present disclosure, unless otherwise specified, the term "plurality" refers to two or more. The term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. The terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In order to facilitate understanding of the technical solution of the present disclosure, the relevant concepts involved in the present disclosure are first introduced.

[0024] With the rapid development of computer technology, the production of character images in animation, games, metaverse and other fields has been fully transferred from manual frame-by-frame drawing to the use of professional 3D animation production software such as Maya and 3Ds Max.

[0025] Facial motion capture technology, also known as facial expression motion capture technology, is a part of motion capture technology. It refers to the process of using sensors, video cameras, and other equipment to record facial expressions and movements of people and convert them into a series of expression parameters. The converted expression parameters can be used in the production of computer graphics (CG) and animation, so that the facial expressions and movements of people can be restored on the face of virtual characters. Facial expression motion capture technology is widely used in virtual reality, game development, film production, human-computer interaction and other fields. Through facial expression motion capture technology, the facial expressions of virtual characters can be produced based on the facial expressions of real people. Compared with manually created virtual character facial expression frames, since the facial expressions of computer graphics CG characters are derived from the facial expressions of real people, the facial expression effects of the CG characters will be more realistic and delicate, and the efficiency of expression production will be greatly improved.

[0026] Below, the related technologies involved in this disclosure and the problems existing in the related technologies are explained:

[0027] The 3D facial expression motion capture solutions in related technologies mainly rely on commercial systems such as Faceware or Dynamicxyz, and require wearing a special helmet as a hardware device. Facial expression animation production includes the following steps: facial movement recording, data solution and manual repair. Among them, facial movement recording: the actors need to wear a special helmet as a hardware camera system in a specific scene to perform and record facial expressions. Data solution: after the recording is completed, specific commercial software is needed to solve the captured expression data and output facial animation. Manual repair: due to the limitations of equipment accuracy and actual recording, the computer graphics or animations obtained through solution cannot be directly applied, and professional and experienced animators are required to correct, repair and iterate parameters multiple times in the later stage.

[0028] However, this related technical solution still has the following defects: (1) The diversity and concurrency of facial expression images are limited by the number of hardware and commercial software copyrights; (2) Specific helmets and scenes limit the recording flexibility and data richness; (3) The equipment accuracy and recording conditions limit the solved computer graphics or animation, resulting in a large difference between the facial expressions of the virtual character and the facial expressions of the real person after facial expression migration, that is, the actual migration effect of facial expressions is difficult to achieve the expected effect and needs to be repaired later; (4) Animation repair requires a lot of manual post-repair by professional animators, which consumes a lot of manpower and time costs.

[0029] To address at least some of the aforementioned issues and enhance the effectiveness of facial expression transfer, the present disclosure provides a method for transferring facial expressions, a corresponding device for transferring facial expressions, an electronic device capable of implementing the method, and a computer-readable storage medium. The following examples provide detailed descriptions of the aforementioned method, device, electronic device, and computer-readable storage medium.

[0030] In order to make the purpose and technical solution of the present disclosure clearer and more intuitive, the method provided by the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure. It is understood that the following embodiments can exist separately, and the following embodiments and features in the embodiments can be combined with each other when there is no conflict between the embodiments provided in the present disclosure. For the same or similar content, it will not be repeated in different embodiments. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation. In some cases, the steps shown or described can be performed in a different order.

[0031] The present disclosure provides an expression migration method, device, electronic device and computer-readable storage medium. Specifically, the expression migration method of one embodiment of the present disclosure can be executed by a computer device, wherein the computer device can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, etc. The terminal can also include a client, which can be a game application client, a browser client carrying a game program, or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0032] Next, the expression migration method provided in one embodiment of the present disclosure will be described with reference to FIG1 . FIG1 is a flow chart of the expression migration method provided in one embodiment of the present disclosure.

[0033] As shown in FIG1 , the expression transfer method includes steps S10-S30:

[0034] S10: Acquire a first facial image containing a target facial expression.

[0035] The first facial image described above can be a facial image of a person, a facial image of another animal, or a facial image of another virtual object (such as a virtual human character or a virtual animal). This is merely an example, and the present disclosure does not impose any limitations thereto. In other words, the expression transfer method provided in the embodiments of the present disclosure can be applied not only to the transfer of facial expressions on human faces, but also to the transfer of facial expressions on any other object having facial expressions.

[0036] As mentioned above, the first facial image may be obtained by scanning or photographing a two-dimensional image, and the two-dimensional image may be a facial image of a person or other animal captured by a monocular camera. The person or other animal here may be real or may be a virtual image (for example, a virtual character or virtual animal in a three-dimensional (3D) game).

[0037] The target facial expression mentioned above can be any facial expression, such as happiness, sadness, anger, fatigue, etc., which is only an example and is not limited in the present embodiment.

[0038] S20. Input the first facial image into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression; wherein the expression feature extraction model is obtained by training a first network model based on multiple sample triplets and annotation information corresponding to each sample triplet, each sample triplet includes three first sample images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet.

[0039] The first network model described above can be a convolutional neural network. The first network model is trained using multiple sample triplets and the annotation information corresponding to each sample triplet. After training, the first network model becomes an expression feature extraction model. Each sample triplet includes three first sample images with facial expressions, and the annotation information describes the facial expressions of the three facial images in the sample triplet.

[0040] The annotation information described above is only used to describe the facial expression of the first sample image and is unrelated to the background, lighting, identity information, and other information in the facial image. Training the first network model based on this annotation information enables the expression feature extraction model to extract facial expression information from any facial image while removing irrelevant, redundant, and interfering information (such as the person's identity, background, and lighting).

[0041] An optional implementation method of training the first network model to obtain the expression feature extraction model is described in detail, including the following steps S201-S204:

[0042] S201. Obtain multiple sample triplets and labeling information corresponding to each sample triplet, where the sample triplets comprise three first sample images, namely, an anchor sample image, a positive sample image, and a negative sample image. The anchor sample image and the positive sample image are two facial images with the same or similar facial expressions, and the anchor sample image and the negative sample image are two facial images with different or dissimilar facial expressions.

[0043] S202. Input the anchor sample image, positive sample image, negative sample image and their respective annotation information in the sample triplet into the first network model for feature extraction to obtain the third expression feature vector corresponding to the anchor sample image, positive sample image and negative sample image.

[0044] S203. Based on the third loss function, calculate the third loss function value according to the third expression feature vectors corresponding to the anchor sample image, the positive sample image, and the negative sample image.

[0045] S204: Adjusting parameters of the first network model based on the third loss function value to obtain an expression feature extraction model. The parameter adjustment is used to shorten the distance between the anchor sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor sample image and the negative sample image in the expression feature space.

[0046] As mentioned above, the sample triplet (A, P, N) includes three first sample images, which are divided into an anchor sample image (Anchor, A), a positive sample image (Positive, P) and a negative sample image (Negative, N). Among them, the anchor sample image A is a sample image to be classified, that is, a sample image that requires the first network model to classify its facial expression. The positive sample image P is a facial image that belongs to the same facial expression category as the anchor sample, that is, a sample image with a similar or identical facial expression to the anchor sample. The negative sample image N is a facial image that belongs to a different or dissimilar facial expression than the anchor sample image, that is, a facial image with a different or dissimilar facial expression than the anchor sample image. For example, the facial expression in the anchor sample image in the sample triplet is laughing, the facial expression in the positive sample image is smiling, and the facial expression in the negative sample image is crying. The annotation information corresponding to the sample triplet is used to describe the facial expressions of the three first sample images in the sample triplet.

[0047] In an embodiment of the present disclosure, the anchor sample image, positive sample image, negative sample image and their respective annotation information in the sample triplet are input into the first network model for feature extraction to obtain the third expression feature vector corresponding to the anchor sample image, positive sample image and negative sample image.

[0048] In the embodiment of the present disclosure, the purpose of training the first network model based on the sample triples is to train the first network model so that it can achieve, in the expression feature space, the distance between the expression feature vectors of the first sample images with the same or similar facial expressions is shortened, and the distance between the expression feature vectors of the first sample images with different or dissimilar facial expressions is increased. In other words, the goal of optimizing the training of the first network model is to shorten the distance between the anchor sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor sample image and the negative sample image in the expression feature space. For example, if the facial expression in the anchor sample image in the sample triples is laughter, the facial expression in the positive sample image is smiling, and the facial expression in the negative sample image is crying, then after training the first network model, the distance between the expression feature vector corresponding to laughter and the expression feature vector corresponding to smiling in the expression feature space is smaller, and the distance between the expression feature vector corresponding to laughter and the expression feature vector corresponding to crying in the expression feature space is larger.

[0049] In the disclosed embodiment, a third loss function is used to calculate a third loss function value based on the third expression feature vectors corresponding to the anchor sample image, the positive sample image, and the negative sample image. The parameters of the first network model are adjusted based on the third loss function value to obtain an expression feature extraction model. This parameter adjustment is used to reduce the distance between the anchor sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor sample image and the negative sample image in the expression feature space.

[0050] As mentioned above, the third loss function can be a triplet loss function, a contrast loss function, etc., which is only an example and the embodiments of the present disclosure do not impose any limitation on this.

[0051] An optional implementation method is to adjust the parameters of the first network model through the stochastic gradient descent (SGD) method according to the third loss function value until the loss function converges (such as the third loss function value is less than the preset loss threshold), and then determine the trained first network model as the above-mentioned expression feature extraction model.

[0052] In an embodiment of the present disclosure, a first facial image is input into an expression feature extraction model, and a target expression feature vector is obtained as output by the expression feature extraction model, which is used to represent the target facial expression. The distance (e.g., Euclidean distance) between the expression feature vectors of two different first facial images reflects the degree of similarity between the facial expressions of the two first facial images, i.e., a smaller distance indicates a greater degree of similarity between the first facial expressions of the two facial images, and a larger distance indicates a lesser degree of similarity between the first facial expressions of the two facial images.

[0053] For example, the expression feature vector is exemplarily described in conjunction with FIG2 , which is a schematic diagram of an expression feature space visualization provided by one embodiment of the present disclosure.

[0054] Similar expressions tend to cluster in the same region, while expressions vary widely between different regions. As shown in Figure 2, the black point cloud is a visualization of the distribution of facial expression feature vectors. Figure 2 shows four regions and three facial images representing each region. As shown in Figure 2, the facial image in region one displays a calm expression (i.e., no expression), the facial image in region two displays a surprised expression (i.e., mouth wide open, eyes wide open, eyebrows raised); the facial image in region three displays a normal smile (i.e., mouth open, mouth corners raised, eyes curved); and the facial image in region four displays a wide smile (i.e., mouth opened wider, mouth corners raised higher, eyes curved more significantly). It can be seen that the closer the distance between regions three and four, the greater the similarity in facial expression between the facial images in region three and four; the farther the distance between regions one and four, the less similarity in facial expression between the facial images in region three and four.

[0055] Currently, facial expression feature extraction models in related technologies require the pre-defined categories of several expressions, such as smiling, laughing, calm, and crying. These categories are often "discrete." Because many ambiguous expressions fall between different categories and cannot be accurately categorized into a single category, facial expression feature extraction models are prone to inaccurate classification when processing complex expressions, resulting in the extraction of incorrect facial expression feature vectors.

[0056] Compared with related technologies, the training method adopted by the expression feature extraction model provided by the present invention does not require the definition of expression categories. It mainly uses the first network model to train any facial expression image in the form of sample triples to obtain the expression feature extraction model. Therefore, the expression feature extraction model can extract any fine-grained, continuous expression features. Moreover, since the annotation information corresponding to the sample triples only contains expression information and does not contain any irrelevant information (such as identity information, background, light, etc.), the expression feature vector extracted by the expression feature extraction model is only related to the expression, thereby realizing the decoupling of the expression feature vector from irrelevant information such as identity information, so that fine-grained expressions between different identities can also be accurately perceived, and expression migration from any object to any virtual character can be realized.

[0057] S30: Input the target expression feature vector into the expression transfer model to obtain a second facial image of the target virtual character having the target facial expression output by the expression transfer model.

[0058] The expression transfer model is used to output a second facial image of the target virtual character with the target facial expression based on the expression feature vector representing the target facial expression.

[0059] As mentioned above, the model structure of the expression transfer model can be a generative adversarial network, which is only an example and is not limited in any way in the embodiments of the present disclosure.

[0060] The expression transfer method provided by the embodiment of the present disclosure first obtains an arbitrary first facial image containing a target facial expression. Subsequently, the first facial image is input into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression. The first facial image used in the present disclosure can be recorded by an ordinary shooting device, without the need to use specific equipment such as a specific helmet to capture the expression of a real person, and is not subject to any scene restrictions. This greatly improves the diversity and concurrency of facial expression image acquisition and avoids a large amount of labor and time costs. The first facial image is input into the expression feature extraction model to obtain a target expression feature vector for characterizing the target facial expression in the first facial image. The expression feature extraction model is obtained by training a first network model based on multiple sample triplets and the annotation information corresponding to each sample triplet. Each sample triplet includes three facial images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet. The expression feature extraction model disclosed in the present invention can extract any fine-grained, continuous expression features. Moreover, since the annotation information corresponding to the sample triples only contains expression information and does not contain any irrelevant information (such as identity information, background, light, etc.), the expression feature vector extracted by the expression feature extraction model is only related to the expression, thereby achieving the decoupling of the expression feature vector from irrelevant information such as identity information, which can accurately perceive the fine-grained expressions between different identities and realize the expression transfer from any object to any virtual character. Finally, the target expression feature vector is input into the expression transfer model to obtain the second facial image of the target virtual character with the target facial expression output by the expression transfer model. The present invention can greatly improve the effect of expression transfer.

[0061] Based on the above embodiments, the expression transfer method provided by the embodiments of the present disclosure is further described below.

[0062] In an optional implementation manner, the first sample image includes a facial image of a real person and a facial image of a virtual character.

[0063] Compared with the expression transfer scheme based on facial expression motion capture technology, the facial image used in the expression transfer method provided by the embodiment of the present disclosure can be a facial expression image taken with any ordinary camera, without the need for specific equipment for facial expression motion capture, such as wearable devices, and is no longer restricted by specific equipment and recording scenes. Therefore, the present disclosure greatly improves the flexibility and diversity of expression transfer, and avoids the high equipment cost.

[0064] In an optional implementation manner, the expression migration method provided in the embodiment of the present disclosure further includes steps S40-S50:

[0065] S40. Input the second facial image into an expression parameter determination model to obtain target expression driving parameters output by the expression parameter determination model, wherein the expression parameter determination model is used to generate expression driving parameters that cause the face of the three-dimensional model to present the target facial expression.

[0066] The expression parameter determination model is configured to output target expression driving parameters based on the target facial expression in the second facial image. This expression parameter determination model is configured to generate expression driving parameters that cause the face of the three-dimensional model to exhibit the target facial expression. In other words, based on these target expression driving parameters, the face of the three-dimensional model corresponding to the target virtual character can exhibit the target facial expression.

[0067] S50: Input the target expression driving parameters into the three-dimensional model corresponding to the target virtual character, so that the face of the three-dimensional model presents the target facial expression.

[0068] The 3D model corresponding to the target virtual character is a 3D model that includes the target virtual character's face, and can also be a 3D facial model of the target virtual character. This is for illustrative purposes only and is not limited in this disclosure. As long as the 3D model includes the target virtual character's face, it is sufficient. The facial expression of the 3D model corresponding to the target virtual character can be any expression, such as a neutral face, laughter, a smile, or tears. A neutral face is one without any expression, also known as a base face or standard face.

[0069] For example, the expression migration method provided by the embodiment of the present disclosure is exemplarily described in conjunction with FIG3 , which is a flowchart of the expression migration provided by one embodiment of the present disclosure.

[0070] As shown in FIG3 , a first facial image having a facial expression is input into an expression feature extraction model (corresponding to the Expr Extractor in FIG3 ) to obtain a target expression feature vector for representing the facial expression in the first facial image. The target expression feature vector is input into an expression transfer model (corresponding to the Facial Constrcutor in FIG3 ) to obtain a second facial image of a target virtual character having a target facial expression output by the expression transfer model. The second facial image is input into an expression parameter determination model to obtain target expression driving parameters output by the expression parameter determination model. The target expression driving parameters are input into the three-dimensional model corresponding to the target virtual character through a three-dimensional model renderer (corresponding to the Facial Render in FIG3 ) so that the face of the three-dimensional model exhibits the same facial expression as the second facial image.

[0071] In an embodiment of the present disclosure, a second facial image of a target virtual character having a target facial expression is first input into an expression parameter determination model to obtain target expression driving parameters output by the expression parameter determination model. Subsequently, the target expression driving parameters are input into a three-dimensional model corresponding to the target virtual character, and the face of the three-dimensional model exhibits the target facial expression. The three-dimensional model renderer can utilize the generated target expression driving parameters to reconstruct the facial expression of the three-dimensional character, thereby completing the migration of facial expressions. This method does not require matching data between a real person's face and a virtual character, nor does it require the collection or annotation of feature point positions on the face, which greatly reduces the time cost and process complexity of three-dimensional expression migration.

[0072] In an optional implementation manner, the expression migration method provided in the embodiment of the present disclosure further includes steps S301-S302:

[0073] S301: Construct an image generation model and an image discrimination model.

[0074] In the embodiment of the present disclosure, the image generation model may be an adversarial generation model, and the image discrimination model may be a convolutional neural network or a recurrent neural network.

[0075] S302. For each second sample image in the plurality of second sample images, execute the first step to train the image generation model and the image discrimination model until the first convergence condition is met, and determine the trained image generation model as an expression transfer model, and the second sample image is a facial image containing a facial expression.

[0076] The first step includes steps S3021-S3024:

[0077] S3021. Input the second sample image into the expression feature extraction model to obtain a first expression feature vector output by the expression feature extraction model.

[0078] S3022: Input the first facial expression feature vector into an image generation model to obtain a third facial image of the target virtual character having a facial expression output by the image generation model.

[0079] S3023. Input the third facial image and the preset rendered facial image as two input images into the image discrimination model for discrimination, and obtain a discrimination result output by the image discrimination model; wherein the preset rendered facial image is a two-dimensional rendered image obtained by rendering the three-dimensional face model, and the discrimination result is used to indicate the probability value of each input image belonging to the rendered image and the generated image, respectively.

[0080] S3024. Based on the discrimination result and the first loss function, calculate the first loss function value and adjust the parameters of the image generation model and the image discrimination model according to the first loss function value.

[0081] Below, the above steps S3021-S3024 are described in detail.

[0082] As mentioned above, the training process of the expression transfer model involves two models: an image generation model (Generator) and an image discriminator model (Discriminator). The input of the image generation model is the first expression feature vector extracted from the second sample image, and the output is a third facial image of the target virtual character with facial expressions. The goal of the image generation model is to generate images that are as realistic as possible in an attempt to deceive the image discriminator model. The input of the image discriminator model is the third facial image and a preset rendered facial image. The goal of the image discriminator model is to distinguish between the rendered image and the generated image as accurately as possible.

[0083] In the embodiment of the present disclosure, the training steps of the expression transfer model are described below:

[0084] Step 1: Initialize the image generation model and image discrimination model.

[0085] Step 2: Input the second sample image into the expression feature extraction model to obtain a first expression feature vector output by the expression feature extraction model.

[0086] Step 3: Input the first facial expression feature vector into an image generation model to obtain a third facial image of the target virtual character with facial expression output by the image generation model.

[0087] Step 4: Input the third facial image and the preset rendered facial image as input images into the image discrimination model, and obtain a discrimination result output by the image discrimination model. The preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional face model. The discrimination result indicates the probability of each input image belonging to the rendered image and the generated image, respectively.

[0088] The preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional facial model corresponding to any virtual character. The virtual character in the preset rendered facial image can be any virtual character, and the facial expression in the preset rendered facial image can be any facial expression. This disclosure does not impose any restrictions on the virtual character in the preset rendered facial image or the facial expression of the virtual character, as long as the preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional facial model.

[0089] As described above, the discrimination results output by the image discrimination model are used to indicate the probability values ​​of each input image belonging to a rendered image and a generated image, respectively. This probability value can be a probability value between 0 and 1, used to indicate the likelihood that each input image belongs to a rendered image or a generated image. The larger the probability value of belonging to a rendered image, the more likely the corresponding input image is a rendered image; the larger the probability value of belonging to a generated image, the more likely the corresponding input image is a generated image (i.e., a pseudo image) output by the image generation model.

[0090] For example, in the present disclosure, the image discrimination model is fed with an image including a third facial image and a preset rendered facial image. The image discrimination model outputs the following discrimination results: the third facial image has a probability of 0.2 of being a rendered image and a probability of 0.8 of being a generated image; the preset rendered facial image has a probability of 0.9 of being a rendered image and a probability of 0.1 of being a generated image.

[0091] Step 5: Based on the discrimination result and the first loss function, calculate the first loss function value and adjust the parameters of the image generation model and the image discrimination model according to the first loss function value.

[0092] In the above, the first loss function can be a cross entropy loss function, which can be specifically referred to Formula 1:

[0093] Among them, N is the number of input images of the input image discrimination model, y i is the value of the category of the input image i. When the category is a rendered image, y i The value of is 1; when the category is generated image, y i The value of p is 0. i is the probability value of judging that the input image i is a rendered image, p' i To determine the probability value of the input image i being the generated image, the logarithmic base of log can be e, and the specific value of the first loss function L is the first loss function value.

[0094] In the disclosed embodiment, based on the discrimination result and the first loss function, the first loss function value is calculated and the parameters (such as weights) of the image generation model and the image discrimination model are adjusted according to the first loss function value. Specifically, backpropagation and an optimization algorithm (such as the Adam optimization algorithm) can be used to achieve this.

[0095] Repeat steps 2-5 until the first convergence condition is met, i.e., the trained model converges. The image generation model with adjusted parameters is used as the expression transfer model. Specifically, the generated image output by the trained expression transfer model is sufficiently close to the rendered image. The first convergence condition can be, for example, that the first loss function value is less than a preset loss threshold.

[0096] In the disclosed embodiment, by training the image generation model and the image discrimination model, the trained image generation model is capable of outputting a third facial image (i.e., a generated image) of the target virtual character with the target facial expression based on the expression feature vector, and the image quality of the generated image is sufficiently close to that of the rendered image. The trained image generation model is determined as the expression transfer model.

[0097] Compared to related technologies, the expression transfer method provided in the embodiment of the present disclosure inputs the expression feature vector into the expression transfer model, and the expression transfer model can output a third facial image (i.e., a generated image) of the target virtual character with the target facial expression. The expression transfer effect of the present disclosure is good, which can save a large number of manual repair processes for relevant personnel, save a lot of manpower costs and greatly improve the efficiency of expression animation production. Furthermore, the picture quality of the generated image is close enough to the rendered image, so that the expression transfer method provided by the present disclosure can be used to create facial expressions for virtual characters in projects such as animation and game development, which greatly improves the efficiency of creating facial expressions of virtual characters and extremely reduces the threshold for creating facial expressions of virtual characters.

[0098] In an optional implementation manner, before step S40 "inputting the second facial image into the expression parameter determination model to obtain the target expression driving parameters output by the expression parameter determination model", the expression transfer method provided by the embodiment of the present disclosure further includes steps S401-S402:

[0099] S401: Obtain a second network model and a differentiable rendering network model.

[0100] The second network model is configured such that its input is a facial image of a target virtual character with a target facial expression, and its output is a 3D model expression driving parameter corresponding to the target facial expression. The network structure of the second network model can be a convolutional neural network, for example only.

[0101] The aforementioned differentiable neural network, also known as a differentiable neural renderer, is a pre-trained convolutional neural network whose input is expression-driven parameters and whose output is a facial image of an avatar's facial expression. Because the entire convolutional neural network is differentiable and is used to replace analog rendering engines for rendering avatar facial images, it is called a differentiable neural renderer.

[0102] S402. For each virtual character facial image in the plurality of virtual character facial images, repeatedly perform the second step to train the second network model until the first convergence condition is met, and determine the trained second network model as the expression parameter determination model.

[0103] The second step includes S4021-S4023:

[0104] S4021. Input the virtual character's facial image into the second network model to obtain the first expression driving parameter output by the second network model.

[0105] S4022: Input the first expression driving parameter into the differentiable rendering network model to obtain a predicted facial image output by the differentiable rendering network model.

[0106] S4023. Calculate a loss value based on the virtual character's facial image, the predicted facial image, and the second loss function, and adjust parameters of the second network model based on the loss value.

[0107] As mentioned above, the second loss function may be a mean square error loss function (ie, MSE loss function).

[0108] In one optional embodiment, the second loss function includes a parameter constraint sub-function and an image pixel constraint sub-function. The parameter constraint sub-function is used to constrain the first expression driving parameter to be consistent with the second expression driving parameter, where the second expression driving parameter is the expression driving parameter of the three-dimensional model corresponding to the virtual character in the virtual character facial image. The image pixel constraint sub-function is used to constrain the pixels of the virtual character facial image and the predicted facial image to be consistent.

[0109] In the disclosed embodiment, the second loss function consists of two parts: one is the animation parameter constraint, which is used to constrain the generated second expression driving parameters to be consistent with the actual expression driving parameters; the other is the image pixel constraint, which is used to constrain the input image and the image generated by the differentiable renderer to be consistent. This step can be accomplished using backpropagation and optimization algorithms (such as the Adam optimizer).

[0110] In the disclosed embodiments, a differentiable rendering network model is used to train an expression parameter determination model whose input is a virtual character's facial image and whose output is expression driving parameters. Therefore, by inputting the virtual character's facial image into the expression parameter determination model, expression driving parameters are generated that can drive the 3D model to produce the facial expression conveyed by the virtual character's facial image.

[0111] In an optional implementation manner, before step S202 of "inputting the anchor sample image, positive sample image, negative sample image, and their respective annotation information in the sample triple into the first network model for feature extraction to obtain third expression feature vectors corresponding to the anchor sample image, positive sample image, and negative sample image respectively", the expression transfer method provided in the embodiment of the present disclosure further includes steps S501-S503:

[0112] S501: perform labeling processing on the sample triples based on a preset labeling model to obtain first labeling information corresponding to the sample triples output by the preset labeling model.

[0113] S502. Obtain target labeling information corresponding to the sample triples. If the first labeling information and the target labeling information are inconsistent, train the preset labeling model. Repeat the above steps until the second convergence condition is met, and determine the trained preset labeling model as the target labeling model.

[0114] S503: Label multiple sample triples according to the target labeling model to obtain labeling information corresponding to each sample triple.

[0115] The above steps S501-S503 are described in detail below.

[0116] In the disclosed embodiment, a large amount of sample data (i.e., sample triples and corresponding annotation information) is provided through human-computer collaboration to continuously support the iteration and update of the expression feature extraction model and the expression migration model. A large number of sample triples can be manually annotated to obtain the annotation information of the sample triples. In a crowdsourcing scenario (i.e., a large annotation task is divided into multiple small sub-annotation tasks, and the sub-tasks are assigned to different user subjects to complete the annotation tasks), considering that the reliability of the annotation information of a single user is low, multiple users are introduced to participate in the annotation. Relying on interval estimation and truth inference algorithms, the number of annotations is automatically controlled to obtain more reliable annotation information recognized by the public, which can ensure the accuracy and reliability of the annotation information corresponding to the sample triples.

[0117] In the disclosed embodiment, after a large number of expression task annotations, portrait data of each user can be obtained. This portrait data can not only be used for the access, automatic quality inspection and other links of the current expression sample triple task to improve the accuracy of the annotation information, but can also be migrated to more abundant expression-related annotation tasks in the future, and a group of expression-related field experts can be obtained from crowdsourcing users.

[0118] Furthermore, considering that labeling a large number of sample triples requires a lot of manpower and time, in order to reduce labeling costs and improve labeling efficiency, the present disclosure introduces a preset labeling model (i.e., regarded as an AI worker) to label the sample triples and obtain the first labeling information corresponding to the sample triples output by the preset labeling model.

[0119] In an embodiment of the present disclosure, before manual labeling, a preset labeling model can be used to label the sample triples to obtain the first labeling information corresponding to the sample triples. After manually labeling the sample triples, the target labeling information corresponding to the sample triples is obtained. The target labeling information of the sample triples is sent to the preset labeling model. The preset labeling model is trained based on the first labeling information and the target labeling information to gradually improve the labeling accuracy of the preset labeling model. The trained preset labeling model is used to determine the target labeling model. The target labeling model can then be used directly to label the sample triples. In the present disclosure, labeling the sample triples using the target labeling model can reduce the cost of manual labeling and improve the efficiency of labeling the sample triples. For example, if a piece of data originally requires 6 labelers to label, if the labeling results are consistent, it is a valid piece of data. If the first labeling information after the target labeling model is pre-labeled, it can be labeled by 3 labelers first. If the target labeling information labeled by the three labelers is consistent with the first labeling information, it can be considered a valid labeling information, thus saving 50% of the manual labeling cost.

[0120] In the disclosed embodiments, manual labeling can be considered a process of revising and improving the preset labeling model. For example, if the preset labeling model labels a sample triple as '1' while the manual labeling result is '0', the preset labeling model can be considered to have an inaccurate labeling result for that sample triple. Such inaccurately labeled sample triplets can be collected and used to train the preset labeling model to improve its labeling accuracy. Subsequently, the preset labeling model with improved labeling accuracy can be used for pre-labeling again. This process can be repeated, iteratively, to continuously improve its accuracy.

[0121] Exemplarily, in conjunction with FIG4 , the algorithm iteration based on human-computer collaboration and data closed loop is exemplarily described. FIG4 is a flowchart of the algorithm iteration based on human-computer collaboration and data closed loop provided in one embodiment of the present disclosure.

[0122] As shown in Figure 4, the algorithm iteration process includes a data closed-loop process and a business processing process. In the data closed-loop process, the expression feature extraction model and the expression transfer model are trained mainly based on sample triples and corresponding annotation information. Subsequently, in the business processing process, a facial image with a target facial expression (such as laughter) is input, and the expression feature extraction model and expression transfer model trained in the data closed-loop process are used to obtain a facial image of a virtual character with the target facial expression (such as laughter) output by the expression transfer model. It is determined whether the facial image of the virtual character with the target facial expression output by the expression transfer model meets the release requirements. If not, the input facial image is collected. Subsequently, in the data closed-loop process, sample triples are constructed using this type of facial image, and the expression feature extraction model and expression transfer model are iteratively trained.

[0123] The following is a detailed description of the data closed-loop process.

[0124] A data cold start refers to the initial phase of a business, when, without the first batch of data, it is impossible to use the target annotation model or facial expression motion capture results to screen facial images. At this time, constructing sample triplets requires completely random sampling, and the annotation process relies solely on manual labeling. During the data cold start phase, manual labeling is used to annotate sample triplets to obtain annotation information. During the data hot start phase, sample triplets can be annotated using a collaborative approach of manual labeling and the target annotation model, or they can be annotated solely using the target annotation model.

[0125] The sample triples and corresponding annotation information are input into an expression feature extraction model, which is then trained to enable it to extract facial expression features. The expression feature vectors output by the expression feature extraction model are input into an expression transfer model, which is then trained to enable it to transfer expressions, i.e., to output a facial image of a target virtual character with a target facial expression.

[0126] The expression transfer apparatus provided by the present disclosure is described below. The expression transfer apparatus described below and the expression transfer method described above can be referenced to each other.

[0127] FIG5 is a schematic diagram of the structure of an expression transfer apparatus provided by one embodiment of the present disclosure. As shown in FIG5 , the expression transfer apparatus includes: an acquisition module 501 , a feature extraction module 502 , and a processing module 503 .

[0128] An acquisition module 501 is configured to acquire a first facial image containing a target facial expression. A feature extraction module 502 is configured to input the first facial image into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model, which is used to represent the target facial expression. The expression feature extraction model is obtained by training a first network model based on multiple sample triplets and the annotation information corresponding to each sample triplet. Each sample triplet includes three first sample images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet. A processing module 503 is configured to input the expression feature vector into an expression transfer model to obtain a second facial image of a target virtual character containing the target facial expression, which is output by the expression transfer model. The device obtains an arbitrary first facial image containing the target facial expression. Subsequently, the first facial image is input into the expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model, which is used to represent the target facial expression. The first facial image used in this application can be recorded with an ordinary shooting device. There is no need to use specific equipment such as a specific helmet to capture the expression of a real person, and it is not subject to any scene restrictions. This greatly improves the diversity and concurrency of facial expression image acquisition and avoids a lot of manpower and time costs. The first facial image is input into the expression feature extraction model to obtain a target expression feature vector for characterizing the target facial expression in the first facial image. Among them, the expression feature extraction model is obtained by training the first network model based on multiple sample triplets and the annotation information corresponding to each sample triplet. Each sample triplet includes three facial images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet. The expression feature extraction model in this application can extract any fine-grained, continuous expression features. Moreover, since the annotation information corresponding to the sample triples only contains expression information and does not contain any irrelevant information (such as identity information, background, light, etc.), the expression feature vector extracted by the expression feature extraction model is only related to the expression, thereby achieving the decoupling of the expression feature vector from irrelevant information such as identity information, which can accurately perceive the fine-grained expressions between different identities and realize the expression transfer from any object to any virtual character. Finally, the target expression feature vector is input into the expression transfer model to obtain the second facial image of the target virtual character with the target facial expression output by the expression transfer model. This application can greatly improve the effect of expression transfer.

[0129] Optionally, the device also includes a driving module, which is specifically used to: input the second facial image into the expression parameter determination model to obtain the target expression driving parameters output by the expression parameter determination model, and the expression parameter determination model is used to generate expression driving parameters that make the face of the three-dimensional model present the target facial expression; input the target expression driving parameters into the three-dimensional model corresponding to the target virtual character, so that the face of the three-dimensional model presents the target facial expression. Through the above module, the three-dimensional model renderer can use the generated target expression driving parameters to reconstruct the facial expression of the three-dimensional character, thereby completing the migration of facial expressions. This method does not require matching data between the real face and the virtual character, nor does it require the collection or annotation of feature point positions on the face, which greatly reduces the time cost and process complexity of three-dimensional expression migration.

[0130] Optionally, the device also includes a first training module, which is specifically used to: construct an image generation model and an image discrimination model; for each second facial image in a plurality of second sample images, execute the first step to train the image generation model and the image discrimination model until the first convergence condition is met, and determine the image generation model after training as an expression transfer model, and the second sample image is a facial image containing a facial expression; the first step includes: inputting the second sample image into the expression feature extraction model to obtain a first expression feature vector output by the expression feature extraction model; inputting the first expression feature vector into the image generation model to obtain a third facial image of the target virtual character with a facial expression output by the image generation model; inputting the third facial image and a preset rendered facial image as two input images into the image discrimination model for discrimination, and obtaining a discrimination result output by the image discrimination model; wherein the preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional face model, and the discrimination result is used to indicate the probability value of each input image belonging to the rendered image and the generated image respectively; based on the discrimination result and the first loss function, calculating the first loss function value and adjusting the parameters of the image generation model and the image discrimination model according to the first loss function value. The expression transfer effect achieved by the above module is good, which can save a lot of manual repair processes for relevant personnel, save a lot of labor costs and significantly improve the efficiency of expression animation production. Furthermore, the picture quality of the generated image is close enough to that of the rendered image. In this way, the expression transfer method provided by this application can be used to create facial expressions for virtual characters in projects such as animation and game development. This greatly improves the efficiency of creating facial expressions for virtual characters and extremely reduces the threshold for creating virtual character facial expressions.

[0131] Optionally, the device also includes a second training module, which is specifically used to: obtain a second network model and a differentiable rendering network model; for each virtual character facial image in a plurality of virtual character facial images, repeat the second step to train the second network model until the second convergence condition is met, and determine the second network model after training as the expression parameter determination model; the second step includes: inputting the virtual character facial image into the second network model to obtain the first expression driving parameter output by the second network model; inputting the first expression driving parameter into the differentiable rendering network model to obtain the predicted facial image output by the differentiable rendering network model; calculating the second loss function value based on the virtual character facial image, the predicted facial image and the second loss function, and adjusting the parameters of the second network model based on the second loss function value.

[0132] Optionally, the second loss function includes a parameter constraint sub-function and an image pixel constraint sub-function; wherein, the parameter constraint sub-function is used to constrain the first expression driving parameter and the second expression driving parameter to be consistent, and the second expression driving parameter is the expression driving parameter of the three-dimensional model corresponding to the virtual character in the virtual character facial image; the image pixel constraint sub-function is used to constrain the pixels of the virtual character facial image and the predicted facial image to be consistent.

[0133] Optionally, the device also includes a third training module, which is specifically used to: obtain multiple sample triplets and annotation information corresponding to each sample triplet, the three first sample images included in the sample triplet are an anchor sample image, a positive sample image, and a negative sample image, the anchor sample image and the positive sample image are two facial images with the same or similar facial expressions, and the anchor sample image and the negative sample image are two facial images with different or dissimilar facial expressions; the anchor sample image, positive sample image, negative sample image and their respective annotation information in the sample triplet are input into the first network model for feature extraction to obtain third expression feature vectors corresponding to the anchor sample image, positive sample image and negative sample image; based on the third loss function, the third loss function value is calculated according to the third expression feature vectors corresponding to the anchor sample image, positive sample image and negative sample image; the parameters of the first network model are adjusted based on the third loss function value to obtain an expression feature extraction model; the parameter adjustment is used to shorten the distance between the anchor sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor sample image and the negative sample image in the expression feature space.

[0134] Through the modules of the above-mentioned embodiments, a large amount of sample data (i.e., sample triples and corresponding annotation information) is provided in a human-computer collaborative manner to continuously support the iteration and update of the expression feature extraction model and the expression migration model. A large number of sample triples can be manually annotated to obtain the annotation information of the sample triples. In a crowdsourcing scenario (i.e., a large annotation task is divided into multiple small sub-annotation tasks, and the sub-tasks are assigned to different user subjects to complete the annotation tasks), considering that the reliability of the annotation information of a single user is low, multiple users are introduced to participate in the annotation. Relying on interval estimation and truth inference algorithms, the number of annotations is automatically controlled to obtain more reliable annotation information recognized by the public, so as to ensure the accuracy and reliability of the annotation information corresponding to the sample triples.

[0135] Optionally, the device further includes a fourth training module, which is specifically used to: label the sample triples based on a preset labeling model to obtain first labeling information corresponding to the sample triples output by the preset labeling model; obtain target labeling information corresponding to the sample triples, and when the first labeling information and the target labeling information are inconsistent, train the preset labeling model, repeat the above steps until the second convergence condition is met, and determine the trained preset labeling model as the target labeling model; label multiple sample triples according to the target labeling model to obtain labeling information corresponding to each sample triple. By introducing the preset labeling model, the labeling cost is reduced and the labeling efficiency is improved.

[0136] Optionally, the first sample image includes a real person's facial image and a virtual character's facial image. Through the module in this embodiment, the facial images used can be facial expression images taken with any ordinary camera, eliminating the need for specific equipment for facial expression motion capture, such as wearable devices, and thus no longer subject to the limitations of specific equipment and recording scenarios. As a result, the present application greatly improves the flexibility and diversity of expression migration and avoids the incurring large equipment costs.

[0137] The expression migration device provided in this embodiment can be used to implement the technical solution of the above-mentioned expression migration method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0138] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. As shown in FIG6 , the electronic device 600 of this embodiment includes: a processor 601 and a memory 602; wherein the memory 602 is used to store computer-executable instructions; the processor 601 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the expression migration method in the above embodiment. For details, please refer to the relevant description of the above method embodiment. For example:

[0139] A first facial image containing a target facial expression is obtained; the first facial image is input into an expression feature extraction model to obtain a target expression feature vector, which is output by the expression feature extraction model and is used to represent the target facial expression. The expression feature extraction model is trained by a first network model based on multiple sample triplets and corresponding annotation information for each sample triplet, where each sample triplet includes three first sample images, and the annotation information is used to describe the facial expression of the three first sample images in the corresponding sample triplet. The target expression feature vector is input into an expression transfer model to obtain a second facial image of a target virtual character, which is output by the expression transfer model and has the target facial expression. By obtaining any first facial image containing the target facial expression, the first facial image is then input into the expression feature extraction model to obtain a target expression feature vector, which is output by the expression feature extraction model and is used to represent the target facial expression. The first facial image used in this application can be recorded with a common camera, eliminating the need for specific equipment, such as a helmet, to capture expressions on real people and is not subject to any scene restrictions. This greatly increases the diversity and concurrency of facial expression image acquisition, avoiding significant labor and time costs. The first facial image is input into an expression feature extraction model to obtain a target expression feature vector for representing the target facial expression in the first facial image. The expression feature extraction model is obtained by training a first network model based on multiple sample triplets and the annotation information corresponding to each sample triplet. Each sample triplet includes three facial images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet. The expression feature extraction model in this application can extract any fine-grained, continuous expression features. Moreover, since the annotation information corresponding to the sample triplet only contains expression information and does not contain any irrelevant information (such as identity information, background, light, etc.), the expression feature vector extracted by the expression feature extraction model is only related to the expression, thereby achieving decoupling of the expression feature vector from irrelevant information such as identity information, allowing accurate perception of fine-grained expressions between different identities, and enabling expression transfer from any object to any virtual character. Finally, the target expression feature vector is input into the expression transfer model to obtain a second facial image of the target virtual character with the target facial expression as output by the expression transfer model. This application can greatly improve the effect of expression transfer.

[0140] Optionally, the method further includes: inputting the second facial image into an expression parameter determination model to obtain target expression driving parameters output by the expression parameter determination model, wherein the expression parameter determination model is used to generate expression driving parameters that cause the face of the three-dimensional model to exhibit the target facial expression; and inputting the target expression driving parameters into the three-dimensional model corresponding to the target virtual character, so that the face of the three-dimensional model exhibits the target facial expression. In this embodiment, the three-dimensional model renderer can use the generated target expression driving parameters to reconstruct the facial expression of the three-dimensional character, thereby completing the migration of facial expressions. This method does not require matching data between the real face and the virtual character, nor does it require the collection or annotation of feature point positions on the face, greatly reducing the time cost and process complexity of three-dimensional expression migration.

[0141] Optionally, the method also includes: constructing an image generation model and an image discrimination model; for each second sample image in a plurality of second sample images, executing the first step to train the image generation model and the image discrimination model until the first convergence condition is met, and determining the image generation model after training as an expression transfer model, and the second sample image is a facial image containing a facial expression; the first step includes: inputting the second sample image into the expression feature extraction model to obtain a first expression feature vector output by the expression feature extraction model; inputting the first expression feature vector into the image generation model to obtain a third facial image of the target virtual character with a facial expression output by the image generation model; inputting the third facial image and a preset rendered facial image as two input images into the image discrimination model for discrimination, and obtaining a discrimination result output by the image discrimination model; wherein the preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional face model, and the discrimination result is used to indicate the probability value of each input image belonging to the rendered image and the generated image respectively; based on the discrimination result and the first loss function, calculating the first loss function value and adjusting the parameters of the image generation model and the image discrimination model according to the first loss function value. The above-described embodiment demonstrates a good expression transfer effect, eliminating the need for a large number of manual repair processes by relevant personnel, saving significant labor costs and significantly improving the efficiency of expression animation production. Furthermore, the image quality of the generated image is sufficiently close to that of a rendered image. Thus, the expression transfer method provided by this application can be used to create facial expressions for virtual characters in projects such as animation and game development. This greatly improves the efficiency of creating virtual character facial expressions and significantly lowers the threshold for creating virtual character facial expressions.

[0142] Optionally, before inputting the second facial image into the expression parameter determination model to obtain the target expression driving parameters output by the expression parameter determination model, the method also includes: obtaining a second network model and a differentiable rendering network model; for each virtual character facial image in a plurality of virtual character facial images, repeating the second step to train the second network model until the second convergence condition is met, and determining the trained second network model as the expression parameter determination model; the above-mentioned second step includes: inputting the virtual character facial image into the second network model to obtain the first expression driving parameters output by the second network model; inputting the first expression driving parameters into the differentiable rendering network model to obtain the predicted facial image output by the differentiable rendering network model; calculating the second loss function value based on the virtual character facial image, the predicted facial image and the second loss function, and adjusting the parameters of the second network model based on the second loss function value.

[0143] Optionally, the second loss function includes a parameter constraint sub-function and an image pixel constraint sub-function; wherein, the parameter constraint sub-function is used to constrain the first expression driving parameter and the second expression driving parameter to be consistent, and the second expression driving parameter is the expression driving parameter of the three-dimensional model corresponding to the virtual character in the virtual character facial image; the image pixel constraint sub-function is used to constrain the pixels of the virtual character facial image and the predicted facial image to be consistent.

[0144] Optionally, the method also includes: obtaining multiple sample triplets and annotation information corresponding to each sample triplet, the three first sample images included in the sample triplet are an anchor sample image, a positive sample image, and a negative sample image, the anchor sample image and the positive sample image are two facial images with the same or similar facial expressions, and the anchor sample image and the negative sample image are two facial images with different or dissimilar facial expressions; inputting the anchor sample image, positive sample image, negative sample image and their respective annotation information in the sample triplet into the first network model for feature extraction to obtain a third expression feature vector corresponding to the anchor sample image, the positive sample image, and the negative sample image; based on the third loss function, calculating the third loss function value according to the third expression feature vector corresponding to the anchor sample image, the positive sample image, and the negative sample image; adjusting the parameters of the first network model based on the third loss function value to obtain an expression feature extraction model; the parameter adjustment is used to shorten the distance between the anchor sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor sample image and the negative sample image in the expression feature space.

[0145] Through the above embodiments, a large amount of sample data (i.e., sample triples and corresponding annotation information) is provided in a human-computer collaborative manner to continuously support the iteration and update of the expression feature extraction model and the expression migration model. A large number of sample triples can be manually annotated to obtain the annotation information of the sample triples. In a crowdsourcing scenario (i.e., a large annotation task is divided into multiple small sub-annotation tasks, and the sub-tasks are assigned to different user subjects to complete the annotation tasks), considering that the reliability of the annotation information of a single user is low, multiple users are introduced to participate in the annotation. Relying on interval estimation and truth inference algorithms, the number of annotations is automatically controlled to obtain more reliable annotation information recognized by the public, which can ensure the accuracy and reliability of the annotation information corresponding to the sample triples.

[0146] Optionally, before inputting the anchor sample image, positive sample image, negative sample image, and their respective annotation information in the sample triplet into the first network model for feature extraction and obtaining the third expression feature vector corresponding to each of the anchor sample image, positive sample image, and negative sample image, the method further includes: annotating the sample triplet based on a preset annotation model to obtain first annotation information corresponding to the sample triplet output by the preset annotation model; obtaining target annotation information corresponding to the sample triplet, and when the first annotation information and the target annotation information are inconsistent, training the preset annotation model, repeating the above steps until the second convergence condition is met, and determining the trained preset annotation model as the target annotation model; annotating multiple sample triplets according to the target annotation model to obtain annotation information corresponding to each sample triplet. By introducing a preset annotation model, annotation costs are reduced and annotation efficiency is improved.

[0147] Optionally, the first sample image includes a real person's facial image and a virtual object's facial image. The facial images used in this embodiment can be facial expression images captured using any common camera, eliminating the need for specific equipment for facial expression motion capture, such as wearable devices. Furthermore, the application is no longer restricted by specific equipment and recording scenarios. This significantly improves the flexibility and diversity of expression migration and avoids significant equipment costs.

[0148] Optionally, the memory 602 may be independent or integrated with the processor 601 .

[0149] When the memory 602 is independently provided, the electronic device further includes a bus 603 for connecting the memory 602 and the processor 601 .

[0150] One embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the technical solution corresponding to the expression migration method in any of the above embodiments is implemented by the electronic device.

[0151] One embodiment of the present disclosure further provides a computer program product, comprising: a computer program stored in a readable storage medium; at least one processor of an electronic device can read the computer program from the readable storage medium; at least one processor executes the computer program so that the electronic device executes the technical solution corresponding to the expression transfer method in any of the above embodiments. For example:

[0152] A first facial image containing a target facial expression is obtained; the first facial image is input into an expression feature extraction model to obtain a target expression feature vector, which is output by the expression feature extraction model and is used to represent the target facial expression. The expression feature extraction model is trained by a first network model based on multiple sample triplets and corresponding annotation information for each sample triplet, where each sample triplet includes three first sample images, and the annotation information is used to describe the facial expression of the three first sample images in the corresponding sample triplet. The target expression feature vector is input into an expression transfer model to obtain a second facial image of a target virtual character, which is output by the expression transfer model and has the target facial expression. By obtaining any first facial image containing the target facial expression, the first facial image is then input into the expression feature extraction model to obtain a target expression feature vector, which is output by the expression feature extraction model and is used to represent the target facial expression. The first facial image used in this application can be recorded with a common camera, eliminating the need for specific equipment, such as a helmet, to capture expressions on real people and is not subject to any scene restrictions. This greatly increases the diversity and concurrency of facial expression image acquisition, avoiding significant labor and time costs. The first facial image is input into an expression feature extraction model to obtain a target expression feature vector for representing the target facial expression in the first facial image. The expression feature extraction model is obtained by training a first network model based on multiple sample triplets and the annotation information corresponding to each sample triplet. Each sample triplet includes three facial images, and the annotation information is used to describe the facial expressions of the three first sample images in the corresponding sample triplet. The expression feature extraction model in this application can extract any fine-grained, continuous expression features. Moreover, since the annotation information corresponding to the sample triplet only contains expression information and does not contain any irrelevant information (such as identity information, background, light, etc.), the expression feature vector extracted by the expression feature extraction model is only related to the expression, thereby achieving decoupling of the expression feature vector from irrelevant information such as identity information, allowing accurate perception of fine-grained expressions between different identities, and enabling expression transfer from any object to any virtual character. Finally, the target expression feature vector is input into the expression transfer model to obtain a second facial image of the target virtual character with the target facial expression as output by the expression transfer model. This application can greatly improve the effect of expression transfer.

[0153] Optionally, the method further includes: inputting the second facial image into an expression parameter determination model to obtain target expression driving parameters output by the expression parameter determination model, wherein the expression parameter determination model is used to generate expression driving parameters that cause the face of the three-dimensional model to exhibit the target facial expression; and inputting the target expression driving parameters into the three-dimensional model corresponding to the target virtual character, so that the face of the three-dimensional model exhibits the target facial expression. In this embodiment, the three-dimensional model renderer can use the generated target expression driving parameters to reconstruct the facial expression of the three-dimensional character, thereby completing the migration of facial expressions. This method does not require matching data between the real face and the virtual character, nor does it require the collection or annotation of feature point positions on the face, greatly reducing the time cost and process complexity of three-dimensional expression migration.

[0154] Optionally, the method also includes: constructing an image generation model and an image discrimination model; for each second sample image in a plurality of second sample images, executing the first step to train the image generation model and the image discrimination model until the first convergence condition is met, and determining the image generation model after training as an expression transfer model, and the second sample image is a facial image containing a facial expression; the first step includes: inputting the second sample image into the expression feature extraction model to obtain a first expression feature vector output by the expression feature extraction model; inputting the first expression feature vector into the image generation model to obtain a third facial image of the target virtual character with a facial expression output by the image generation model; inputting the third facial image and a preset rendered facial image as two input images into the image discrimination model for discrimination, and obtaining a discrimination result output by the image discrimination model; wherein the preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional face model, and the discrimination result is used to indicate the probability value of each input image belonging to the rendered image and the generated image respectively; based on the discrimination result and the first loss function, calculating the first loss function value and adjusting the parameters of the image generation model and the image discrimination model according to the first loss function value. The above-described embodiment demonstrates a good expression transfer effect, eliminating the need for a large number of manual repair processes by relevant personnel, saving significant labor costs and significantly improving the efficiency of expression animation production. Furthermore, the image quality of the generated image is sufficiently close to that of a rendered image. Thus, the expression transfer method provided by this application can be used to create facial expressions for virtual characters in projects such as animation and game development. This greatly improves the efficiency of creating virtual character facial expressions and significantly lowers the threshold for creating virtual character facial expressions.

[0155] Optionally, before inputting the second facial image into the expression parameter determination model to obtain the target expression driving parameters output by the expression parameter determination model, the method also includes: obtaining a second network model and a differentiable rendering network model; for each virtual character facial image in a plurality of virtual character facial images, repeating the second step to train the second network model until the second convergence condition is met, and determining the trained second network model as the expression parameter determination model; the above-mentioned second step includes: inputting the virtual character facial image into the second network model to obtain the first expression driving parameters output by the second network model; inputting the first expression driving parameters into the differentiable rendering network model to obtain the predicted facial image output by the differentiable rendering network model; calculating the second loss function value based on the virtual character facial image, the predicted facial image and the second loss function, and adjusting the parameters of the second network model based on the second loss function value.

[0156] Optionally, the second loss function includes a parameter constraint sub-function and an image pixel constraint sub-function; wherein, the parameter constraint sub-function is used to constrain the first expression driving parameter and the second expression driving parameter to be consistent, and the second expression driving parameter is the expression driving parameter of the three-dimensional model corresponding to the virtual character in the virtual character facial image; the image pixel constraint sub-function is used to constrain the pixels of the virtual character facial image and the predicted facial image to be consistent.

[0157] Optionally, the method also includes: obtaining multiple sample triplets and annotation information corresponding to each sample triplet, the three first sample images included in the sample triplet are an anchor sample image, a positive sample image, and a negative sample image, the anchor sample image and the positive sample image are two facial images with the same or similar facial expressions, and the anchor sample image and the negative sample image are two facial images with different or dissimilar facial expressions; inputting the anchor sample image, positive sample image, negative sample image and their respective annotation information in the sample triplet into the first network model for feature extraction to obtain a third expression feature vector corresponding to the anchor sample image, the positive sample image, and the negative sample image; based on the third loss function, calculating the third loss function value according to the third expression feature vector corresponding to the anchor sample image, the positive sample image, and the negative sample image; adjusting the parameters of the first network model based on the third loss function value to obtain an expression feature extraction model; the parameter adjustment is used to shorten the distance between the anchor sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor sample image and the negative sample image in the expression feature space.

[0158] Through the above embodiments, a large amount of sample data (i.e., sample triples and corresponding annotation information) is provided in a human-computer collaborative manner to continuously support the iteration and update of the expression feature extraction model and the expression migration model. A large number of sample triples can be manually annotated to obtain the annotation information of the sample triples. In a crowdsourcing scenario (i.e., a large annotation task is divided into multiple small sub-annotation tasks, and the sub-tasks are assigned to different user subjects to complete the annotation tasks), considering that the reliability of the annotation information of a single user is low, multiple users are introduced to participate in the annotation. Relying on interval estimation and truth inference algorithms, the number of annotations is automatically controlled to obtain more reliable annotation information recognized by the public, which can ensure the accuracy and reliability of the annotation information corresponding to the sample triples.

[0159] Optionally, before inputting the anchor sample image, positive sample image, negative sample image, and their respective annotation information in the sample triplet into the first network model for feature extraction and obtaining the third expression feature vector corresponding to each of the anchor sample image, positive sample image, and negative sample image, the method further includes: annotating the sample triplet based on a preset annotation model to obtain first annotation information corresponding to the sample triplet output by the preset annotation model; obtaining target annotation information corresponding to the sample triplet, and when the first annotation information and the target annotation information are inconsistent, training the preset annotation model, repeating the above steps until the second convergence condition is met, and determining the trained preset annotation model as the target annotation model; annotating multiple sample triplets according to the target annotation model to obtain annotation information corresponding to each sample triplet. By introducing a preset annotation model, annotation costs are reduced and annotation efficiency is improved.

[0160] Optionally, the first sample image includes a real person's facial image and a virtual object's facial image. The facial images used in this embodiment can be facial expression images captured using any common camera, eliminating the need for specific equipment for facial expression motion capture, such as wearable devices. Furthermore, the application is no longer restricted by specific equipment and recording scenarios. This significantly improves the flexibility and diversity of expression migration and avoids significant equipment costs.

[0161] Although the present disclosure is disclosed as above in terms of preferred embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

[0162] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0163] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing an electronic device (which can be a personal computer, server, or network device, etc.) or a processor to perform some of the steps of the methods of various embodiments of the present disclosure.

[0164] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0165] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0166] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the figures of this disclosure are not limited to just one bus or just one type of bus.

[0167] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0168] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for transferring facial expressions, comprising: Acquire a first facial image containing a target facial expression; Inputting the first facial image into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression; wherein the expression feature extraction model is obtained by training a first network model based on a plurality of sample triplets and annotation information corresponding to each of the sample triplets, each sample triplet including three first sample images, and the annotation information being used to describe the facial expressions corresponding to the three first sample images in the sample triplet; The target expression feature vector is input into an expression transfer model to obtain a second facial image of the target virtual character having the target facial expression output by the expression transfer model.

2. The method according to claim 1, wherein The method further comprises: Inputting the second facial image into an expression parameter determination model to obtain target expression driving parameters output by the expression parameter determination model, wherein the expression parameter determination model is used to generate expression driving parameters that cause the face of the three-dimensional model to present the target facial expression; The target expression driving parameters are input into the three-dimensional model corresponding to the target virtual character, so that the face of the three-dimensional model presents the target facial expression.

3. The method according to claim 1, wherein The method further comprises: Build image generation model and image discrimination model; For each of the plurality of second sample images, executing the first step to train the image generation model and the image discrimination model until a first convergence condition is satisfied, and determining the trained image generation model as the expression transfer model, wherein the second sample image is a facial image containing a facial expression; the first step comprises: Inputting the second sample image into the expression feature extraction model to obtain a first expression feature vector output by the expression feature extraction model; Inputting the first expression feature vector into the image generation model to obtain a third facial image of the target virtual character having a facial expression output by the image generation model; The third facial image and the preset rendered facial image are input as two input images into the image discrimination model for discrimination, thereby obtaining a discrimination result output by the image discrimination model; wherein the preset rendered facial image is a two-dimensional rendered image obtained by rendering a three-dimensional face model, and the discrimination result is used to indicate the probability value of each input image belonging to the rendered image and the generated image respectively; Based on the discrimination result and the first loss function, a first loss function value is calculated and parameters of the image generation model and the image discrimination model are adjusted according to the first loss function value.

4. The method according to claim 2, wherein: Before inputting the second facial image into the expression parameter determination model to obtain the target expression driving parameters output by the expression parameter determination model, the method further includes: Obtain a second network model and a differentiable rendering network model; For each of the plurality of virtual character facial images, repeatedly performing the second step to train the second network model until a second convergence condition is met, and determining the trained second network model as the expression parameter determination model; The second step includes: Inputting the virtual character's facial image into the second network model to obtain a first expression driving parameter output by the second network model; Inputting the first expression driving parameter into the differentiable rendering network model to obtain a predicted facial image output by the differentiable rendering network model; According to the virtual character facial image, the predicted facial image and the second loss function, a second loss function value is calculated and parameters of the second network model are adjusted according to the second loss function value.

5. The method according to claim 4, wherein The second loss function includes a parameter constraint sub-function and an image pixel constraint sub-function; Among them, the parameter constraint sub-function is used to constrain the first expression driving parameter and the second expression driving parameter to be consistent, and the second expression driving parameter is the expression driving parameter of the three-dimensional model corresponding to the virtual character in the virtual character facial image; the image pixel constraint sub-function is used to constrain the pixels of the virtual character facial image and the predicted facial image to be consistent.

6. The method according to claim 1, wherein The method further comprises: Obtaining the multiple sample triplets and annotation information corresponding to each of the sample triplets, wherein the three first sample images included in the sample triplets are respectively an anchor sample image, a positive sample image, and a negative sample image, the anchor sample image and the positive sample image are two facial images with the same or similar facial expressions, and the anchor sample image and the negative sample image are two facial images with different or dissimilar facial expressions; Inputting the anchor sample image, the positive sample image, the negative sample image and their respective annotation information in the sample triplet into the first network model for feature extraction, and obtaining third expression feature vectors corresponding to the anchor sample image, the positive sample image and the negative sample image respectively; Calculating a third loss function value based on a third loss function according to the third expression feature vectors corresponding to the anchor sample image, the positive sample image, and the negative sample image; The parameters of the first network model are adjusted based on the third loss function value to obtain the expression feature extraction model; the parameter adjustment is used to shorten the distance between the anchor point sample image and the positive sample image in the expression feature space, and to increase the distance between the anchor point sample image and the negative sample image in the expression feature space.

7. The method according to claim 6, wherein: Before inputting the anchor sample image, the positive sample image, the negative sample image, and their respective annotation information in the sample triplet into the first network model for feature extraction to obtain third expression feature vectors corresponding to the anchor sample image, the positive sample image, and the negative sample image, the method further includes: Performing labeling processing on the sample triples based on a preset labeling model to obtain first labeling information corresponding to the sample triples output by the preset labeling model; Obtain target labeling information corresponding to the sample triplet; if the first labeling information and the target labeling information are inconsistent, train the preset labeling model, repeat the above steps until a second convergence condition is met, and determine the trained preset labeling model as the target labeling model; A labeling process is performed on multiple sample triples according to the target labeling model to obtain labeling information corresponding to each of the sample triples.

8. The method according to claim 1, wherein The first sample images include a facial image of a real person and a facial image of a virtual object.

9. An expression transfer device, comprising: An acquisition module is configured to acquire a first facial image containing a target facial expression; a feature extraction module configured to input the first facial image into an expression feature extraction model to obtain a target expression feature vector output by the expression feature extraction model for characterizing the target facial expression; wherein the expression feature extraction model is obtained by training a first network model based on a plurality of sample triplets and annotation information corresponding to each of the sample triplets, each sample triplet including three first sample images, and the annotation information being used to describe the facial expressions corresponding to the three first sample images in the sample triplet; The processing module is configured to input the expression feature vector into an expression transfer model to obtain a second facial image of the target virtual character having the target facial expression output by the expression transfer model.

10. An electronic device, comprising: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the expression migration method according to any one of claims 1 to 8 is executed.

11. A computer-readable storage medium storing a data processing program, wherein the program is executed by a processor to perform the expression transfer method according to any one of claims 1 to 8.

Images