Three-dimensional facial expression migration method and device

Abstract

The invention relates to a three-dimensional face expression migration method and device. The method comprises the following steps: acquiring a target face image; performing three-dimensional reconstruction on the target face image based on the standard three-dimensional portrait model to obtain a target three-dimensional portrait model; and migrating expressions on a standard three-dimensional portrait model (called as an expression three-dimensional portrait model) with expressions to a target three-dimensional portrait model to obtain a three-dimensional portrait model after expression migration. According to the three-dimensional face expression migration method and device, fine expression migration can be achieved with low cost, and face structure distortion can be prevented.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly to a three-dimensional facial expression transfer method and device. BACKGROUND

[0002] With the development of virtual reality and metaverse technologies, three-dimensional facial expression transfer technology, as a core technology for virtual character interaction and experience, has received extensive attention in recent years. Prior to this, two-dimensional facial image expression transfer technology has made significant progress, but it has limitations in expression naturalness, view robustness, and three-dimensional space editability. In contrast, three-dimensional facial expression transfer technology based on three-dimensional facial models can more realistically restore facial dynamic features through geometric deformation and texture mapping, supporting natural expression expression from any viewing angle. However, existing three-dimensional methods still face technical bottlenecks: first, when transferring expressions to three-dimensional facial models, it is difficult to achieve vertex or control point level geometric deformation without professional equipment and a large amount of manual time cost; second, the input three-dimensional facial model is sensitive to specific expression poses, and some complex expressions are prone to facial structure distortion under the transfer effect of traditional three-dimensional transfer algorithms, resulting in expression transfer results deviating from real facial features. SUMMARY

[0003] The present application aims to provide a three-dimensional facial expression transfer method and device to solve the technical problems existing in the prior art.

[0004] To achieve the above purpose, the present application provides a three-dimensional facial expression transfer method, comprising:

[0005] obtaining a target facial image;

[0006] performing three-dimensional reconstruction on the target facial image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model;

[0007] transferring an expression on an expression three-dimensional portrait model to the target three-dimensional portrait model to obtain a three-dimensional portrait model after expression transfer.

[0008] Optionally, performing three-dimensional reconstruction on the target facial image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model, specifically comprising:

[0009] deforming a standard mesh model based on the target facial image to obtain a deformed mesh model;

[0010] converting the deformed mesh model to the coordinate system of the standard mesh model to obtain a target mesh model;

[0011] replacing a standard portrait skin based on the target facial image to obtain a target portrait skin;

[0012] The target portrait skin is pasted onto the target mesh model to obtain a target three-dimensional portrait model.

[0013] Optionally, the standard portrait skin is replaced based on the target face image to obtain the target portrait skin, and the method specifically comprises:

[0014] The standard portrait skin is divided into a standard face region and a standard non-face region, the face region is a main expression region including eyes, a nose, a forehead, a mouth, etc., and the non-face region is other regions except the face region.

[0015] The target face region corresponding to the standard face region is detected on the target face image, the skin of the target face region is replaced with the skin of the standard face region to obtain the skin of the replaced standard face region;

[0016] The target non-face region corresponding to the standard non-face region is detected on the target face image, and the skin of the replaced standard non-face region is obtained based on the skin of the standard non-face region and the skin of the target non-face region.

[0017] The skin of the replaced standard face region and the skin of the replaced standard non-face region are fused to obtain the target portrait skin.

[0018] Optionally, the expression of the expression three-dimensional portrait model is migrated to the target three-dimensional portrait model to obtain an expression-migrated three-dimensional portrait model, and the method specifically comprises:

[0019] The first migration is performed on each vertex of a face vertex set of the target three-dimensional portrait model based on the coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait, by using a double-layer deformation method, to obtain a first-migrated three-dimensional portrait model.

[0020] The second migration is performed on each vertex of an upper mouth vertex set of the first-migrated three-dimensional portrait model based on the coordinates of each vertex of the upper mouth vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait, and the second migration is performed on each vertex of a lower mouth vertex set of the first-migrated three-dimensional portrait model based on the coordinates of each vertex of the lower mouth vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait, to obtain a second-migrated three-dimensional portrait model.

[0021] Obtain a standard 3D human portrait model with the left eye completely closed, as the left-eye fully closed human portrait model; obtain a standard 3D human portrait model with the right eye completely closed, as the right-eye fully closed human portrait model; using a two-layer deformation method, based on the coordinates of each vertex of the face vertex set of the expression 3D human portrait model and the left-eye fully closed human portrait model, perform a third migration on each vertex of the face vertex set of the target 3D human portrait model, to obtain the 3D human portrait model after the third migration; and then, based on the coordinates of each vertex of the face vertex set of the expression 3D human portrait model and the right-eye fully closed human portrait model, perform a fourth migration on each vertex of the face vertex set of the target 3D human portrait model, to obtain the 3D human portrait model after the fourth migration.

[0022] Calculate the new coordinates of each vertex of the left eyelid vertex set based on the coordinates of each vertex of the left eyelid vertex set in the three-dimensional portrait model with facial expression, the standard three-dimensional portrait model, and the three-dimensional portrait model after the third migration; calculate the new coordinates of each vertex of the right eyelid vertex set based on the coordinates of each vertex of the right eyelid vertex set in the three-dimensional portrait model with facial expression, the standard three-dimensional portrait model, and the three-dimensional portrait model after the fourth migration.

[0023] The vertices of the left and right eyelid vertex sets of the 3D human figure model after the second migration are moved to their respective new coordinate positions to obtain the 3D human figure model after the fifth migration.

[0024] Based on the coordinates of each vertex in the eyeball vertex set, oral cavity vertex set, and head vertex set of the standard 3D human portrait model, a sixth migration is performed on each vertex in the eyeball vertex set, oral cavity vertex set, and head vertex set of the 3D human portrait model after the fifth migration, so as to obtain the 3D human portrait model after the sixth migration, which serves as the 3D human portrait model after expression migration.

[0025] Optionally, using a two-layer deformation method, the vertices of the face vertex set of the target 3D portrait model are first migrated based on the coordinates of each vertex of the facial vertex set of the expression 3D portrait model and the standard 3D portrait, to obtain the 3D portrait model after the first migration, specifically including:

[0026] Select a subset of vertices from the set of face vertices as each first control point, and select a subset of each first control point as each first key point;

[0027] For each first key point, calculate the displacement of the first key point of the facial expression 3D portrait model relative to the first key point of the standard 3D portrait model, and use it as the displacement of the first key point;

[0028] Calculate the new coordinates of each first control point based on the displacement of each first key point;

[0029] For each first control point, a displacement of the first control point is calculated according to a new coordinate of the first control point and a coordinate of the first control point of the standard three-dimensional portrait model;

[0030] New coordinates of each vertex of the face vertex set are calculated according to the displacements of the first control points;

[0031] Each vertex of the face vertex set of the target three-dimensional portrait model is migrated to a respective new coordinate to obtain a three-dimensional portrait model after the first migration.

[0032] Optionally, new coordinates of each vertex of the left eyelid vertex set are calculated based on coordinates of each vertex of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the three-dimensional portrait model after the third migration; and new coordinates of each vertex of the right eyelid vertex set are calculated based on coordinates of each vertex of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the three-dimensional portrait model after the fourth migration, specifically including:

[0033] A left eye closure degree is determined based on coordinates of each vertex of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the left eye fully closed portrait model;

[0034] New coordinates of each vertex of the left eyelid vertex set are determined based on the left eye closure degree, coordinates of each vertex of the left eyelid vertex set of the three-dimensional portrait model after the second migration and the three-dimensional portrait model after the third migration;

[0035] A right eye closure degree is determined based on coordinates of each vertex of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the right eye fully closed portrait model;

[0036] New coordinates of each vertex of the right eyelid vertex set are determined based on the right eye closure degree, coordinates of each vertex of the right eyelid vertex set of the three-dimensional portrait model after the second migration and the three-dimensional portrait model after the fourth migration.

[0037] Optionally, a sixth migration is performed on each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration based on coordinates of each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model to obtain a three-dimensional portrait model after the sixth migration as the three-dimensional portrait model after the expression migration, specifically including:

[0038] Deviation angles of the eyeball vertex center, the inside mouth vertex center and the head vertex center in the expression three-dimensional portrait model and the standard three-dimensional portrait model on the x, y and z axes are calculated respectively to obtain an eyeball rotation angle, an inside mouth rotation angle and a head rotation angle;

[0039] determining coordinates of the eyeball rotation center, the inside-of-mouth rotation center and the head rotation center of the third migrated three-dimensional portrait model based on the coordinates of each vertex of the set of vertices of the eyeball, the set of vertices of the inside of the mouth and the set of vertices of the head of the third migrated three-dimensional portrait model;

[0040] determining new coordinates of each vertex in the set of vertices of the eyeball based on the eyeball rotation angle and the coordinates of the eyeball rotation center of the third migrated three-dimensional portrait model, determining new coordinates of each vertex in the set of vertices of the inside of the mouth based on the inside-of-mouth rotation angle and the coordinates of the inside-of-mouth rotation center of the third migrated three-dimensional portrait model, and determining new coordinates of each vertex in the set of vertices of the head based on the head rotation angle and the coordinates of the head rotation center of the third migrated three-dimensional portrait model;

[0041] migrating each vertex of the set of vertices of the eyeball, the set of vertices of the inside of the mouth and the set of vertices of the head of the third migrated three-dimensional portrait model to the respective new coordinates to obtain a fourth migrated three-dimensional portrait model.

[0042] In another aspect, the present application provides a three-dimensional facial expression migration device, which comprises:

[0043] an acquisition module configured to acquire a target facial image;

[0044] a three-dimensional reconstruction module configured to perform three-dimensional reconstruction on the target facial image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model;

[0045] an expression migration module configured to migrate an expression on an expression three-dimensional portrait model to the target three-dimensional portrait model to obtain a three-dimensional portrait model after expression migration.

[0046] Optionally, the three-dimensional reconstruction on the target facial image based on the standard three-dimensional portrait model to obtain the target three-dimensional portrait model specifically comprises:

[0047] deforming a standard mesh model based on the target facial image to obtain a deformed mesh model;

[0048] converting the deformed mesh model to a coordinate system of the standard mesh model to obtain a target mesh model;

[0049] replacing a standard portrait skin based on the target facial image to obtain a target portrait skin;

[0050] attaching the target portrait skin to the target mesh model to obtain the target three-dimensional portrait model;

[0051] Optionally, the replacing the standard portrait skin based on the target facial image to obtain the target portrait skin specifically comprises:

[0052] The standard human portrait skin is divided into a standard human face region and a standard non-human face region, the human face region is a main expression region including eyes, nose, forehead, mouth and the like, and the non-human face region is other regions except the human face region;

[0053] A target human face region corresponding to the standard human face region is detected on the target human face image, the skin of the target human face region is replaced with the skin of the standard human face region, and a replaced standard human face region skin is obtained;

[0054] A target non-human face region corresponding to the standard non-human face region is detected on the target human face image, and a replaced standard non-human face region skin is obtained based on the skin of the standard non-human face region and the skin of the target non-human face region;

[0055] The replaced standard human face region skin and the replaced standard non-human face region skin are fused to obtain a target human portrait skin.

[0056] Optionally, an expression on the expression three-dimensional human portrait model is migrated to the target three-dimensional human portrait model to obtain an expression-migrated three-dimensional human portrait model, and the method specifically comprises:

[0057] The coordinates of each vertex of the face vertex set of the expression three-dimensional human portrait model and the standard three-dimensional human portrait are used to perform first migration on each vertex of the face vertex set of the target three-dimensional human portrait model by using a double-layer deformation method, and a first-migrated three-dimensional human portrait model is obtained;

[0058] The coordinates of each vertex of the upper mouth vertex set of the expression three-dimensional human portrait model and the standard three-dimensional human portrait are used to perform second migration on each vertex of the upper mouth vertex set of the first-migrated three-dimensional human portrait model, and the coordinates of each vertex of the lower mouth vertex set of the expression three-dimensional human portrait model and the standard three-dimensional human portrait are used to perform second migration on each vertex of the lower mouth vertex set of the first-migrated three-dimensional human portrait model, and a second-migrated three-dimensional human portrait model is obtained;

[0059] A standard three-dimensional human portrait model with the left eye completely closed is obtained as a left-eye-completely-closed human portrait model, and a standard three-dimensional human portrait model with the right eye completely closed is obtained as a right-eye-completely-closed human portrait model; the coordinates of each vertex of the face vertex set of the expression three-dimensional human portrait model and the left-eye-completely-closed human portrait model are used to perform third migration on each vertex of the face vertex set of the target three-dimensional human portrait model by using a double-layer deformation method, and a third-migrated three-dimensional human portrait model is obtained; and the coordinates of each vertex of the face vertex set of the expression three-dimensional human portrait model and the right-eye-completely-closed human portrait model are used to perform fourth migration on each vertex of the face vertex set of the target three-dimensional human portrait model, and a fourth-migrated three-dimensional human portrait model is obtained;

[0060] The new coordinates of the vertices of the left eyelid vertex set are calculated based on the coordinates of the vertices of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third time migrated three-dimensional portrait model; the new coordinates of the vertices of the right eyelid vertex set are calculated based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the fourth time migrated three-dimensional portrait model;

[0061] The vertices of the left eyelid vertex set and the right eyelid vertex set of the second time migrated three-dimensional portrait model are moved to the respective new coordinate positions to obtain the fifth time migrated three-dimensional portrait model;

[0062] The sixth migration is performed on the vertices of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the fifth time migrated three-dimensional portrait model based on the coordinates of the vertices of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model to obtain the sixth time migrated three-dimensional portrait model as the expression migrated three-dimensional portrait model;

[0063] The first migration is performed on the vertices of the face vertex set of the target three-dimensional portrait model based on the coordinates of the vertices of the face vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait by using a double-layer deformation method to obtain the first time migrated three-dimensional portrait model, and the first migration specifically includes:

[0064] Some vertices are selected from the face vertex set as first control points, and some of the first control points are selected as first key points;

[0065] For each first key point, the displacement of the first key point of the expression three-dimensional portrait model relative to the first key point of the standard three-dimensional portrait model is calculated as the displacement of the first key point;

[0066] The new coordinates of the first control points are calculated according to the displacements of the first key points;

[0067] For each first control point, the displacement of the first control point is calculated according to the new coordinates of the first control point and the coordinates of the first control point of the standard three-dimensional portrait model;

[0068] The new coordinates of the vertices of the face vertex set are calculated according to the displacements of the first control points;

[0069] The vertices of the face vertex set of the target three-dimensional portrait model are migrated to the respective new coordinates to obtain the first time migrated three-dimensional portrait model;

[0070] The new coordinates of the vertices of the left eyelid vertex set are calculated based on the coordinates of the vertices of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model, and the third migrated three-dimensional portrait model; the new coordinates of the vertices of the right eyelid vertex set are calculated based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model, and the fourth migrated three-dimensional portrait model, specifically including:

[0071] The left eye closure degree is determined based on the vertex coordinates of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model, and the left eye fully closed portrait model;

[0072] The new coordinates of the vertices of the left eyelid vertex set are determined based on the coordinates of the vertices of the left eyelid vertex set of the second migrated three-dimensional portrait model and the third migrated three-dimensional portrait model according to the left eye closure degree;

[0073] The right eye closure degree is determined based on the vertex coordinates of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model, and the right eye fully closed portrait model;

[0074] The new coordinates of the vertices of the right eyelid vertex set are determined based on the coordinates of the vertices of the right eyelid vertex set of the second migrated three-dimensional portrait model and the fourth migrated three-dimensional portrait model according to the right eye closure degree;

[0075] The sixth migration is performed on the vertices of the eyeball vertex set, the inside mouth vertex set, and the head vertex set of the fifth migrated three-dimensional portrait model based on the coordinates of the vertices of the eyeball vertex set, the inside mouth vertex set, and the head vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model to obtain a sixth migrated three-dimensional portrait model as the expression migrated three-dimensional portrait model, specifically including:

[0076] The deviation angles of the eyeball vertex center, the inside mouth vertex center, and the head vertex center in the expression three-dimensional portrait model and the standard three-dimensional portrait model on the x, y, and z axes are calculated respectively to obtain the eyeball rotation angle, the inside mouth rotation angle, and the head rotation angle;

[0077] The coordinates of the eyeball rotation center, the inside mouth rotation center, and the head rotation center of the fifth migrated three-dimensional portrait model are determined based on the coordinates of the vertices of the eyeball vertex set, the inside mouth vertex set, and the head vertex set of the fifth migrated three-dimensional portrait model;

[0078] determining new coordinates of each vertex in the eye vertex set based on the eye rotation angle and the coordinates of the eye rotation center of the three-dimensional portrait model after the fifth migration, determining new coordinates of each vertex in the oral interior vertex set based on the oral rotation angle and the coordinates of the oral interior rotation center of the three-dimensional portrait model after the fifth migration, and determining new coordinates of each vertex in the head vertex set based on the head rotation angle and the coordinates of the head rotation center of the three-dimensional portrait model after the fifth migration;

[0079] migrating each vertex of the eye vertex set, the oral interior vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration to the respective new coordinates to obtain a three-dimensional portrait model after a sixth migration. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 a flow chart of a three-dimensional facial expression migration method according to an embodiment of the present application;

[0081] Figure 2 a structural schematic diagram of a three-dimensional facial expression migration device according to an embodiment of the present application. DETAILED DESCRIPTION

[0082] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] As shown in Figure 1 the present embodiment provides a three-dimensional facial expression migration method, which comprises the following steps S100-S300:

[0084] S100: obtaining a target facial image.

[0085] The target facial image is a receptor carrier of expression migration, and the expression migration is to migrate a desired expression to the target facial image. The target facial image is a two-dimensional digital image containing a single complete face, which can be obtained by any suitable way such as static image acquisition, dynamic video stream interception, database calling, etc.

[0086] S200: performing three-dimensional reconstruction on the target facial image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model.

[0087] In some embodiments, the standard three-dimensional portrait model includes a standard mesh model and a standard portrait skin, the standard mesh model being a geometric skeleton of the three-dimensional portrait, including a plurality of vertices and their connection relationship, for defining the shape, topology and deformable range of the face, the static contour of the face being described by the coordinates of the vertices, and the expression change being realized by the vertex displacement. The standard portrait skin is the appearance layer of the three-dimensional portrait, and the visual details of the face, including the skin color, wrinkles, pores, gloss, etc., can be defined by the texture mapping and material properties. The standard three-dimensional portrait includes the head, neck and part of the body below the shoulder. The standard three-dimensional portrait model can be obtained from the prior art.

[0088] Step S200 specifically includes the following steps S210-S240:

[0089] S210: morphing the standard mesh model based on the target face image to obtain a morphed mesh model.

[0090] In step S210, the key points / contours of the target face image, such as eyes, mouth, nose, etc., can be detected and aligned with the key points of the standard mesh model, and then the morphing parameters are estimated to morph each vertex of the standard mesh model, thereby obtaining the morphed mesh model. Step S210 can be implemented by using existing algorithms, such as the free-form deformation algorithm, etc.

[0091] S220: converting the morphed mesh model to the coordinate system of the standard mesh model to obtain a target mesh model.

[0092] Since the morphed mesh model and the standard mesh model are not in the same coordinate system, in step S220, the coordinates of each vertex of the morphed mesh model need to be converted to the coordinate system of the standard mesh model, and the coordinate conversion formula is as follows:

[0093] n

[0094] wherein, is the coordinate of vertex i of the morphed mesh model, is the coordinate of vertex i of the target mesh model, w and h are the width and height of the target face image pixels respectively, and the depth d of the target three-dimensional portrait model is obtained by proportional conversion in combination with w and h in the target face image, with reference to the width, height and depth of the face in the standard three-dimensional portrait model, is the scale size ratio of the coordinate system of the standard mesh model to the coordinate system of the target face image, wherein, is the distance between the centers of the left and right eyes in the standard mesh model, is the distance between the centers of the left and right eyes in the target face image. When the coordinates of all the vertices of the target mesh model are known, the target mesh model can be obtained.

[0095] S230: replacing the standard portrait skin based on the target face image to obtain a target portrait skin.

[0096] In some embodiments, step S230 specifically comprises steps S231-S234:

[0097] S231: dividing the standard portrait skin into a standard face region and a standard non-face region, the face region being a facial expression main performance region including eyes, nose, forehead, mouth, etc., and the non-face region being other regions except the face region;

[0098] S232: detecting a target face region corresponding to the standard face region on the target face image, replacing the skin of the target face region with the skin of the standard face region to obtain the skin of the replaced standard face region;

[0099] S233: detecting a target non-face region corresponding to the standard non-face region on the target face image, obtaining the skin of the replaced standard non-face region based on the skin of the standard non-face region and the skin of the target non-face region;

[0100] S234: fusing (e.g., Poisson fusion algorithm) the skin of the replaced standard face region and the skin of the replaced standard non-face region to obtain the target portrait skin.

[0101] In step S232, the vertices of the target face region can be detected first, and then the vertices are subjected to Delaunay triangulation to obtain a plurality of triangular regions, and then the skin (i.e., RGB pixel value) of each triangular region of the target face region is replaced with the skin of the corresponding triangular region of the standard face region to obtain the skin of the replaced standard face region.

[0102] In step S233, the skin of the standard non-face region and the target non-face region can be first converted from the RGB color space to the YUV color space, and then subjected to linear transformation for skin replacement. Specifically, the linear transformation formula is as follows:

[0103]

[0104] wherein, is the value (hereinafter referred to as YUV value) of the skin of the standard non-face region converted from the RGB color space to the YUV color space, and denote the mean and standard deviation of the YUV values in the standard non-face region, and denote the mean and standard deviation of the YUV values in the target non-face region, The YUV value of the replaced standard non-face region skin is converted back to the RGB color space to obtain the RGB value of the replaced standard non-face region skin, and thus the replaced standard non-face region skin is obtained.

[0105] S240: The target portrait skin is pasted onto the target mesh model to obtain a target three-dimensional portrait model.

[0106] The method of pasting the skin onto the mesh model is prior art, and its specific steps and principles will not be repeated here.

[0107] S300: The expression of the expression three-dimensional portrait model is migrated to the target three-dimensional portrait model to obtain an expression-migrated three-dimensional portrait model.

[0108] The expression three-dimensional portrait model (also referred to as a standard three-dimensional portrait model with expression) is obtained based on the standard three-dimensional portrait model, that is, the difference between the expression three-dimensional portrait model and the standard three-dimensional portrait model is only that the coordinates of at least some of the vertices of the expression three-dimensional portrait model are different from those of the standard three-dimensional portrait model. That is, by changing the coordinates of the vertices of the standard three-dimensional portrait model, the standard three-dimensional portrait model can have a specific expression. The standard three-dimensional portrait model can also be referred to as a three-dimensional portrait model with a neutral expression. The target three-dimensional portrait model is also obtained based on the standard three-dimensional portrait model, and the target three-dimensional portrait model, the expression three-dimensional portrait model, and the standard three-dimensional portrait model have the same number of vertices and the same connection relationship between the vertices, except that the positions of the vertices are different. Therefore, the vertices in the target three-dimensional portrait model, the expression three-dimensional portrait model, and the standard three-dimensional portrait model correspond to each other, that is, for each vertex in the standard three-dimensional portrait model, a vertex with the same semantic can be found in the target three-dimensional portrait model and the expression three-dimensional portrait model. For example, if there is a mouth left vertex in the standard three-dimensional portrait model, there is also a mouth left vertex in the target three-dimensional portrait model and the expression three-dimensional portrait model.

[0109] The vertices of the standard three-dimensional portrait model can be divided into a face vertex set, an eyeball vertex set, an internal oral cavity vertex set, and a head vertex set, and the face vertex set includes a set of vertices that constitute a face region (excluding the eyeball vertex set), such as a mouth vertex set, an eyelid vertex set, an eyebrow vertex set, a nose vertex set, etc.

[0110] In some embodiments, step S300 specifically includes steps S310-S360:

[0111] S310: The coordinates of the vertices of the face vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model are used to perform a first migration of the vertices of the face vertex set of the target three-dimensional portrait model by using a double-layer deformation method, to obtain a first-migrated three-dimensional portrait model.

[0112] The two-layer deformation method involves selecting a portion of each vertex as control points, and then selecting a portion of each control point as keypoints. The movement of the control points is controlled by the displacement of the keypoints, and then the movement of each vertex is achieved through the control points. This process yields the changes in all vertices, allowing facial expressions to be transferred to the target 3D human model. Because the movement of all vertices is obtained using a relatively small number of keypoints, the computational load is reduced, and the selection of keypoints ensures the accuracy of expression transfer.

[0113] Step S310 specifically includes the following steps S311-S316:

[0114] S311: Select a portion of the vertices from the set of face vertices as each first control point, and select a portion of the first control points as each first key point;

[0115] S312: For each first key point, calculate the displacement of the first key point of the facial expression 3D portrait model relative to the first key point of the standard 3D portrait model, and use it as the displacement of the first key point;

[0116] S313: Calculate the new coordinates of each first control point based on the displacement of each first key point;

[0117] S314: For each first control point, calculate the displacement of the first control point based on the new coordinates of the first control point and the coordinates of the first control point in the standard three-dimensional human portrait model;

[0118] S315: Calculate the new coordinates of each vertex in the face vertex set based on the displacement of each first control point;

[0119] S316: Migrate each vertex of the face vertex set of the target 3D human portrait model to its new coordinates to obtain the 3D human portrait model after the first migration.

[0120] In steps S313 and S315, existing deformation algorithms can be used to calculate the new coordinates of each control point and each vertex.

[0121] For example, the face vertex set includes 478 vertices, 85 first control points, and 10 first key points. Each first key point is the right vertex, center point, and left vertex of the left eyebrow, the left vertex, center point, and right vertex of the right eyebrow, and the left vertex, right vertex, upper vertex, and lower vertex of the mouth.

[0122] Step S310 involves migrating all vertices in the face vertex set, and can therefore be called the global deformation stage. During this period, the opening and closing motion of the mouth can be temporarily ignored, so the displacements of the upper and lower vertices of the mouth can be set to 0.

[0123] S320: performing a second migration on each vertex of the upper mouth vertex set of the three-dimensional portrait model after the first migration based on the coordinates of each vertex of the upper mouth vertex set of the standard three-dimensional portrait and the expression three-dimensional portrait, performing a second migration on each vertex of the lower mouth vertex set of the three-dimensional portrait model after the first migration based on the coordinates of each vertex of the lower mouth vertex set of the standard three-dimensional portrait and the expression three-dimensional portrait, and obtaining a three-dimensional portrait model after the second migration.

[0124] In the second migration, the double-layer morphing method is also used, i.e., a plurality of control points are selected from all the vertices to be migrated, a plurality of key points are selected from the control points, the new coordinates of the control points are calculated through the displacement of the key points to control the movement of the control points, and then the new coordinates of the vertices are calculated through the displacement of the control points to control the movement of the vertices. In the second migration, the upper mouth vertex in the upper mouth vertex set can be taken as a key point to control the movement of the remaining vertices, and the lower mouth vertex in the lower mouth vertex set can be taken as a key point to control the movement of the remaining vertices. Since step S320 only needs to migrate the mouth vertices, it can be called a local morphing stage. During this period, the mouth left vertex and the mouth right vertex remain unchanged.

[0125] S330: obtaining a standard three-dimensional portrait model with the left eye fully closed as a left eye fully closed portrait model, obtaining a standard three-dimensional portrait model with the right eye fully closed as a right eye fully closed portrait model, performing a third migration on each vertex of the face vertex set of the target three-dimensional portrait model based on the coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and the left eye fully closed portrait model using the double-layer morphing method to obtain a three-dimensional portrait model after the third migration, and performing a fourth migration on each vertex of the face vertex set of the target three-dimensional portrait model based on the coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and the right eye fully closed portrait model to obtain a three-dimensional portrait model after the fourth migration.

[0126] The third migration and the fourth migration have the same method as the first migration, and the only difference is that the standard three-dimensional portrait model and the expression three-dimensional portrait model are used to calculate the displacement of the key points, the new coordinates of the control points, and the new coordinates of the vertices in the first migration, while the left eye fully closed portrait model and the expression three-dimensional portrait model are used to calculate in the third migration, and the right eye fully closed portrait model and the expression three-dimensional portrait model are used to calculate in the fourth migration.

[0127] S340: calculating new coordinates of each vertex of the left eyelid vertex set based on the coordinates of each vertex of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third migrated three-dimensional portrait model; calculating new coordinates of each vertex of the right eyelid vertex set based on the coordinates of each vertex of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the fourth migrated three-dimensional portrait model.

[0128] Step S340 specifically includes steps S341-S344:

[0129] S341: determining the left eye closure degree based on the coordinates of each vertex of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the left eye fully closed portrait model;

[0130] S342: determining new coordinates of each vertex of the left eyelid vertex set based on the left eye closure degree, the coordinates of each vertex of the left eyelid vertex set of the second migrated three-dimensional portrait model and the third migrated three-dimensional portrait model;

[0131] S343: determining the right eye closure degree based on the coordinates of each vertex of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the right eye fully closed portrait model;

[0132] S344: determining new coordinates of each vertex of the right eyelid vertex set based on the right eye closure degree, the coordinates of each vertex of the right eyelid vertex set of the second migrated three-dimensional portrait model and the fourth migrated three-dimensional portrait model.

[0133] Suppose the left eye closure degree is t, which is equal to the distance between the left eyelid vertices of the expression three-dimensional portrait model and the standard three-dimensional portrait model divided by the distance between the left eyelid vertices of the left eye fully closed portrait model and the standard three-dimensional portrait model (i.e. the ratio of the two distances), suppose the coordinates of each vertex of the left eyelid vertex set of the second migrated three-dimensional portrait model are , the coordinates of each vertex of the left eyelid vertex set of the third migrated three-dimensional portrait model are , and the new coordinates of each vertex of the left eyelid vertex set are , then:

[0134]

[0135] Similarly, the new coordinates of each vertex of the right eyelid vertex set can also be obtained by using the same calculation method, please refer to the calculation method of the left eyelid vertex set for details, which will not be repeated here.

[0136] S350: moving each vertex of the left eyelid vertex set and the right eyelid vertex set of the three-dimensional portrait model after the second migration to a new coordinate position respectively to obtain a three-dimensional portrait model after the fifth migration.

[0137] Through steps S330-S350, the eye sawtooth problem caused by global deformation can be solved, distortion is prevented, and the expression after migration is more natural.

[0138] S360: performing a sixth migration on each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration based on the coordinates of each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model to obtain a three-dimensional portrait model after the sixth migration as the three-dimensional portrait model after expression migration.

[0139] The eyeball, the inside mouth and the head three regions are not involved in deformation operation, and only a certain three-dimensional rotation is needed to realize migration.

[0140] Step S360 specifically includes the following steps S361-S364:

[0141] S361: respectively calculating the deviation angles of the eyeball vertex center, the inside mouth vertex center and the head vertex center in the expression three-dimensional portrait model and the standard three-dimensional portrait model on the x, y and z axes to obtain an eyeball rotation angle, an inside mouth rotation angle and a head rotation angle;

[0142] S362: determining the coordinates of the eyeball rotation center, the inside mouth rotation center and the head rotation center of the three-dimensional portrait model after the fifth migration based on the coordinates of each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration;

[0143] S363: determining the new coordinates of each vertex in the eyeball vertex set based on the eyeball rotation angle and the coordinates of the eyeball rotation center of the three-dimensional portrait model after the fifth migration, determining the new coordinates of each vertex in the inside mouth vertex set based on the inside mouth rotation angle and the coordinates of the inside mouth rotation center of the three-dimensional portrait model after the fifth migration, and determining the new coordinates of each vertex in the head vertex set based on the head rotation angle and the coordinates of the head rotation center of the three-dimensional portrait model after the fifth migration;

[0144] S364: migrating each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration to a new coordinate position respectively to obtain a three-dimensional portrait model after the sixth migration.

[0145] When the eye, the inside of the mouth and the head are rotated, the rotation matrix is as follows:

[0146]

[0147] wherein, , , are the angles of rotation around the x, y and z axes respectively.

[0148] The x coordinate of the eye rotation center is the average of the x coordinates of all the vertices in the eye vertex set, the y coordinate is the average of the y coordinates of all the vertices in the eye vertex set, and the z coordinate is the minimum of the z coordinates of all the vertices in the eye vertex set. The coordinates of the eye rotation center can be calculated by the following formula:

[0149]

[0150] wherein, , , are the x, y and z coordinates of the eye rotation center respectively, and n is the number of vertices in the eye vertex set, , , are the x, y and z coordinates of the vertex i in the eye vertex set, and i is the vertex serial number.

[0151] The new coordinates of the vertices in the eye vertex set are calculated as follows:

[0152]

[0153] wherein, is the new coordinate of the vertex i, is the coordinate (i.e. the original coordinate) of the vertex i of the three-dimensional portrait model after the fifth migration, is the coordinate of the eye rotation center, is the rotation angle of the x axis of the eye, is the rotation matrix obtained by substituting into in the rotation matrix, is the rotation angle of the y axis of the eye, is the rotation angle obtained by substituting into in the rotation matrix. The influence of the z axis rotation angle does not need to be considered when calculating the eye (i.e. the eye does not rotate along the z axis).

[0154] The rotation of the inside of the mouth (including the teeth, the inner wall of the mouth and the tongue) is the same as that of the eye (i.e. the calculation method of the new coordinates of the vertices of the inside of the mouth is the same as that of the eye), which is not described here again.

[0155] For the head, a Cartesian three-dimensional coordinate system can be established with the head rotation center as the origin, and when rotating around the x-axis, it behaves as up-and-down rotating the head, when rotating around the y-axis, it behaves as left-and-right rotating the head, and when rotating around the z-axis, it behaves as tilting the head, the rotation angles of the head around the x, y and z axes are calculated according to the change of the coordinates of the nose tip, and then the new coordinates of the vertex of the head after migration are calculated according to the following formula:

[0156]

[0157] wherein, is the new coordinates of the vertex of the head, is the original coordinates, is the coordinates of the head rotation center, , , are the rotation angles of the head around the x, y and z axes respectively, , , is the rotation matrix of the head.

[0158] The three-dimensional facial expression migration method of the embodiment of the application introduces the concepts of global double-layer deformation and local double-layer deformation on the basis of the traditional free deformation algorithm (DFFD), migrates the vertex of the eyelid by fusing the standard three-dimensional portrait model and the eye-closed portrait model to solve the sawtooth problem of the eyes caused by global deformation, and avoids the feature distortion phenomenon in the complex expression migration process by rotating the eyeball, the inside of the mouth and the head, so that relatively fine expression migration can be realized at a relatively small cost, and the occurrence of facial structure distortion can be prevented.

[0159] As shown in Figure 2 , the embodiment of the application further provides a three-dimensional facial expression migration device, which comprises an acquisition module 10, a three-dimensional reconstruction module 20 and an expression migration module 30.

[0160] The acquisition module 10 is used for acquiring a target facial image.

[0161] The three-dimensional reconstruction module 20 is used for performing three-dimensional reconstruction on the target facial image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model.

[0162] The expression migration module 30 is used for migrating the expression on the expression three-dimensional portrait model to the target three-dimensional portrait model to obtain a three-dimensional portrait model after expression migration.

[0163] The acquisition module 10, the three-dimensional reconstruction module 20 and the expression migration module 30 are functional modules for realizing steps S100-S300 of the three-dimensional facial expression migration method, and the specific implementation methods can be referred to the description in the method embodiment, which will not be described here again.

[0164] Yet another embodiment of the present application provides a readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the steps of the three-dimensional facial expression transfer method in the above-mentioned embodiments of the present application.

[0165] Yet another embodiment of the present application provides an electronic device comprising a memory and a processor, the memory having stored thereon executable code, which, when executed in the processor, performs the steps of the three-dimensional facial expression transfer method in the above-mentioned embodiments of the present application.

[0166] The system, apparatus, module or unit illustrated in the above-mentioned embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0167] For the convenience of description, the above apparatus is described in various units by function respectively in the description. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0168] Those skilled in the art should understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in accordance with the flowcharts and / or block diagrams. Figure 1 The device for performing the function specified in one flow or multiple flows and / or blocks Figure 1 The device for performing the function specified in one flow or multiple flows and / or blocks

[0170] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0172] In a typical configuration, an electronic device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0173] The memory can include non-persistent memory and / or volatile memory, e.g., random access memory (RAM) and / or non-volatile memory, e.g., read-only memory (ROM) or flash memory. The memory is an example of computer readable media.

[0174] Computer readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by an electronic device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carriers.

[0175] It is also important to note that the term "comprising" or "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0176] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of computer-implemented processes, methods, or apparatuses. Accordingly, embodiments of the present application can be embodied in a variety of different forms, all of which have been contemplated to be within the scope of the application. In addition, each of the embodiments described herein can be implemented, individually and / or in combination, by a computer system having a processor and a memory storing computer program instructions executed by the processor. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a "circuit" or "module." Furthermore, the present application can take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium.

[0177] The present application can be described in the general context of computer- executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0178] Embodiments of the present application are described with reference to the attached figures. The same numbers are used throughout the drawings and corresponding elements have the same numbers. Each figure can be, and often is, utilized to better illustrate at least one aspect of the present application. Each figure can be utilized alone or in combination with at least one or more other figures. Embodiments of the present application will now be described, by way of example only, with reference to the attached figures.

[0179] The above description of the specific embodiments of the application has been presented for the purpose of illustration and description. Other embodiments are within the scope of the following claims. In some instances, the acts or steps can be performed in different order than described and still achieve desirable results. Additionally, the process depicted in the figures does not necessarily require the particular order shown, or sequential order to achieve desirable results. In some instances, multitasking and parallel processing can be advantageous.

[0180] The above merely describes preferred embodiments of the present application, and is not intended to limit the scope of the present application. The above-described embodiments of the present application can be variously changed. That is, simple, equivalent changes and modifications made in accordance with the content of the claims and the specification of the present application are intended to fall within the scope of the present application. The present application is not limited by the above-described embodiments.

Claims

1. A three-dimensional face expression transfer method, characterized in that, The method comprises the following steps: obtaining a target face image; performing three-dimensional reconstruction on the target face image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model; migrating an expression on an expression three-dimensional portrait model to the target three-dimensional portrait model to obtain a three-dimensional portrait model after expression migration.

2. The three-dimensional facial expression transfer method of claim 1, wherein, The method for performing three-dimensional reconstruction on the target face image based on the standard three-dimensional portrait model to obtain the target three-dimensional portrait model comprises the following steps: deforming a standard mesh model based on the target face image to obtain a deformed mesh model; converting the deformed mesh model to a coordinate system of the standard mesh model to obtain a target mesh model; replacing a standard portrait skin based on the target face image to obtain a target portrait skin; attaching the target portrait skin to the target mesh model to obtain the target three-dimensional portrait model.

3. The three-dimensional facial expression transfer method of claim 2, wherein, The method for replacing the standard portrait skin based on the target face image to obtain the target portrait skin comprises the following steps: dividing the standard portrait skin into a standard face region and a standard non-face region, wherein the face region is an expression main performance region including eyes, a nose, a forehead, a mouth and the like, and the non-face region is other regions except the face region; detecting a target face region corresponding to the standard face region on the target face image, replacing skin of the target face region with skin of the standard face region to obtain skin of the standard face region after replacement; detecting a target non-face region corresponding to the standard non-face region on the target face image, and obtaining skin of the standard non-face region after replacement based on skin of the standard non-face region and skin of the target non-face region; fusing the skin of the standard face region after replacement and the skin of the standard non-face region after replacement to obtain the target portrait skin.

4. The three-dimensional facial expression transfer method of claim 1, wherein, The method for migrating the expression on the expression three-dimensional portrait model to the target three-dimensional portrait model to obtain the three-dimensional portrait model after expression migration comprises the following steps: performing first migration on each vertex of a face vertex set of the target three-dimensional portrait model based on coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and a standard three-dimensional portrait by using a double-layer deformation method to obtain a three-dimensional portrait model after first migration; performing second migration on each vertex of an upper mouth vertex set of the three-dimensional portrait model after first migration based on coordinates of each vertex of the upper mouth vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait, and performing second migration on each vertex of a lower mouth vertex set of the three-dimensional portrait model after first migration based on coordinates of each vertex of the lower mouth vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait to obtain a three-dimensional portrait model after second migration. obtaining a standard three-dimensional portrait model with the left eye fully closed as a left eye fully closed portrait model; obtaining a standard three-dimensional portrait model with the right eye fully closed as a right eye fully closed portrait model; performing third migration on each vertex of the face vertex set of the target three-dimensional portrait model based on the coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and the left eye fully closed portrait model by using the double-layer deformation method to obtain a third-migrated three-dimensional portrait model; and performing fourth migration on each vertex of the face vertex set of the target three-dimensional portrait model based on the coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and the right eye fully closed portrait model to obtain a fourth-migrated three-dimensional portrait model; calculating new coordinates of each vertex of the left eyelid vertex set based on the coordinates of each vertex of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third-migrated three-dimensional portrait model; and calculating new coordinates of each vertex of the right eyelid vertex set based on the coordinates of each vertex of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the fourth-migrated three-dimensional portrait model; moving each vertex of the left eyelid vertex set and the right eyelid vertex set of the second-migrated three-dimensional portrait model to the new coordinate position to obtain a fifth-migrated three-dimensional portrait model; performing sixth migration on each vertex of the eyeball vertex set, the internal oral cavity vertex set and the head vertex set of the fifth-migrated three-dimensional portrait model based on the coordinates of each vertex of the eyeball vertex set, the internal oral cavity vertex set and the head vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model to obtain a sixth-migrated three-dimensional portrait model as the expression-migrated three-dimensional portrait model.

5. The three-dimensional facial expression transfer method of claim 4, wherein, performing first migration on each vertex of the face vertex set of the target three-dimensional portrait model based on the coordinates of each vertex of the face vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait to obtain a first-migrated three-dimensional portrait model by using the double-layer deformation method, specifically including: selecting part of the vertices from the face vertex set as each first control point, and selecting part of the first control points as each first key point; calculating the displacement of each first key point of the expression three-dimensional portrait model relative to the first key point of the standard three-dimensional portrait as the displacement of the first key point; calculating new coordinates of each first control point according to the displacements of the first key points; calculating the displacement of each first control point according to the new coordinates of the first control point and the coordinates of the first control point of the standard three-dimensional portrait; calculating new coordinates of each vertex of the face vertex set according to the displacements of the first control points; migrating each vertex of the face vertex set of the target three-dimensional portrait model to the new coordinate to obtain the first-migrated three-dimensional portrait model.

6. The three-dimensional facial expression transfer method of claim 4, wherein, calculating new coordinates of each vertex of the left eyelid vertex set based on the coordinates of each vertex of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third-migrated three-dimensional portrait model; and calculating new coordinates of each vertex of the right eyelid vertex set based on the coordinates of each vertex of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the fourth-migrated three-dimensional portrait model; The new coordinates of the vertices of the right eyelid vertex set are calculated based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third three-dimensional portrait model after the third migration, and specifically include: The left eye closure degree is determined based on the coordinates of the vertices of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the left eye fully closed portrait model; The new coordinates of the vertices of the left eyelid vertex set are determined based on the left eye closure degree, the coordinates of the vertices of the left eyelid vertex set of the second three-dimensional portrait model after the second migration and the third three-dimensional portrait model after the third migration; The right eye closure degree is determined based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the right eye fully closed portrait model; The new coordinates of the vertices of the right eyelid vertex set are determined based on the right eye closure degree, the coordinates of the vertices of the right eyelid vertex set of the second three-dimensional portrait model after the second migration and the fourth three-dimensional portrait model after the fourth migration.

7. The three-dimensional facial expression transfer method of claim 4, wherein, The sixth migration is performed on the vertices of the eyeball vertex set, the internal oral cavity vertex set and the head vertex set of the fifth three-dimensional portrait model after the fifth migration based on the coordinates of the vertices of the eyeball vertex set, the internal oral cavity vertex set and the head vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model, to obtain a sixth three-dimensional portrait model after the sixth migration, as the three-dimensional portrait model after the expression migration, and specifically includes: The deviation angles of the eyeball vertex center, the internal oral cavity vertex center and the head vertex center in the expression three-dimensional portrait model and the standard three-dimensional portrait model in the x, y and z axes are calculated respectively to obtain the eyeball rotation angle, the oral cavity rotation angle and the head rotation angle; The coordinates of the eyeball rotation center, the internal oral cavity rotation center and the head rotation center of the fifth three-dimensional portrait model after the fifth migration are determined based on the coordinates of the vertices of the eyeball vertex set, the internal oral cavity vertex set and the head vertex set of the fifth three-dimensional portrait model after the fifth migration; The new coordinates of each vertex in the eyeball vertex set are determined based on the eyeball rotation angle and the coordinates of the eyeball rotation center of the fifth three-dimensional portrait model after the fifth migration, the new coordinates of each vertex in the internal oral cavity vertex set are determined based on the oral cavity rotation angle and the coordinates of the internal oral cavity rotation center of the fifth three-dimensional portrait model after the fifth migration, and the new coordinates of each vertex in the head vertex set are determined based on the head rotation angle and the coordinates of the head rotation center of the fifth three-dimensional portrait model after the fifth migration; The vertices of the eyeball vertex set, the internal oral cavity vertex set and the head vertex set of the fifth three-dimensional portrait model after the fifth migration are migrated to the new coordinates respectively to obtain the sixth three-dimensional portrait model after the sixth migration.

8. A three-dimensional face expression transfer device, comprising: It includes: An acquisition module is configured to acquire a target face image; A three-dimensional reconstruction module is configured to perform three-dimensional reconstruction on the target face image based on a standard three-dimensional portrait model to obtain a target three-dimensional portrait model; An expression migration module is configured to migrate an expression on an expression three-dimensional portrait model to the target three-dimensional portrait model to obtain a three-dimensional portrait model after expression migration.

9. The three-dimensional facial expression transfer apparatus of claim 8, wherein, The target three-dimensional portrait model is obtained by performing three-dimensional reconstruction on the target face image based on a standard three-dimensional portrait model, and specifically includes the following steps: The standard mesh model is deformed based on the target face image to obtain a deformed mesh model; The deformed mesh model is converted to the coordinate system of the standard mesh model to obtain a target mesh model; The standard portrait skin is replaced based on the target face image to obtain a target portrait skin; The target portrait skin is pasted onto the target mesh model to obtain the target three-dimensional portrait model; The standard portrait skin is replaced based on the target face image to obtain a target portrait skin, and specifically includes the following steps: The standard portrait skin is divided into a standard face region and a standard non-face region, the face region is a main expression region including eyes, nose, forehead, and mouth, and the non-face region is other regions except the face region; The target face region corresponding to the standard face region is detected on the target face image, and the skin of the target face region is replaced with the skin of the standard face region to obtain the skin of the replaced standard face region; The target non-face region corresponding to the standard non-face region is detected on the target face image, and the skin of the target non-face region is replaced with the skin of the standard non-face region to obtain the skin of the replaced standard non-face region; The skin of the replaced standard face region and the skin of the replaced standard non-face region are fused to obtain the target portrait skin.

10. The three-dimensional facial expression transfer apparatus of claim 8, wherein, The expression of the expression three-dimensional portrait model is migrated to the target three-dimensional portrait model to obtain an expression-migrated three-dimensional portrait model, and specifically includes the following steps: The coordinates of the vertices of the face vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait are used to perform a first migration on the vertices of the face vertex set of the target three-dimensional portrait model by using a double-layer deformation method, to obtain a first-migrated three-dimensional portrait model; The coordinates of the vertices of the upper mouth vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait are used to perform a second migration on the vertices of the upper mouth vertex set of the first-migrated three-dimensional portrait model, and the coordinates of the vertices of the lower mouth vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait are used to perform a second migration on the vertices of the lower mouth vertex set of the first-migrated three-dimensional portrait model, to obtain a second-migrated three-dimensional portrait model; A standard three-dimensional portrait model with the left eye completely closed is obtained as a left eye completely closed portrait model, and a standard three-dimensional portrait model with the right eye completely closed is obtained as a right eye completely closed portrait model; the coordinates of the vertices of the face vertex set of the expression three-dimensional portrait model and the left eye completely closed portrait model are used to perform a third migration on the vertices of the face vertex set of the target three-dimensional portrait model by using a double-layer deformation method, to obtain a third-migrated three-dimensional portrait model; and the coordinates of the vertices of the face vertex set of the expression three-dimensional portrait model and the right eye completely closed portrait model are used to perform a fourth migration on the vertices of the face vertex set of the target three-dimensional portrait model, to obtain a fourth-migrated three-dimensional portrait model; calculating new coordinates of the vertices of the left eyelid vertex set based on the coordinates of the vertices of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third time migrated three-dimensional portrait model; and calculating new coordinates of the vertices of the right eyelid vertex set based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the fourth time migrated three-dimensional portrait model; moving the vertices of the left eyelid vertex set and the right eyelid vertex set of the second time migrated three-dimensional portrait model to the new coordinate positions respectively to obtain a fifth time migrated three-dimensional portrait model; sixth time migrating the vertices of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the fifth time migrated three-dimensional portrait model based on the coordinates of the vertices of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model to obtain a sixth time migrated three-dimensional portrait model as the expression migrated three-dimensional portrait model; first time migrating the vertices of the face vertex set of the target three-dimensional portrait model based on the coordinates of the vertices of the face vertex set of the expression three-dimensional portrait model and the standard three-dimensional portrait model by using a double-layer deformation method to obtain a first time migrated three-dimensional portrait model, specifically including: selecting part of the vertices from the face vertex set as first control points, and selecting part of the first control points as first key points; calculating, for each first key point, a displacement of the first key point of the expression three-dimensional portrait model relative to the first key point of the standard three-dimensional portrait model as a displacement of the first key point; calculating new coordinates of the first control points according to the displacements of the first key points; calculating, for each first control point, a displacement of the first control point according to the new coordinates of the first control point and the coordinates of the first control point of the standard three-dimensional portrait model; calculating new coordinates of the vertices of the face vertex set according to the displacements of the first control points; migrating the vertices of the face vertex set of the target three-dimensional portrait model to the new coordinates respectively to obtain the first time migrated three-dimensional portrait model; calculating new coordinates of the vertices of the left eyelid vertex set based on the coordinates of the vertices of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the third time migrated three-dimensional portrait model; and calculating new coordinates of the vertices of the right eyelid vertex set based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the fourth time migrated three-dimensional portrait model, specifically including: determining a left eye closing degree based on the coordinates of the vertices of the left eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the left eye fully closed portrait model; determining new coordinates of the vertices of the left eyelid vertex set based on the left eye closing degree, the coordinates of the vertices of the left eyelid vertex set of the second time migrated three-dimensional portrait model and the third time migrated three-dimensional portrait model; determining a right eye closing degree based on the coordinates of the vertices of the right eyelid vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model and the right eye fully closed portrait model; determining new coordinates of each vertex of the right eyelid vertex set based on the coordinates of each vertex of the right eyelid vertex set of the three-dimensional portrait model after the fourth migration and the degree of closure of the right eye; performing a sixth migration on each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration based on the coordinates of each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the expression three-dimensional portrait model, the standard three-dimensional portrait model, to obtain a three-dimensional portrait model after the sixth migration as the three-dimensional portrait model after the expression migration, specifically comprising: calculating the deviation angles of the eyeball vertex center, the inside mouth vertex center and the head vertex center in the expression three-dimensional portrait model and the deviation angles of the eyeball vertex center, the inside mouth vertex center and the head vertex center in the standard three-dimensional portrait model on the x, y and z axes respectively to obtain the eyeball rotation angle, the inside mouth rotation angle and the head rotation angle; determining the coordinates of the eyeball rotation center, the inside mouth rotation center and the head rotation center of the three-dimensional portrait model after the fifth migration based on the coordinates of each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration; determining the new coordinates of each vertex of the eyeball vertex set based on the eyeball rotation angle and the coordinates of the eyeball rotation center of the three-dimensional portrait model after the fifth migration, determining the new coordinates of each vertex of the inside mouth vertex set based on the inside mouth rotation angle and the coordinates of the inside mouth rotation center of the three-dimensional portrait model after the fifth migration, and determining the new coordinates of each vertex of the head vertex set based on the head rotation angle and the coordinates of the head rotation center of the three-dimensional portrait model after the fifth migration; migrating each vertex of the eyeball vertex set, the inside mouth vertex set and the head vertex set of the three-dimensional portrait model after the fifth migration to the new coordinates respectively to obtain the three-dimensional portrait model after the sixth migration.

Images