Three-dimensional mesh determination method, apparatus, device, medium and product

Abstract

The present disclosure provides a three-dimensional mesh determination method, an apparatus, a device, a medium and a product. The method comprises: first acquiring a two-dimensional image, such that the two-dimensional image can describe at least one region of a face in a two-dimensional space; then, constructing a three-dimensional mesh corresponding to the two-dimensional image, such that the three-dimensional mesh can describe at least one region in a three-dimensional space, and acquiring a key point detection result of the two-dimensional image, such that the key point detection result comprises key points of the at least one region; and then, on the basis of the key point detection result, performing deformation processing on the three-dimensional mesh to obtain a deformed mesh, such that the state of the at least one region described by the deformed mesh in the two-dimensional space is consistent with the state of the at least one region described by the key point detection result, or the state of the at least one region described by the deformed mesh is consistent with the state of the at least one region described by the key point detection result in the three-dimensional space, thus helping to improve the mesh construction effect.

Description

A method, apparatus, equipment, medium, and product for determining a three-dimensional mesh.

[0001] This application claims priority to Chinese Patent Application No. 202410804691.4, filed on June 20, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to a method, apparatus, device, medium, and product for determining a three-dimensional mesh. Background Technology

[0003] In some scenarios, such as game design or virtual reality scenarios, the following requirements may exist: first, obtain a two-dimensional image provided by the user, which is used to describe the state of the face in two-dimensional space; then, construct a three-dimensional mesh corresponding to the two-dimensional image, which is used to describe the state of the face in three-dimensional space.

[0004] However, some 3D mesh construction schemes have defects, resulting in poor mesh construction effects based on these schemes. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, medium, and product for determining three-dimensional meshes, which is beneficial for improving mesh construction results.

[0006] To achieve the above objectives, the technical solution provided in this disclosure is as follows:

[0007] This disclosure provides a method for determining a three-dimensional mesh. The method includes: acquiring a two-dimensional image, the two-dimensional image being used to describe at least one region of a face in a two-dimensional space; constructing a three-dimensional mesh corresponding to the two-dimensional image; and acquiring key point detection results of the two-dimensional image, the three-dimensional mesh being used to describe the at least one region in a three-dimensional space, the key point detection results including key points of the at least one region; and performing deformation processing on the three-dimensional mesh based on the key point detection results to obtain a deformed mesh, wherein the state of the at least one region described by the deformed mesh in the two-dimensional space is consistent with the state of the at least one region described by the key point detection results, or the state of the at least one region described by the deformed mesh is consistent with the state of the at least one region described by the key point detection results in the three-dimensional space.

[0008] In one possible implementation, for any of the regions, the projection points of the vertices on the deformed mesh used to describe the region in the two-dimensional space coincide with the key points of the region; or, the projection points of the vertices on the deformed mesh used to describe the region coincide with the projection points of the key points of the region in the three-dimensional space.

[0009] In one possible implementation, the process of determining the deformed mesh includes: for any key point, finding a first vertex in the three-dimensional mesh that matches the key point, and determining the deformation parameters of the first vertex based on the distance between the projection point of the key point in the three-dimensional space and the first vertex; determining the deformation parameters of the three-dimensional mesh based on the deformation parameters of the first vertex, wherein the deformation parameters of the three-dimensional mesh include the deformation parameters of the first vertex; and performing deformation processing on the three-dimensional mesh according to the deformation parameters of the three-dimensional mesh to obtain the deformed mesh.

[0010] In one possible implementation, the at least one region includes a first region, and the key points of the first region include a first key point;

[0011] The process of determining the first vertex matching the first key point includes: obtaining the projection result of the three-dimensional mesh onto the two-dimensional image, the projection result including the projection points of each vertex on the three-dimensional mesh in the two-dimensional space; determining the first vertex matching the first key point based on the distance between the first key point and the projection points of each vertex in the two-dimensional space; the projection result includes a target projection point, the target projection point being the projection point of the first vertex matching the first key point in the two-dimensional space, and the distance between the first key point and the target projection point being no greater than the distance between the first key point and other projection points in the projection result besides the target projection point.

[0012] In one possible implementation, the first region is the cheek;

[0013] After obtaining the projection result of the three-dimensional mesh onto the two-dimensional image, the method further includes: determining a projection point set from the projection result, the projection point set being used to represent the cheek edge formed by the projection points of some vertices of the three-dimensional mesh in the two-dimensional space;

[0014] The step of determining the first vertex matching the first key point based on the distance between the first key point and the projection points of each vertex in the two-dimensional space includes: determining the first vertex matching the first key point based on the distance between the first key point and each projection point in the projection point set, wherein the projection point set includes the target projection point, and the distance between the first key point and the target projection point is not greater than the distance between the first key point and other projection points in the projection point set besides the target projection point.

[0015] In one possible implementation, the at least one region includes a second region, and the key points of the second region include second key points;

[0016] The process of determining the first vertex that matches the second key point includes: obtaining vertex annotation information of the three-dimensional mesh, wherein the vertex annotation information is used to describe the mapping relationship between each vertex on the three-dimensional mesh and the key point of the at least one region; and determining the first vertex that matches the second key point based on the vertex annotation information, wherein the mapping relationship includes the correspondence between the first vertex that matches the second key point and the second key point.

[0017] In one possible implementation, the difference between the distribution state of key points in the second region and the standard distribution state corresponding to the second region is lower than a preset difference, wherein the standard distribution state is determined based on the key point detection results of at least one frontal face image.

[0018] In one possible implementation, the three-dimensional mesh contains at least one vertex for describing a symmetrical region of the second region; the at least one vertex includes a second vertex, and there is a symmetrical relationship between the second vertex and the first vertex that matches the second keypoint;

[0019] The process of determining the deformation parameters of the three-dimensional mesh includes: if the difference between the distribution state of the key points in the symmetrical region and the standard distribution state corresponding to the symmetrical region is not less than a preset difference, then the deformation parameters of the second vertex are predicted based on the deformation parameters of the first vertex that matches the second key point; and the deformation parameters of the three-dimensional mesh are determined based on the deformation parameters of the second vertex and the deformation parameters of the first vertex that matches the second key point.

[0020] In one possible implementation, the third vertex is any vertex in the three-dimensional mesh other than the first vertex and the second vertex; the deformation parameters of the three-dimensional mesh also include the deformation parameters of the third vertex, which are obtained by interpolation based on the deformation parameters of the first vertex and the deformation parameters of the second vertex.

[0021] In one possible implementation, the method further includes: acquiring camera parameters of the two-dimensional image; determining the rotation angle of the face based on the camera parameters; if the rotation angle is 0, then each region in the at least one region is determined as a second region; if the rotation angle is greater than 0, then the cheek in the at least one region is determined as a first region, and the other regions in the at least one region besides the cheek are determined as second regions.

[0022] In one possible implementation, the at least one region includes part or all of the cheek, eyes, nose, mouth, and ears.

[0023] In one possible implementation, the keypoint detection results include at least 20 keypoints of the left ear and 20 keypoints of the right ear; the 20 keypoints of the left ear are used to describe the outer contour of the left ear; the 20 keypoints of the right ear are used to describe the outer contour of the right ear; and / or, the keypoint detection results include at least 20 keypoints of the iris of the left eye and 20 keypoints of the iris of the right eye; the 20 keypoints of the iris of the left eye are used to describe the iris contour of the left eye; the 20 keypoints of the iris of the right eye are used to describe the iris contour of the right eye.

[0024] In one possible implementation, the key point detection results further include 16 key points in the eye socket of the left eye, 16 key points in the eye socket of the right eye, 4 key points on the left side of the nose, and 4 key points on the right side of the nose; the 16 key points in the eye socket of the left eye are used to describe the outline of the left eye; the 16 key points in the eye socket of the right eye are used to describe the outline of the right eye; the 4 key points on the left side of the nose are used to describe the outline of the left side of the bridge of the nose; and the 4 key points on the right side of the nose are used to describe the outline of the right side of the bridge of the nose.

[0025] In one possible implementation, the construction process of the three-dimensional mesh includes: performing three-dimensional reconstruction processing on the two-dimensional image to obtain a reconstruction result; and performing retopology processing on the reconstruction result according to a target mesh standard to obtain the three-dimensional mesh, wherein the three-dimensional mesh conforms to the target mesh standard.

[0026] In one possible implementation, the method further includes: obtaining a texture map corresponding to the two-dimensional image under the deformed mesh; rendering the deformed mesh according to the optimization processing result of the texture map to obtain a three-dimensional face model; the optimization processing result is determined based on at least one pre-constructed texture optimization model, and different texture optimization models have different optimization functions.

[0027] This disclosure provides a three-dimensional mesh determination apparatus, comprising:

[0028] An acquisition unit is used to acquire a two-dimensional image, the two-dimensional image being used to describe at least one region of a face in a two-dimensional space;

[0029] A construction unit is used to construct a three-dimensional mesh corresponding to the two-dimensional image and to obtain key point detection results of the two-dimensional image. The three-dimensional mesh is used to describe the at least one region in three-dimensional space, and the key point detection results include key points of the at least one region.

[0030] A deformation unit is used to deform the three-dimensional mesh based on the key point detection results to obtain a deformed mesh. The state of at least one region described by the deformed mesh in the two-dimensional space is consistent with the state of at least one region described by the key point detection results, or the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in the three-dimensional space.

[0031] This disclosure provides an electronic device, the device comprising: a processor and a memory;

[0032] The memory is used to store instructions or computer programs;

[0033] The processor is configured to execute the instructions or computer program in the memory, so that the electronic device performs the three-dimensional mesh determination method provided in this disclosure.

[0034] This disclosure provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the three-dimensional mesh determination method provided in this disclosure.

[0035] This disclosure provides a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the three-dimensional mesh determination method provided in this disclosure. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 is a flowchart of a three-dimensional mesh determination method provided in an embodiment of this disclosure;

[0038] Figure 2 is a schematic diagram of the construction process of a three-dimensional face model provided in an embodiment of this disclosure;

[0039] Figure 3 is a schematic diagram of facial key points provided in an embodiment of this disclosure;

[0040] Figure 4 is a schematic diagram of another facial key point provided in an embodiment of this disclosure;

[0041] Figure 5 is a schematic diagram of a three-dimensional mesh determination device provided in an embodiment of this disclosure;

[0042] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0043] Research has found that some 3D mesh construction schemes can be implemented as follows: First, a model is trained using sample images so that the trained model can learn how to construct the 3D mesh corresponding to these sample images, thus enabling the trained model to present a good mesh construction effect on these sample images; then, the trained model is used to process the 2D image provided by the user to obtain the 3D mesh corresponding to the 2D image.

[0044] The study also found that the above-mentioned scheme has the following drawbacks: because these sample images can only cover a limited number of facial features, such as round chins and round ears, the model trained based on these sample images is more suitable for processing two-dimensional images with these facial features. As a result, when the model is used to process two-dimensional images with other facial features, such as pointed chins and pointed ears, it will produce a relatively poor mesh construction effect.

[0045] Based on the above research, in order to improve the mesh construction effect, this disclosure provides a three-dimensional mesh determination method. The method includes: first, acquiring a two-dimensional image so that the two-dimensional image can describe at least one region of the face in two-dimensional space, such as facial features and cheeks; then, constructing a three-dimensional mesh corresponding to the two-dimensional image so that the three-dimensional mesh can describe the at least one region in three-dimensional space, and acquiring the key point detection results of the two-dimensional image so that the key point detection results include key points of at least one region, thereby making the key point detection results used to describe the characteristics of these regions, such as shape or size; then, based on the key point detection results, performing deformation processing on the three-dimensional mesh to obtain a deformed mesh, so that the state of at least one region described by the deformed mesh in two-dimensional space is consistent with the state of at least one region described by the key point detection results, or so that the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in three-dimensional space, thereby enabling the deformed mesh to more accurately describe the characteristics of at least one region presented in the two-dimensional image. This can overcome the defects mentioned in the above research by means of deformation processing, thereby improving the mesh construction effect.

[0046] Furthermore, this disclosure does not limit the executing entity of the three-dimensional mesh determination method provided in the embodiments of this disclosure. For example, the three-dimensional mesh determination method provided in the embodiments of this disclosure can be applied to a terminal device or a server. Alternatively, the three-dimensional mesh determination method provided in the embodiments of this disclosure can also be implemented through a data interaction process between a terminal device and a server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0047] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present disclosure.

[0048] To better understand the technical solutions provided in this disclosure, the three-dimensional mesh determination method provided in this disclosure will be described below with reference to some accompanying drawings. As shown in Figure 1, the three-dimensional mesh determination method provided in this disclosure includes S1-S4 as described below. Figure 1 is a flowchart of a three-dimensional mesh determination method provided in an embodiment of this disclosure.

[0049] S1: Acquire a two-dimensional image for describing at least one region of the face in two-dimensional space.

[0050] Two-dimensional images are used to describe the state of part or all of the face in two-dimensional space, such as the distribution of facial features. It should be noted that the face refers to the part located above the neck and directly in front of the head, and the face can include areas such as eyebrows, eyes, ears, nose, mouth, and cheeks. Furthermore, facial features include eyebrows, eyes, ears, nose, and mouth.

[0051] Furthermore, two-dimensional space refers to the space in which a two-dimensional image is located; and this disclosure does not limit the implementation of the two-dimensional space. For example, the two-dimensional space may include at least all pixels in the two-dimensional image.

[0052] Furthermore, at least one region refers to a region clearly presented in a two-dimensional image, such as a complete left eye, a complete ear, or a cheek. For example, if the rotation angle of the face in the two-dimensional image is 0, then the two-dimensional image is a frontal image, so that the two-dimensional image can clearly describe the eyebrows, eyes, ears, nose, mouth, and cheeks of the face, thereby making the at least one region include the eyebrows, eyes, ears, nose, mouth, and cheeks. As another example, if the rotation angle of the face in the two-dimensional image is greater than 0, then the two-dimensional image is a side profile image, so that the two-dimensional image can describe a portion of the eyebrows, eyes, ears, nose, mouth, and cheeks of the face, such as the area involving the left half of the face, thereby making the at least one region include that portion of the face. Therefore, in one possible implementation, the at least one region can include part or all of the cheeks, eyes, nose, mouth, and ears.

[0053] Furthermore, this disclosure does not limit the method of acquiring the two-dimensional image. For example, in some scenarios, the two-dimensional image may refer to a facial image input by a user using a certain input device, so that the facial image can be used to describe the face of an object. It should be noted that this disclosure does not limit the implementation method of the object; for example, it may be implemented using a real person, a digital human, an anime character, a robot, or an animal. Additionally, this disclosure does not limit the implementation method of the input device; for example, it may be implemented using any existing or future device capable of image input, such as a camera.

[0054] S2: Construct a three-dimensional mesh corresponding to the two-dimensional image, which is used to describe at least one region in three-dimensional space.

[0055] The 3D mesh is used to describe the state of the face in 3D space.

[0056] Furthermore, three-dimensional space refers to the space in which the three-dimensional mesh is located; and this disclosure does not limit the implementation of the three-dimensional space. For example, the three-dimensional space may include at least all vertices in the three-dimensional mesh.

[0057] Furthermore, this disclosure does not limit the implementation of S2 above. For example, in some scenarios, S2 may include: performing three-dimensional reconstruction processing on a two-dimensional image to obtain a three-dimensional mesh corresponding to the two-dimensional image, such as mesh 1 as shown in Figure 2. It should be noted that this disclosure does not limit the implementation of the three-dimensional reconstruction processing. For example, it can be implemented using any existing or future method capable of performing three-dimensional mesh reconstruction processing on an image, such as a method based on a 3D Morphable Face Model (3DMM) reconstruction, or a method implemented using a pre-built machine learning model with three-dimensional mesh reconstruction capabilities. Therefore, in one possible implementation, the three-dimensional mesh can be obtained by processing the two-dimensional image using a 3DMM-based reconstruction method. Here, 3DMM is a statistical model of facial shape and appearance.

[0058] Furthermore, for some scenarios, such as game design scenarios, the 3D meshes involved in the scenario need to conform to a specific standard, such as the MetaHuman standard. Therefore, in order to better meet this requirement, this disclosure provides a possible implementation of S2 above. In this implementation, S2 may specifically include steps 11-12 below. MetaHuman is an open-source tool for creating and analyzing human character animations, which provides the functionality to create and edit complex meshes.

[0059] Step 11: Perform 3D reconstruction processing on the 2D image to obtain the reconstruction result.

[0060] It should be noted that the relevant content of the 3D reconstruction process in step 11 can be found above.

[0061] As can be seen, for some scenarios, after obtaining a two-dimensional image, as shown in Figure 2, a three-dimensional reconstruction process is performed on the two-dimensional image to obtain a reconstruction result, as shown in grid 1 in Figure 2, so that the reconstruction result can describe the state of the face in three-dimensional space.

[0062] Step 12: Retopologically process the reconstruction results according to the target mesh standard to obtain a three-dimensional mesh that conforms to the target mesh standard.

[0063] The target mesh standard refers to the mesh standard required in the actual application scenario, such as the MetaHuman standard; moreover, the target mesh standard is different from the mesh standard used in the 3D reconstruction process in step 11 above, such as the 3DMM standard.

[0064] Retopology processing is used to migrate facial geometry from the reconstructed result to the mesh topology of the target mesh standard; and this disclosure does not limit the implementation of the retopology processing, for example, it can be determined according to the actual application scenario. For example, when the reconstructed result conforms to the 3DMM standard and the target mesh standard is the MetaHuman standard, the retopology processing can be used to migrate facial geometry from 3DMM to the MetaHuman mesh topology.

[0065] Based on the relevant content of steps 11 to 12 above, for some scenarios, such as scenarios with specific grid standards, after obtaining a two-dimensional image, as shown in Figure 2, the two-dimensional image is first subjected to three-dimensional reconstruction processing to obtain the reconstruction result, as shown in grid 1 in Figure 2; then, according to the grid standard required in the scenario, the reconstruction result is subjected to retopology processing to obtain a three-dimensional grid, as shown in grid 2 in Figure 2, so that the three-dimensional grid conforms to the grid standard required in the scenario, thereby making the three-dimensional grid meet the grid requirements of the scenario, which is beneficial to improving the grid construction effect.

[0066] Based on the relevant content of S2 above, after obtaining the two-dimensional image, certain processing can be performed on the two-dimensional image, such as three-dimensional reconstruction and retopology processing as shown in Figure 2, to obtain the three-dimensional mesh corresponding to the two-dimensional image, so that the three-dimensional mesh can describe at least one region of the face in three-dimensional space, thereby enabling the three-dimensional mesh to represent some features of the face.

[0067] S3: Obtain the key point detection results of the two-dimensional image, which include key points in at least one region.

[0068] The keypoint detection result is used to describe the state of at least one region of the two-dimensional image, such as the distribution of facial features; and the keypoint detection result can include keypoints of the at least one region. Specifically, the keypoints of the i-th region are used to describe the state of the i-th region, such as its position in the two-dimensional image; and the keypoints of the i-th region can include one or more keypoints. For example, when the i-th region is an eye, such as the left eye or right eye, the keypoints of the i-th region can include multiple keypoints. Here, i is a positive integer, i≤I, I is a positive integer, and I represents the number of regions in the at least one region.

[0069] Furthermore, this disclosure does not limit the implementation method of the key point detection results mentioned above. For example, it can use any existing or future facial key points, such as the 106 key points shown in Figure 3. Specifically, these 106 key points are:

[0070] 33 key points of the cheek are used to represent uniform sampling points for the outer contour of the cheek, so that the 33 key points can describe the characteristics of the outer contour of the cheek.

[0071] 20 key points for the mouth, so that these 20 key points can describe the characteristics of the mouth;

[0072] The nose has 15 key points, including 3 key points for describing the left side of the bridge of the nose, 3 key points for describing the right side of the bridge of the nose, and 9 key points for describing other features of the nose, so that the 15 key points can describe the characteristics of the nose.

[0073] The eye has 20 key points, including 8 key points describing the outline of the left eye, 8 key points describing the outline of the right eye, 2 key points describing the center point of the left eyeball (such as the pupil), and 2 key points describing the center point of the right eyeball. These 20 key points are used to describe the characteristics of the eye. (It should be noted that in some scenarios, the two key points describing the center point of the left eyeball may overlap, as shown by point 1 in Figure 3, so that point 1 can describe the position of the left pupil; similarly, the two key points describing the center point of the right eyeball may also overlap, as shown by point 2 in Figure 3, so that point 2 can describe the position of the right pupil.)

[0074] The 18 key points of the eyebrow include 9 key points for describing the outline of the left eyebrow and 9 key points for describing the outline of the right eyebrow.

[0075] In reality, in some scenarios, the ears of different types of objects may have different characteristics. Specifically, some objects have pointed ears; some objects have round ears; some objects have long ears; some objects have short ears; and so on. Based on this, in order to better describe facial features, this disclosure also provides a possible implementation of the keypoint detection results mentioned above. In this implementation, the keypoint detection results may include at least 20 keypoints for the left ear and 20 keypoints for the right ear, so that the keypoint detection results can at least describe the characteristics of each ear.

[0076] Among them, 20 key points of the left ear are used to describe the outer contour of the left ear, and this disclosure does not limit the implementation of the 20 key points of the left ear. For example, the 20 key points of the left ear can be used to represent uniform sampling points for the outer contour of the left ear that exists on the face, as shown in Figure 4, which are used to describe the outer contour of the left ear, so that the 20 key points of the left ear can describe the characteristics of the left ear as accurately as possible.

[0077] In addition, 20 key points of the right ear are used to describe the outer contour of the right ear, and this disclosure does not limit the implementation of the 20 key points of the right ear. For example, the 20 key points of the right ear can be used to represent uniform sampling points for the outer contour of the right ear that exists on the face, as shown in FIG4, which are used to describe the outer contour of the right ear, so that the 20 key points of the right ear can describe the characteristics of the right ear as accurately as possible.

[0078] In reality, in some scenarios, there may be certain differences between the irises of different objects, such as different shapes and / or sizes of the irises. Based on this, in order to better describe facial features, this disclosure also provides a possible implementation of the key point detection results mentioned above. In this implementation, the key point detection results may include at least 20 key points of the iris in the left eye and 20 key points of the iris in the right eye, so that the key point detection results can at least describe the characteristics of the iris in each eye.

[0079] The “20 key points of the iris in the left eye” above is used to describe the outline of the iris of the left eye so that the “20 key points of the iris in the left eye” can represent the features of the left eye on the face, such as shape and / or size.

[0080] In addition, the above "20 key points of the iris in the right eye" is used to describe the outline of the iris of the right eye so that the "20 key points of the iris in the right eye" can represent the characteristics of the right eye on the face, such as its shape and / or size.

[0081] In fact, in some scenarios, in order to better describe the characteristics of the eyes, such as shape and / or size, this disclosure also provides a possible implementation of the key point detection results mentioned above. In this way, the key point detection results may include at least 16 key points in the eye socket of the left eye and 16 key points in the eye socket of the right eye, so that the key point detection results can at least more accurately describe the characteristics of the eye socket of each eye.

[0082] The "16 key points of the left eye socket" mentioned above are used to describe the outline of the left eye so that the "16 key points of the left eye socket" can be used to represent uniform sampling points for the outline of the left eye on the face, as shown in Figure 4, which describes the 16 key points for describing the outline of the left eye. This allows the "16 key points of the left eye socket" to describe the characteristics of the left eye in more detail.

[0083] In addition, the "16 key points of the right eye socket" mentioned above are used to describe the outline of the right eye so that the "16 key points of the right eye socket" can be used to represent uniform sampling points for the outline of the right eye on the face, as shown in Figure 4, which describes the 16 key points for describing the outline of the right eye. This allows the "16 key points of the right eye socket" to describe the characteristics of the right eye in more detail.

[0084] In fact, in some scenarios, in order to better describe the characteristics of the nose, such as the contours of the sides of the bridge of the nose, this disclosure also provides a possible implementation of the key point detection results mentioned above. In this way, the key point detection results may include at least four key points on the left side of the nose and four key points on the right side of the nose, so that the key point detection results can at least more accurately describe the characteristics of the sides of the nose.

[0085] The "4 key points on the left side of the nose" mentioned above are used to describe the contour of the left side of the bridge of the nose. These "4 key points on the left side of the nose" represent uniform sampling points for the contour of the left side of the bridge of the nose, as shown in Figure 4. These 4 key points describe the contour of the left side of the bridge of the nose, thus enabling the "4 key points on the left side of the nose" to more accurately describe the characteristics of the left side of the bridge of the nose.

[0086] In addition, the "4 key points on the right side of the nose" mentioned above are used to describe the contour of the right side of the bridge of the nose. These "4 key points on the right side of the nose" represent uniform sampling points for the contour of the right side of the bridge of the nose, as shown in Figure 4. These 4 key points describe the contour of the right side of the bridge of the nose, so that the "4 key points on the right side of the nose" can more accurately describe the characteristics of the right side of the bridge of the nose.

[0087] Based on the above key point detection results, in some scenarios, in order to describe as many facial features as possible with as few key points as possible, this disclosure proposes a new facial key point system based on the 106 key points mentioned above. This new facial key point system can include the 202 key points shown in Figure 4. Compared with the 106 key points, the 202 key points have been improved as follows: ① 20 key points for the left ear, 20 key points for the right ear, and 20 key points for the iris of the left eye have been added. ① Twenty key points of the iris in the right eye, eight key points of the eye socket in the left eye and eight key points of the eye socket in the right eye, one key point on the left side of the nose (point 5 as shown in Figure 4), and one key point on the right side of the nose (point 6 as shown in Figure 4) are used to describe as many facial features as possible; ② One key point used to describe the center point of the left eyeball and one key point used to describe the center point of the right eyeball are deleted, which reduces the number of key points while accurately describing the pupil features.

[0088] Based on the above content and the 202 key points shown in Figure 4, the 202 key points provided in this disclosure for more accurately describing facial features are as follows:

[0089] 33 key points of the cheek are used to represent uniform sampling points for the outer contour of the cheek, so that the 33 key points can describe the characteristics of the outer contour of the cheek.

[0090] 20 key points for the mouth, so that these 20 key points can describe the characteristics of the mouth;

[0091] The nose has 17 key points, including 4 key points for describing the left side of the bridge of the nose, 4 key points for describing the right side of the bridge of the nose, and 9 key points for describing other features of the nose, so that the 17 key points can more accurately describe the characteristics of the nose.

[0092] The eye has 74 key points, including 16 key points for describing the outline of the left eye, 16 key points for describing the outline of the right eye, 20 key points for describing the outline of the iris of the left eye, 20 key points for describing the outline of the iris of the right eye, 1 key point for describing the center point of the eyeball of the left eye (point 3 as shown in Figure 4), and 1 key point for describing the center point of the eyeball of the right eye (point 4 as shown in Figure 4), so that the 74 key points can more accurately describe the characteristics of the eye.

[0093] The 18 key points of the eyebrow, including 9 key points for describing the outline of the left eyebrow and 9 key points for describing the outline of the right eyebrow.

[0094] The 40 key points of the ear, including 20 key points for describing the outer contour of the left ear and 20 key points for describing the outer contour of the right ear.

[0095] Furthermore, this disclosure does not limit the implementation of S3 above. For example, in some scenarios, such as when 106 key points are used to describe facial features, S3 can specifically be: using a pre-built machine learning model with 106 key point detection capabilities to perform key point detection processing on a two-dimensional image, obtaining and outputting the key point detection result of the two-dimensional image, so that the key point detection result can describe the distribution state of the 106 key points in the two-dimensional image. The machine learning model is trained using some sample images and the annotation information of the 106 key points in these sample images.

[0096] It should be noted that this disclosure does not limit the method of acquiring sample images; it can be implemented using existing or future methods capable of acquiring facial images. This disclosure also does not limit the method of acquiring annotation information; for example, it can be implemented using manual annotation.

[0097] For example, in some scenarios, such as when 202 keypoints are used to describe facial features, S3 above can specifically be: using a pre-built machine learning model with 202 keypoint detection capabilities to perform keypoint detection processing on a two-dimensional image, obtaining and outputting the keypoint detection results of the two-dimensional image, so that the keypoint detection results can describe the distribution of the 202 keypoints in the two-dimensional image. Here, the machine learning model is trained using some sample images and the annotation information of the 202 keypoints in these sample images.

[0098] Furthermore, this disclosure does not limit the relationship between the facial keypoints mentioned above and the keypoint detection results mentioned above. For example, when the facial keypoints are implemented using the 202 keypoints shown in Figure 4, if the facial keypoints include 202 keypoint numbers, such as 1 to 202, then the keypoint detection results can include the predicted keypoints corresponding to each keypoint number. For any given keypoint number, the predicted keypoint corresponding to that keypoint number is used to describe the position of that keypoint number in the two-dimensional image. It should be noted that regardless of the pose of the face in the two-dimensional image, the keypoint detection results of the two-dimensional image can include the predicted keypoints corresponding to all keypoint numbers. However, in some poses, such as a large side profile, there may be some incomplete or undisplayed areas, which may cause the predicted keypoints corresponding to some keypoint numbers to be null values ​​or pre-set meaningless values.

[0099] Furthermore, this disclosure does not limit the relationship between the execution time of S3 and the execution time of S2 above. For example, they may be the same. Or, the former may precede the latter. Or, the latter may precede the former.

[0100] Based on the relevant content in S4 above, after acquiring a two-dimensional image, keypoint detection processing can be performed on the two-dimensional image to obtain the keypoint detection results. These keypoint detection results can describe the characteristics of at least one region of the face, such as facial features and cheeks, or some or all of these regions. This allows for subsequent correction processing of at least one region described by the three-dimensional mesh corresponding to the two-dimensional image, ensuring that the characteristics of the corrected region are consistent with those described by the keypoint detection results. This effectively overcomes the inherent defects of the three-dimensional mesh construction scheme, such as poor universality, thereby improving the mesh construction effect.

[0101] S4: Based on the key point detection results, the three-dimensional mesh is deformed to obtain a deformed mesh. The state of at least one region described by the deformed mesh in two-dimensional space is consistent with the state of at least one region described by the key point detection results, or the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in three-dimensional space.

[0102] The deformed mesh refers to the three-dimensional mesh obtained by correcting the key point detection results of the two-dimensional image, such as mesh 3 shown in Figure 2, so that the deformed mesh satisfies the following constraints: the state of at least one region described by the deformed mesh in two-dimensional space is consistent with the state of at least one region described by the key point detection results, or the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in three-dimensional space.

[0103] In fact, since the key point detection results describe the state of the corresponding region by using the key points of each region, in order to improve accuracy, this disclosure also provides a possible implementation of the deformed mesh described above. In this way, when the key point detection results include key points of at least one region of the face, the deformed mesh satisfies the following constraints: for any region, the projection point of the vertex on the deformed mesh used to describe the region in two-dimensional space coincides with the key point of the region, or the projection point of the vertex on the deformed mesh used to describe the region coincides with the projection point of the key point of the region in three-dimensional space.

[0104] As can be seen, in one possible implementation, S4 above can specifically be: based on the key point detection results, the three-dimensional mesh is deformed to obtain a deformed mesh, so that the projection points of the vertices used to describe each region on the deformed mesh in two-dimensional space coincide with the key points of the corresponding regions, or, so that the projection points of the vertices used to describe each region on the deformed mesh coincide with the projection points of the key points of the corresponding regions in three-dimensional space, thereby making the distribution state of facial key points presented by the deformed mesh consistent with the distribution state of facial key points described by the key point detection results, and thus enabling the deformed mesh to better display the facial features presented in the two-dimensional image, such as pointed ears, pointed chins, etc., which is beneficial to improving the mesh construction effect.

[0105] Furthermore, this disclosure does not limit the implementation of S4 above. For example, it can be implemented using a pre-built machine learning model capable of deforming a 3D mesh. This model can be trained based on keypoint detection results of sample images, a 3D mesh constructed from these sample images, and ground truth values ​​of the mesh labeled on these sample images, so that the trained model has good deformation performance. For any sample image, the ground truth values ​​of the mesh labeled on that sample image serve as guidance information for the deformation result of the 3D mesh constructed from that sample image. This deformation result is obtained by the model deforming the corresponding 3D mesh based on the keypoint detection results of the sample image. It should be noted that this disclosure does not limit the method of obtaining the ground truth values ​​of the mesh; for example, it can be implemented using manual annotation.

[0106] Furthermore, the scheme described above may have the following drawbacks: (1) it requires a large amount of resources, such as time and computing resources; (2) training data, such as ground truth values ​​of grids, is difficult to obtain. These drawbacks result in poor universality of the scheme, making it applicable only to a few scenarios.

[0107] Furthermore, to improve universality, this disclosure also provides a possible implementation of S4 above, in which, when the key point detection result of the two-dimensional image above includes N key points, S4 may specifically include steps 21-24 below.

[0108] Step 21: Find the first vertex in the 3D mesh that matches the nth keypoint, where n is a positive integer, n≤N, and N is a positive integer.

[0109] Specifically, for the key point detection results mentioned above, if the key point detection results include N key points, then the nth key point in the key point detection results can be used to represent the position in the two-dimensional image that is identified by the key point number n, so that the nth key point can represent the state of the key point number n in the two-dimensional space.

[0110] In addition, for the nth key point mentioned above, the first vertex that matches the nth key point is used to represent the vertex in the three-dimensional mesh that corresponds to the key point number n, so that the vertex can represent the state of the key point number n in the three-dimensional space.

[0111] Furthermore, this disclosure does not limit the method of determining the first vertex mentioned above. For ease of understanding, the following explanation combines two scenarios.

[0112] Scenario 1: In some scenarios, we can use the relevant information of each vertex in the 3D mesh, such as vertex number, to analyze the vertices that match the key points in the 2D image.

[0113] Based on the above situation 1, this disclosure provides a method for determining the first vertex mentioned above. In this method, when at least one region of the face mentioned above includes a second region, such as the eyes, nose, mouth, ears, etc.; and the key point of the second region includes a second key point, the process of determining the first vertex that matches the second key point may include steps 211-212 below.

[0114] Step 211: Obtain the vertex annotation information of the above 3D mesh. The vertex annotation information is used to describe the mapping relationship between each vertex on the 3D mesh and the key points of at least one region above.

[0115] The second region refers to the area on the face mentioned above where vertex annotation information is needed to find the vertices that match each key point in the 3D mesh, such as the eyes, nose, mouth, and ears.

[0116] Furthermore, this disclosure does not limit the implementation of the second region mentioned above. For example, in some scenarios, such as scenarios where the cheek construction effect is required to be high, the second region may include other regions besides the cheek in at least one of the regions mentioned above, such as the eyes, nose, mouth, ears, etc.

[0117] For example, in some scenarios, such as those requiring high flexibility, the process of determining the second region mentioned above can be as follows: first, obtain the camera parameters of the two-dimensional image; then, based on the camera parameters, determine the rotation angle of the face in the two-dimensional image so that the degree of rotation can represent the deflection of the face relative to the frontal face; then, based on the rotation angle, determine the second region from at least one region of the face so that the second region includes a part of the at least one region, thereby enabling the second region to represent the region in the at least one region where the vertex matching each key point in the three-dimensional mesh needs to be found with the help of vertex annotation information.

[0118] It should be noted that the camera parameters mentioned above are used to describe some characteristics of the two-dimensional image, such as the degree of deflection of the face relative to a frontal view. Furthermore, this disclosure does not limit the scope of these camera parameters; for example, they may include parameters such as the camera's three-dimensional coordinates, rotation angle, and viewing angle. Additionally, this disclosure does not limit the method of obtaining these camera parameters; for example, it can employ any existing or future method capable of obtaining camera parameters for an image, such as a 3DMM-based reconstruction method.

[0119] As can be seen, in one possible implementation, after acquiring a two-dimensional image, a 3DMM-based reconstruction method can be used to perform three-dimensional reconstruction processing on the two-dimensional image to obtain a reconstruction result. This reconstruction result includes at least camera parameters and a mesh under the 3DMM standard, so that the three-dimensional mesh corresponding to the two-dimensional image can be determined based on the mesh, and the mapping relationship between the three-dimensional mesh and the two-dimensional image can be determined based on the camera parameters. This mapping relationship is used to represent the mapping relationship between the three-dimensional coordinates of each vertex in the three-dimensional mesh and the two-dimensional coordinates of each pixel in the two-dimensional image. Thus, this mapping relationship can, to a certain extent, represent the coordinates of the projection points of each vertex in the three-dimensional mesh in two-dimensional space, and the coordinates of the projection points of each pixel in the two-dimensional image in three-dimensional space, thereby enabling the subsequent conversion between three-dimensional space and two-dimensional space based on this mapping relationship.

[0120] Furthermore, this disclosure does not limit the above-described implementation of "determining a second region from at least one region of the face based on the rotation angle." For example, it can specifically be: if the rotation angle is 0, it can be determined that the face in the two-dimensional image is a frontal face. Therefore, in order to improve efficiency, each region in the at least one region can be directly determined as the second region, so that the key points of each region in the at least one region can be determined to match the vertices in the three-dimensional mesh using vertex annotation information. If the rotation angle is greater than 0, it can be determined that the face in the two-dimensional image has undergone a certain degree of deflection, so that the cheeks in the two-dimensional image have a certain degree of deformation. Therefore, in order to avoid interference caused by such deformation, other regions in the at least one region besides the cheeks can be determined as the second region, so that the key points of the other regions can be determined to match the vertices in the three-dimensional mesh using vertex annotation information.

[0121] Furthermore, in some scenarios, such as those with significant facial distortion, a 2D image may only display a single complete eye, a single complete eyebrow, or a single complete ear. Therefore, to improve the mesh construction effect, this disclosure also provides a possible implementation of the second region mentioned above. In this implementation, the second region can satisfy the following constraint: the difference between the distribution state of the key points in the second region and the corresponding standard distribution state is lower than a preset difference. It should be noted that this preset difference can be set according to the actual application scenario.

[0122] The standard distribution state corresponding to the second region is used to represent the distribution state of the key points in the second region under the frontal view, so that the difference between the distribution state of the key points in the second region and the standard distribution state corresponding to the second region can represent the degree of deformation of the second region in the two-dimensional image relative to the corresponding region in the frontal view. Furthermore, this disclosure does not limit the method of obtaining the standard distribution state; for example, it can be implemented by manual annotation. Alternatively, to reduce the difficulty of data acquisition, the standard distribution state can be determined based on the key point detection results of at least one frontal view image.

[0123] Based on the above two paragraphs, it can be seen that in one possible implementation, when at least one region mentioned above includes I regions, the process of determining the second region mentioned above may include at least the following: if the rotation angle of the face in the two-dimensional image is greater than 0, then first determine the distribution state of the key points in the i-th region; then calculate the difference between the distribution state of the key points in the i-th region and the standard distribution state corresponding to the i-th region; then, determine whether the difference is lower than a preset difference. If so, then determine the i-th region as the second region, so that the vertex matching the key points of the i-th region in the three-dimensional mesh can be determined by using vertex annotation information. i is a positive integer, i≤I, and I is a positive integer.

[0124] As can be seen, in one possible implementation, this disclosure can use vertex annotation information to determine the vertices in a 3D mesh that match the key points of some regions in a complete state. It should be noted that for regions in an incomplete state or regions not appearing in the 2D image, it is not necessary to search for the vertices in the 3D mesh that match the key points of these regions; instead, the deformation information corresponding to these regions can be predicted subsequently through symmetry. Alternatively, the vertices in the 3D mesh that match the key points of these regions can be inferred through a certain method. The specific implementation process can be selected based on the actual application scenario.

[0125] The second key point refers to any key point in the second region, so that the second key point can be used to represent certain characteristics of the second region. For example, if the second region is the left ear, then the second region has 20 key points, and the second key point can refer to any one of the 20 key points.

[0126] Vertex annotation information of a 3D mesh refers to information pre-annotated for each vertex in the 3D mesh, such as vertex number and the key point number corresponding to that vertex. This vertex annotation information can represent the mapping relationship between each vertex on the 3D mesh and the key point numbers of the face, such as 202 key point numbers. Thus, the vertex annotation information can represent the mapping relationship between each vertex on the 3D mesh and the key points of at least one region mentioned above, such as which vertex corresponds to which key point.

[0127] Furthermore, this disclosure does not limit the method of obtaining the vertex annotation information mentioned above.

[0128] Furthermore, this disclosure does not limit the execution time of step 211 above, as long as it is ensured that the execution time of step 211 is earlier than the execution time of step 212 below.

[0129] Step 212: Based on the vertex annotation information above, determine the first vertex that matches the second key point. The mapping relationship above includes the correspondence between the "first vertex that matches the second key point" and the second key point.

[0130] It should be noted that this disclosure does not limit the implementation of step 212 above.

[0131] Based on the relevant content of steps 211 to 212 above, it can be seen that for any key point in the second key point of a two-dimensional image, such as the eyes, nose, mouth, ears, etc., a first vertex matching the second key point can be queried from the vertex annotation information of the three-dimensional mesh, so that the first vertex can represent the initial position of the second key point in the three-dimensional space, so that the deformation processing of the three-dimensional mesh can be completed based on the distance between the initial position and the projection position of the second key point in the three-dimensional space.

[0132] Based on the relevant content of the second key point above, in some scenarios, when the face in the 2D image is a frontal view, the vertex annotation information of the 3D mesh can be used to determine the first vertex that matches each key point in the 2D image, so that subsequent deformation processing of the 3D mesh can be completed based on these vertices and their corresponding key points. When the face in the 2D image has a small degree of deflection, the vertex annotation information can be used to determine the first vertex that matches any key point in the 2D image other than the key point of the cheek, so that subsequent deformation processing of the 3D mesh can be completed based on these vertices and their corresponding key points. When the face in the 2D image has a large degree of deflection, the vertex annotation information can be used to determine the first vertex that matches any key point in the region of the 2D image that is in a complete state, so that subsequent deformation processing of the 3D mesh can be completed based on these vertices and their corresponding key points.

[0133] Scenario 2: In some scenarios, such as non-frontal scenes, the nearest distance method can be used to find the region of vertices that match some key points in the 3D mesh.

[0134] Based on situation 2 above, this disclosure also provides a method for determining the first vertex above. In this method, when at least one region of the face above includes a first region, such as a cheek, and the key point of the first region includes a first key point, the process of determining the first vertex that matches the first key point may include steps 213-214 below.

[0135] Step 213: Obtain the projection result of the above three-dimensional mesh onto the two-dimensional image. The projection result includes the projection points of each vertex on the three-dimensional mesh in the two-dimensional space.

[0136] The first region refers to the area on the face mentioned above where the nearest neighbor method is used to find the vertices that match each key point in the 3D mesh, such as the cheek region.

[0137] Furthermore, this disclosure does not limit the implementation of the first region mentioned above. For example, in some scenarios, such as scenarios where the cheek construction effect is required to be high, the first region may include the cheek, so that the vertex matching each key point in the region in the three-dimensional mesh can be found in the subsequent process by means of the nearest distance method, as shown in steps 213-214.

[0138] For example, in some scenarios, such as those requiring high flexibility, the process of determining the first region described above can be as follows: First, obtain the camera parameters of the 2D image; then, based on the camera parameters, determine the rotation angle of the face in the 2D image; then, based on the rotation angle, determine the first region from at least one region of the face, so that the first region includes part or all of the at least one region, thereby enabling the first region to represent the region in the at least one region where the vertices that match each keypoint in the 3D mesh need to be found using the nearest distance method. For details regarding the camera parameters, please refer to the above.

[0139] Furthermore, this disclosure does not limit the above-described implementation of "determining a first region from at least one region of the face based on the rotation angle". For example, it can specifically be: if the rotation angle is greater than 0, it can be determined that the face in the two-dimensional image has undergone a certain degree of deflection, thereby determining that the cheek region is suitable for determining its matching vertex by means of the nearest distance method, so the cheek can be determined as the first region.

[0140] The first keypoint refers to any keypoint in a first region, such that the first keypoint is used to represent certain characteristics of the first region. For example, if the first region is a cheek, then the first region has 33 keypoints, and the first keypoint can refer to any one of those 33 keypoints.

[0141] The projection result refers to the result obtained by projecting each vertex on the three-dimensional mesh onto a two-dimensional image, so that the projection result includes the projection points of each vertex on the three-dimensional mesh in the two-dimensional space, so that the projection points can represent the state of the corresponding vertex in the two-dimensional space, such as position.

[0142] Furthermore, this disclosure does not limit the implementation of step 213 above. For example, it can be implemented using any existing or future method capable of projecting a three-dimensional mesh onto a two-dimensional image.

[0143] For example, in some scenarios, step 213 above can specifically be: based on the camera parameters of the 2D image, project each vertex on the 3D mesh onto the 2D image to obtain a projection result, so that the projection result includes the projection points of each vertex on the 3D mesh in 2D space, thereby enabling the projection result to represent, to a certain extent, the mapping relationship between the 3D coordinates of each vertex in the 3D mesh and the 2D coordinates of each pixel in the 2D image. For details regarding the camera parameters, please refer to the above text.

[0144] Based on the relevant content of step 213 above, after obtaining the three-dimensional mesh corresponding to the two-dimensional image above, the vertices of the three-dimensional mesh can be projected onto the two-dimensional image according to the camera parameters of the two-dimensional image to obtain the projection result. This projection result includes the projection points of each vertex of the three-dimensional mesh in the two-dimensional space, so that these projection points can be used to determine which vertices in the three-dimensional mesh match the key points of the first region.

[0145] Step 214: Based on the distance between the first key point and the projection points of each vertex in the two-dimensional space, determine the first vertex that matches the first key point; the projection result includes the target projection point, which is the projection point of the "first vertex that matches the first key point" in the two-dimensional space. The distance between the first key point and the target projection point is not greater than the distance between the first key point and other projection points in the projection result except for the target projection point.

[0146] The target projection point refers to the projection point in the above projection result that is closest to the first key point, so that the target projection point can be used to represent the projection point of the "first vertex matching the first key point" in the two-dimensional space, thereby making the target projection point satisfy the following constraint: the distance between the first key point and the target projection point is not greater than the distance between the first key point and other projection points in the projection result other than the target projection point.

[0147] Furthermore, this disclosure does not limit the implementation of step 214 above. For example, it can specifically be: first calculate the distance between the first key point and the projection points of each vertex in the two-dimensional space to obtain the distance corresponding to each vertex; then compare the distances corresponding to these vertices to obtain the vertex corresponding to the minimum distance, which is used as the first vertex that matches the first key point.

[0148] In practice, the projection points obtained by projecting a 3D mesh usually form a cheek edge, so that the projection points of vertices that match the key points of the cheek are usually located on the cheek edge. Therefore, in order to improve efficiency, this disclosure also provides a possible implementation of step 214 above. In this way, when the first region above is the cheek, step 214 can specifically be: firstly, determine the projection point set from the above projection results so that the projection point set is used to represent the cheek edge formed by the projection points of some vertices of the 3D mesh in two-dimensional space; then, based on the distance between the first key point and each projection point in the projection point set, determine the first vertex that matches the first key point so that the projection point set includes the target projection point above, and make the distance between the first key point and the target projection point not greater than the distance between the first key point and other projection points in the projection point set besides the target projection point.

[0149] As can be seen, in one possible implementation, step 214 above can specifically be as follows: First, determine the projection point set from the projection results above, so that the projection point set can represent the cheek edge formed by the projection points of some vertices on the three-dimensional mesh in two-dimensional space; then calculate the distance between the first key point and each projection point in the projection point set to obtain the distance corresponding to each projection point; then compare the distances corresponding to these projection points to obtain the projection point corresponding to the minimum distance, which is taken as the target projection point; then, based on the target projection point, determine the first vertex that matches the first key point, so that the target projection point is used to represent the projection point of the "first vertex that matches the first key point" in two-dimensional space.

[0150] Based on the relevant content of steps 213 to 214 above, for a first key point in a two-dimensional image, such as any key point on the cheek, the vertex corresponding to the minimum distance between the first key point and the projection points of each vertex in the three-dimensional mesh in the two-dimensional space can be selected as the first vertex to match the first key point. This first vertex can represent the initial position of the first key point in the three-dimensional space, so that the deformation processing of the three-dimensional mesh can be completed based on the distance between the initial position and the projection position of the first key point in the three-dimensional space.

[0151] In addition, in some scenarios, such as when the face is significantly distorted, the process shown in steps 213-214 can be used to determine the vertices in the 3D mesh that match the key points of some areas other than the cheeks, such as incomplete eyes, incomplete ears, and the left ear that is not present in the image.

[0152] Furthermore, regarding the face, due to its symmetry, certain regions within the face also exhibit symmetrical relationships. For example, there is symmetry between the left and right eyes, the left and right eyebrows, the left and right ears, the left and right sides of the nose, and the left and right halves of the mouth. This results in a high degree of similarity between symmetrical regions. Therefore, to improve accuracy, we can use the vertices of a complete region to predict the vertices of another region that is incomplete or not displayed. Additionally, the complete region mentioned earlier usually belongs to the second region mentioned above. Therefore, based on the keypoints of the second region, such as the vertices matched by the second keypoints mentioned above, we can determine the vertices matched by the keypoints of the symmetrical region of that second region.

[0153] As can be seen, this disclosure also provides a method for determining the first vertex. This method is more suitable for regions that are not in a complete state, such as incomplete eyes, incomplete ears, or a left ear that is not present in the image. In this method, when at least one region of the face in the aforementioned two-dimensional image includes the aforementioned second region, the key point of the second region includes the second key point, the symmetrical region of the second region is the third region, and the key point of the third region includes the third key point, and the third key point has a symmetrical relationship with the second key point, if the difference between the distribution state of the key points of the third region and the standard distribution state corresponding to the third region is not less than a preset difference, it can be determined that the third region presented in the two-dimensional image has undergone a relatively large deformation, or that the third region does not exist in the two-dimensional image. Therefore, in order to improve accuracy, a vertex that has a symmetrical relationship with the aforementioned "first vertex matching the second key point" can be found in the three-dimensional mesh as the first vertex matching the third key point. This can better determine the relevant vertices of regions such as incomplete eyes, incomplete ears, and a left ear that is not present in the image. Here, the third region refers to the symmetrical region of the second region; the third key point is used to represent any key point of the third region.

[0154] Based on the relevant content of step 21 above, after determining each key point from the two-dimensional image, a first vertex matching the nth key point can be found in the three-dimensional grid of the two-dimensional image using a certain method. This first vertex can represent the initial position of the corresponding key point in the three-dimensional space, so that the deformation processing of the three-dimensional grid can be completed based on the distance between the initial position and the projection position of the corresponding key point in the three-dimensional space.

[0155] Step 22: Based on the distance between the projection point of the nth key point in 3D space and the "first vertex matching the nth key point" mentioned above, determine the deformation parameters of the first vertex, where n is a positive integer, n≤N, and N is a positive integer.

[0156] The projection point of the nth key point in three-dimensional space refers to the projection of the nth key point onto the three-dimensional space of the three-dimensional mesh, so that the projection point can represent the position of the nth key point in the three-dimensional space.

[0157] Furthermore, this disclosure does not limit the method of obtaining the "projection point of the nth key point in three-dimensional space" mentioned above. For example, it can specifically be: according to the camera parameters of the two-dimensional image, the two-dimensional image is back-projected to three-dimensional space to obtain the projection points of each key point in the two-dimensional image in the three-dimensional space.

[0158] Furthermore, this disclosure does not limit the implementation of step 22 above. For example, it can specifically be: determining the distance between the projection point of the nth key point in three-dimensional space and the first vertex as the deformation parameter of the first vertex, so that the deformation parameter can represent the position offset of the first vertex.

[0159] Step 23: Determine the deformation parameters of the above 3D mesh based on the deformation parameters of the first vertex that matches each key point. The deformation parameters of the 3D mesh include the deformation parameters of the first vertex.

[0160] It should be noted that this disclosure does not limit the implementation of step 23 above. For example, it can specifically be: determining the deformation parameters of the three-dimensional mesh above based on the deformation parameters of the first vertices that match each key point, so that the deformation parameters of the three-dimensional mesh include the deformation parameters of these first vertices, so that subsequent deformation processing of the three-dimensional mesh can be completed based on these deformation parameters.

[0161] For example, in some scenarios, such as when the relevant vertices of regions such as incomplete eyes, incomplete ears, or left ears that are not present in the image are not found, this disclosure also provides a possible implementation of step 23 above. In this implementation, when at least one region of the face in the above two-dimensional image includes the above second region, the key point of the second region includes the second key point, and there is at least one vertex in the three-dimensional mesh for describing the symmetrical region of the second region, the at least one vertex includes the second vertex, and there is a symmetrical relationship between the second vertex and the above "first vertex that matches the second key point", step 23 may include at least steps 231-232 below.

[0162] Step 231: If the difference between the distribution state of the key points in the symmetrical region mentioned above and the standard distribution state corresponding to the symmetrical region is not lower than the preset difference, then predict the deformation parameters of the second vertex based on the deformation parameters of the "first vertex matching the second key point" mentioned above.

[0163] Here, the second vertex refers to a vertex in the 3D mesh that matches a key point in the symmetrical region mentioned above; and there is a symmetrical relationship between the second vertex and the "first vertex that matches the second key point" mentioned above.

[0164] Furthermore, the deformation parameters of the second vertex describe what kind of adjustment is made to the second vertex, such as by what positional offset; and this disclosure does not limit the method of obtaining the deformation parameters of the second vertex. For example, it can specifically be: calculating the deformation parameters of the second vertex based on the deformation parameters of the "first vertex matching the second key point" mentioned above, and the symmetry parameters between the "first vertex matching the second key point" and the second vertex, so that the two vertices obtained after deformation according to the corresponding parameters still satisfy the symmetry relationship described by the symmetry parameters. The symmetry parameters describe the characteristics of the symmetry relationship between the two vertices; and this disclosure does not limit the symmetry parameters.

[0165] Step 232: Based on the deformation parameters of the second vertex and the deformation parameters of the first vertex that matches the second key point mentioned above, determine the deformation parameters of the three-dimensional mesh so that the deformation parameters of the three-dimensional mesh include the deformation parameters of the second vertex and the deformation parameters of the first vertex that matches the second key point.

[0166] Based on the relevant content of steps 231 to 232 above, it can be seen that in some scenarios, this disclosure can first determine the deformation parameters of the relevant vertices of some regions in a complete state, and then predict the deformation parameters of the relevant vertices of the symmetrical region based on symmetry. This can effectively overcome the deformation problem caused by the defects of the symmetrical region itself, thereby improving accuracy.

[0167] In fact, for a 3D mesh, there are not only vertices matching each key point, but also other vertices. Therefore, to improve accuracy, interpolation can be used to determine the deformation parameters of these vertices. Thus, in one possible implementation, when the third vertex is a vertex other than the first and second vertices in the 3D mesh, the deformation parameters of the 3D mesh may also include the deformation parameters of the third vertex. The deformation parameters of the third vertex are obtained by interpolation based on the deformation parameters of the first and second vertices. It should be noted that this disclosure does not limit the implementation of this interpolation process; for example, it can be implemented using radial basis function (RBF) interpolation.

[0168] Based on the relevant content of step 23 above, it can be seen that for the three-dimensional mesh corresponding to the two-dimensional image, the deformation parameters of different types of vertices in the three-dimensional mesh can be determined in different ways. For example, the deformation parameter of the first vertex is determined based on the distance between the projection point of the corresponding key point in the three-dimensional space and the first vertex; the deformation parameter of the second vertex is predicted based on the deformation parameter of the symmetrical vertex of the second vertex; and the deformation parameter of the third vertex is obtained by interpolating the deformation parameters of other vertices.

[0169] Step 24: According to the deformation parameters of the three-dimensional mesh above, perform deformation processing on the three-dimensional mesh to obtain the deformed mesh.

[0170] It should be noted that this disclosure does not limit the implementation of step 24 above.

[0171] Based on the relevant content of steps 21 to 24 above, in some scenarios, each vertex in the 3D mesh can be projected onto the 2D image to obtain the projection points of each vertex in the 2D space. Then, for any key point in the areas such as eyes, nose, mouth, and ears, the vertex matching the key point is determined according to the predefined vertex number. For any key point on the cheek, the projection point closest to the key point is found from the cheek edge formed by these projection points, and the vertex matching the key point is determined based on the closest projection point. Next, the 2D image is back-projected onto the 3D space to obtain the projection points of each key point in the 3D space. Finally, based on the transformation relationship between the vertex matching each key point and the corresponding projection point of the key point in the 3D space, the corresponding vertices in the 3D mesh are deformed. The RBF method can be used to complete the deformation processing of other vertices in the 3D mesh to obtain the deformed network, so that the deformed network can better represent the facial features presented in the 2D image, such as the features of the facial features and cheeks. In particular, since the key points of the cheeks are greatly affected by the rotation angle of the face, mapping the key points of the cheeks to fixed points of the 3D mesh in other facial poses besides the frontal view will result in poor performance. Therefore, this disclosure overcomes the problem caused by the aforementioned fixed points by using the closest distance method, thereby improving the mesh construction effect.

[0172] Based on the relevant content in S1 to S4 above, the three-dimensional mesh determination method provided in this disclosure first acquires a two-dimensional image so that the two-dimensional image can describe at least one region of the face in two-dimensional space, such as facial features and cheeks; then, a three-dimensional mesh corresponding to the two-dimensional image is constructed so that the three-dimensional mesh can describe the at least one region in three-dimensional space, and the key point detection results of the two-dimensional image are acquired so that the key point detection results include key points of at least one region, thereby allowing the key point detection results to describe the characteristics of these regions, such as shape or size; then, based on the key point detection results, the three-dimensional mesh is deformed to obtain a deformed mesh, so that the state of at least one region described by the deformed mesh in two-dimensional space is consistent with the state of at least one region described by the key point detection results, or so that the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in three-dimensional space, thereby enabling the deformed mesh to more accurately describe the characteristics of at least one region presented in the two-dimensional image, which is beneficial to improving the mesh construction effect.

[0173] In addition, in some scenarios, this disclosure also provides a process for constructing a three-dimensional face model, which may specifically include steps S1-S4 above and steps 31-32 below.

[0174] Step 31: Obtain the texture map corresponding to the 2D image under the deformed mesh.

[0175] It should be noted that this disclosure does not limit the implementation of step 31 above. For example, it can specifically be: based on the camera parameters of the two-dimensional image, construct a mapping relationship between the deformed mesh and the two-dimensional image, so that the mapping relationship can represent the mapping relationship between the three-dimensional coordinates of each vertex in the deformed mesh and the two-dimensional coordinates of each pixel in the two-dimensional image; then, based on the mapping relationship, unfold the two-dimensional image to the two-dimensional texture coordinate system corresponding to the deformed mesh, such as UV space, to obtain a texture map. The two-dimensional texture coordinate system is used to define how to place the texture map on the three-dimensional surface of the deformed mesh.

[0176] Step 32: Based on the optimization results of the texture mapping, render the deformed mesh to obtain a 3D face model; the optimization results are determined based on at least one pre-built texture optimization model, and different texture optimization models have different optimization functions.

[0177] The optimization of the texture map involves processing it by removing highlights, shadows, and occlusions to better represent the facial features depicted in the 2D image. It's important to note that occlusions refer to objects that obscure the face, such as hair, glasses, or forehead ornaments.

[0178] Furthermore, this disclosure does not limit the method of obtaining the above-mentioned optimization results. For example, it can be implemented by using a pre-built machine learning model with texture optimization function.

[0179] In addition, to further improve the optimization effect, this disclosure also provides an optimization method for the texture mapping mentioned above, which may specifically include steps a to c below.

[0180] Step a: Build at least one texture optimization model so that different texture optimization models have different optimization functions.

[0181] The texture optimization model refers to a machine learning model with certain texture optimization capabilities obtained through training. This disclosure does not limit the texture optimization model; for example, it can be implemented using Pix2PixHD or any diffusion model, such as the ResShift model. Pix2PixHD is an open-source model capable of image synthesis and processing based on generative adversarial networks.

[0182] Furthermore, this disclosure does not limit the construction process of each texture optimization model. For example, it may specifically be as follows: first, obtain the texture ground truth; then, superimpose some distortion information, such as certain lighting, occlusion edges, occlusion information, etc., onto the texture ground truth to obtain a sample texture; then, use the texture optimization model to optimize the sample texture to obtain the optimized texture output by the texture optimization model; secondly, based on the difference between the optimized texture and the texture ground truth, update the texture optimization model, and return to continue executing the aforementioned "obtaining texture ground truth" step and its subsequent steps until the preset stopping condition is met, thus ending the training process for the texture optimization model.

[0183] It should be noted that the aforementioned preset stopping conditions can be pre-set. For example, they may include: the model loss of the texture optimization model is lower than a preset loss threshold, the rate of change of the model loss of the texture optimization model is less than a preset rate of change threshold, or the number of updates of the texture optimization model reaches a preset number threshold. The model loss of the texture optimization model is determined based on the difference between the optimized texture and the ground truth texture; moreover, this disclosure does not limit the method of calculating the model loss.

[0184] In addition, in order to ensure that the optimization functions of different texture optimization models are different, this disclosure also provides a possible implementation of at least one texture optimization model mentioned above, in which the at least one texture optimization model includes at least one first model and / or at least one second model.

[0185] For at least one first model mentioned above, the model structures of different first models are the same, but the training data of different first models are different. For example, when the at least one first model includes I first models, the sample images in the training data of the first first model are obtained by superimposing light and shadow and occlusion edges onto the texture ground value; the sample images in the training data of the second first model are obtained by first superimposing light and shadow and occlusion edges onto the texture ground value to obtain the superimposed image; and then cropping the superimposed image according to the skin mask of the superimposed image; the sample images in the training data of the third first model are obtained by superimposing light and shadow and at least one occlusion onto the texture ground value; and so on.

[0186] It should be noted that the skin mask mentioned above is obtained using a face segmentation model so that it can represent the location of the unmasked areas of the face. The edges of the occlusions are caused by inaccurate face segmentation, resulting in the edges of occlusions such as hair interfering with the face.

[0187] It should also be noted that this disclosure does not limit the model structure of each first model. For example, they can be implemented using the same diffusion model, such as the ResShift model.

[0188] For at least one second model described above, the model structure of each second model differs from the model structure of each first model described above. Different second models may have the same model structure, but their training data differs. It should be noted that this disclosure does not limit the model structure of the second model; for example, it can be implemented using Pix2PixHD. Furthermore, the training data for different second models can be implemented using methods similar to those used for the training data of the different first models described above.

[0189] Step b: Optimize the texture maps using each texture optimization model to obtain the optimized texture maps corresponding to each texture optimization model.

[0190] The optimized texture corresponding to the k-th texture optimization model is obtained by optimizing the texture above using the k-th texture optimization model, so that the quality of the optimized texture is higher than that of the texture. k is a positive integer, k≤K, K is a positive integer, and K represents the number of models in at least one texture optimization model above.

[0191] Step c: Select the highest quality texture from the optimized textures corresponding to at least one texture optimization model, and determine it as the optimized texture processing result.

[0192] It should be noted that this disclosure does not limit the method of evaluating the quality of the above images; it can be implemented using any existing or future method capable of evaluating the quality of an image.

[0193] Based on the relevant content of steps a to c above, in some scenarios, after obtaining the texture map, multiple texture optimization models can be used to optimize the texture map first; then, the best quality optimized image can be selected as the optimization result of the texture map, so that the optimization result can better present the facial features described by the two-dimensional image, so that the optimization result can be used to render the deformed mesh to obtain a three-dimensional facial model, so that the three-dimensional facial model can better present the facial features, which is beneficial to improving the construction effect of the three-dimensional facial model.

[0194] Based on the relevant content of steps 31 to 32 above, for some scenarios, after obtaining the two-dimensional image provided by the user, the two-dimensional image is first reconstructed using a 3DMM-based reconstruction method to obtain the reconstruction result. Then, the reconstruction result is retopologically processed to transfer the facial shape from 3DMM to the MetaHuman mesh topology, obtaining the three-dimensional mesh corresponding to the two-dimensional image. Next, the key point detection results of the two-dimensional image are used to deform the three-dimensional mesh to obtain a deformed mesh, so that the facial features described by the deformed mesh are consistent with the facial features described by the key point detection results. Next, the texture map of the two-dimensional image under the deformed mesh is obtained. Finally, based on the optimization results of the texture map, the deformed mesh is rendered to obtain a three-dimensional facial model, so that the three-dimensional facial model can better present the facial features, which is beneficial to improving the construction effect of the three-dimensional facial model.

[0195] In addition, in some scenarios, in order to improve the performance of the 3D face model, after obtaining the 3D face model, an automated binding process can be carried out using certain methods, such as Maya, to bind each vertex in the 3D face model with the corresponding expression, so that the 3D face model can be driven to realize any expression.

[0196] Based on the three-dimensional mesh determination method provided in this disclosure, this disclosure also provides a three-dimensional mesh determination device, which will be explained and described below with reference to FIG5. FIG5 is a schematic diagram of the structure of a three-dimensional mesh determination device provided in this disclosure. It should be noted that for technical details of the three-dimensional mesh determination device provided in this disclosure, please refer to the relevant content of the three-dimensional mesh determination method above.

[0197] As shown in Figure 5, the three-dimensional mesh determination device 500 provided in this embodiment includes:

[0198] The acquisition unit 501 is used to acquire a two-dimensional image, which is used to describe at least one region of the face in a two-dimensional space;

[0199] The construction unit 502 is used to construct a three-dimensional mesh corresponding to the two-dimensional image and to obtain the key point detection results of the two-dimensional image. The three-dimensional mesh is used to describe the at least one region in three-dimensional space, and the key point detection results include the key points of the at least one region.

[0200] The deformation unit 503 is used to deform the three-dimensional mesh according to the key point detection results to obtain a deformed mesh. The state of at least one region described by the deformed mesh in the two-dimensional space is consistent with the state of at least one region described by the key point detection results, or the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in the three-dimensional space.

[0201] In one possible implementation, for any of the regions, the projection points of the vertices on the deformed mesh used to describe the region in the two-dimensional space coincide with the key points of the region; or, the projection points of the vertices on the deformed mesh used to describe the region coincide with the projection points of the key points of the region in the three-dimensional space.

[0202] In one possible implementation, the deformation unit 503 is specifically configured to: for any key point, find a first vertex matching the key point in the three-dimensional mesh, and determine the deformation parameters of the first vertex based on the distance between the projection point of the key point in the three-dimensional space and the first vertex; determine the deformation parameters of the three-dimensional mesh based on the deformation parameters of the first vertex, wherein the deformation parameters of the three-dimensional mesh include the deformation parameters of the first vertex; and perform deformation processing on the three-dimensional mesh according to the deformation parameters of the three-dimensional mesh to obtain the deformed mesh.

[0203] In one possible implementation, the at least one region includes a first region, and the key points of the first region include a first key point;

[0204] The process of determining the first vertex matching the first key point includes: obtaining the projection result of the three-dimensional mesh onto the two-dimensional image, the projection result including the projection points of each vertex on the three-dimensional mesh in the two-dimensional space; determining the first vertex matching the first key point based on the distance between the first key point and the projection points of each vertex in the two-dimensional space; the projection result includes a target projection point, the target projection point being the projection point of the first vertex matching the first key point in the two-dimensional space, and the distance between the first key point and the target projection point being no greater than the distance between the first key point and other projection points in the projection result besides the target projection point.

[0205] In one possible implementation, the first region is a cheek; the process of determining the first vertex matching the first key point includes at least: determining a set of projection points from the projection results, the set of projection points representing the cheek edge formed by the projection points of some vertices on the three-dimensional mesh in the two-dimensional space; determining the first vertex matching the first key point based on the distance between the first key point and each projection point in the set of projection points, the set of projection points including the target projection point, the distance between the first key point and the target projection point being no greater than the distance between the first key point and other projection points in the set of projection points besides the target projection point.

[0206] In one possible implementation, the at least one region includes a second region, and the key points of the second region include second key points;

[0207] The process of determining the first vertex that matches the second key point includes: obtaining vertex annotation information of the three-dimensional mesh, wherein the vertex annotation information is used to describe the mapping relationship between each vertex on the three-dimensional mesh and the key point of the at least one region; and determining the first vertex that matches the second key point based on the vertex annotation information, wherein the mapping relationship includes the correspondence between the first vertex that matches the second key point and the second key point.

[0208] In one possible implementation, the difference between the distribution state of key points in the second region and the standard distribution state corresponding to the second region is lower than a preset difference, wherein the standard distribution state is determined based on the key point detection results of at least one frontal face image.

[0209] In one possible implementation, the three-dimensional mesh contains at least one vertex for describing a symmetrical region of the second region; the at least one vertex includes a second vertex, and there is a symmetrical relationship between the second vertex and the first vertex that matches the second keypoint;

[0210] The process of determining the deformation parameters of the three-dimensional mesh includes: if the difference between the distribution state of the key points in the symmetrical region and the standard distribution state corresponding to the symmetrical region is not less than a preset difference, then the deformation parameters of the second vertex are predicted based on the deformation parameters of the first vertex that matches the second key point; and the deformation parameters of the three-dimensional mesh are determined based on the deformation parameters of the second vertex and the deformation parameters of the first vertex that matches the second key point.

[0211] In one possible implementation, the third vertex is any vertex in the three-dimensional mesh other than the first vertex and the second vertex; the deformation parameters of the three-dimensional mesh also include the deformation parameters of the third vertex, which are obtained by interpolation based on the deformation parameters of the first vertex and the deformation parameters of the second vertex.

[0212] In one possible implementation, the three-dimensional mesh determination device 500 further includes:

[0213] A segmentation unit is used to acquire camera parameters of the two-dimensional image; determine the rotation angle of the face based on the camera parameters; if the rotation angle is 0, then each region in the at least one region is defined as a second region; if the rotation angle is greater than 0, then the cheek in the at least one region is defined as a first region, and the other regions in the at least one region besides the cheek are defined as second regions.

[0214] In one possible implementation, the at least one region includes part or all of the cheek, eyes, nose, mouth, and ears.

[0215] In one possible implementation, the key point detection results include at least 20 key points of the left ear and 20 key points of the right ear; the 20 key points of the left ear are used to describe the outer contour of the left ear; and the 20 key points of the right ear are used to describe the outer contour of the right ear.

[0216] And / or,

[0217] The key point detection results include at least 20 key points of the iris in the left eye and 20 key points of the iris in the right eye; the 20 key points of the iris in the left eye are used to describe the iris contour of the left eye; the 20 key points of the iris in the right eye are used to describe the iris contour of the right eye.

[0218] In one possible implementation, the key point detection results further include 16 key points in the eye socket of the left eye, 16 key points in the eye socket of the right eye, 4 key points on the left side of the nose, and 4 key points on the right side of the nose; the 16 key points in the eye socket of the left eye are used to describe the outline of the left eye; the 16 key points in the eye socket of the right eye are used to describe the outline of the right eye; the 4 key points on the left side of the nose are used to describe the outline of the left side of the bridge of the nose; and the 4 key points on the right side of the nose are used to describe the outline of the right side of the bridge of the nose.

[0219] In one possible implementation, the construction process of the three-dimensional mesh includes: performing three-dimensional reconstruction processing on the two-dimensional image to obtain a reconstruction result; and performing retopology processing on the reconstruction result according to a target mesh standard to obtain the three-dimensional mesh, wherein the three-dimensional mesh conforms to the target mesh standard.

[0220] In one possible implementation, the three-dimensional mesh determination device 500 further includes:

[0221] The processing unit is used to obtain the texture map corresponding to the two-dimensional image under the deformed mesh;

[0222] The rendering unit is used to render the deformed mesh based on the optimization results of the texture map to obtain a three-dimensional face model; the optimization results are determined based on at least one pre-built texture optimization model, and different texture optimization models have different optimization functions.

[0223] Based on the aforementioned content of the three-dimensional mesh determination device 500, the working principle of the three-dimensional mesh determination device 500 provided in this disclosure is as follows: First, a two-dimensional image is acquired so that the two-dimensional image can describe at least one region of the face in two-dimensional space, such as facial features and cheeks; then, a three-dimensional mesh corresponding to the two-dimensional image is constructed so that the three-dimensional mesh can describe the at least one region in three-dimensional space, and the key point detection results of the two-dimensional image are acquired so that the key point detection results include key points of at least one region, thereby allowing the key point detection results to describe the characteristics of these regions, such as shape or size; then, based on the key point detection results, the three-dimensional mesh is deformed to obtain a deformed mesh, so that the state of at least one region described by the deformed mesh in two-dimensional space is consistent with the state of at least one region described by the key point detection results, or so that the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in three-dimensional space, thereby enabling the deformed mesh to more accurately describe the characteristics of at least one region presented in the two-dimensional image, which is beneficial to improving the mesh construction effect.

[0224] In addition, this disclosure also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device performs any implementation of the three-dimensional mesh determination method provided in this disclosure.

[0225] Referring to Figure 6, it illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 6 is merely an example and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.

[0226] As shown in Figure 6, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0227] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 6 illustrates electronic device 600 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0228] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0229] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0230] This disclosure also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the three-dimensional mesh determination method provided in this disclosure.

[0231] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0232] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0233] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0234] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0235] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0236] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0237] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0238] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0239] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0240] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0241] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0242] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0243] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0244] The above description of the disclosed embodiments enables those skilled in the art to make or use this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining a three-dimensional mesh, comprising: Acquire a two-dimensional image, the two-dimensional image being used to describe at least one region of a face in two-dimensional space; Construct a three-dimensional mesh corresponding to the two-dimensional image, and obtain the key point detection results of the two-dimensional image. The three-dimensional mesh is used to describe the at least one region in three-dimensional space, and the key point detection results include the key points of the at least one region. Based on the key point detection results, the three-dimensional mesh is deformed to obtain a deformed mesh. The state of at least one region described by the deformed mesh in the two-dimensional space is consistent with the state of at least one region described by the key point detection results, or the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in the three-dimensional space.

2. The method according to claim 1, wherein, For any of the regions, the projection points of the vertices on the deformed mesh used to describe the region in the two-dimensional space coincide with the key points of the region, or the projection points of the vertices on the deformed mesh used to describe the region coincide with the projection points of the key points of the region in the three-dimensional space.

3. The method according to claim 1 or 2, wherein, The process of determining the deformed mesh includes: For any of the key points, find the first vertex that matches the key point from the three-dimensional mesh, and determine the deformation parameters of the first vertex based on the distance between the projection point of the key point in the three-dimensional space and the first vertex; Based on the deformation parameters of the first vertex, the deformation parameters of the three-dimensional mesh are determined, and the deformation parameters of the three-dimensional mesh include the deformation parameters of the first vertex; The three-dimensional mesh is deformed according to its deformation parameters to obtain the deformed mesh.

4. The method according to claim 3, wherein, The at least one region includes a first region, and the key points of the first region include a first key point; The process of determining the first vertex that matches the first key point includes: Obtain the projection result of the three-dimensional mesh onto the two-dimensional image, the projection result including the projection points of each vertex on the three-dimensional mesh in the two-dimensional space; Based on the distance between the first key point and the projection points of each vertex in the two-dimensional space, a first vertex matching the first key point is determined; the projection result includes a target projection point, which is the projection point of the first vertex matching the first key point in the two-dimensional space, and the distance between the first key point and the target projection point is not greater than the distance between the first key point and other projection points in the projection result except for the target projection point.

5. The method according to claim 4, wherein, The first area is the cheek; After obtaining the projection result of the three-dimensional mesh onto the two-dimensional image, the method further includes: A set of projection points is determined from the projection results, the set of projection points being used to represent the cheek edge formed by the projection points of some vertices on the three-dimensional mesh in the two-dimensional space; The step of determining the first vertex matching the first keypoint based on the distance between the first keypoint and the projection points of each vertex in the two-dimensional space includes: Based on the distance between the first key point and each projection point in the projection point set, a first vertex matching the first key point is determined. The projection point set includes the target projection point. The distance between the first key point and the target projection point is not greater than the distance between the first key point and other projection points in the projection point set besides the target projection point.

6. The method according to any one of claims 3-5, wherein, The at least one region includes a second region, and the key points of the second region include a second key point; The process of determining the first vertex that matches the second keypoint includes: Obtain the vertex annotation information of the three-dimensional mesh, wherein the vertex annotation information is used to describe the mapping relationship between each vertex on the three-dimensional mesh and the key points of the at least one region; Based on the vertex annotation information, a first vertex matching the second key point is determined, and the mapping relationship includes the correspondence between the first vertex matching the second key point and the second key point.

7. The method according to claim 6, wherein, The difference between the distribution state of key points in the second region and the standard distribution state corresponding to the second region is lower than a preset difference. The standard distribution state is determined based on the key point detection results of at least one frontal face image.

8. The method according to claim 6, wherein, The three-dimensional mesh contains at least one vertex for describing a symmetrical region of the second region; the at least one vertex includes a second vertex, and there is a symmetrical relationship between the second vertex and the first vertex that matches the second key point; The process of determining the deformation parameters of the three-dimensional mesh includes: If the difference between the distribution state of the key points in the symmetrical region and the standard distribution state corresponding to the symmetrical region is not less than a preset difference, then the deformation parameters of the second vertex are predicted based on the deformation parameters of the first vertex that matches the second key point. The deformation parameters of the three-dimensional mesh are determined based on the deformation parameters of the second vertex and the deformation parameters of the first vertex that matches the second key point.

9. The method according to claim 8, wherein, The third vertex is any other vertex in the three-dimensional mesh besides the first vertex and the second vertex; The deformation parameters of the three-dimensional mesh also include the deformation parameters of the third vertex, which are obtained by interpolation based on the deformation parameters of the first vertex and the deformation parameters of the second vertex.

10. The method according to any one of claims 4-9, further comprising: Obtain the camera parameters of the two-dimensional image; Based on the camera parameters, determine the rotation angle of the face; If the rotation angle is 0, then each region in the at least one region is determined as the second region; If the rotation angle is greater than 0, then the cheek in the at least one region is defined as the first region, and the other regions in the at least one region besides the cheek are defined as the second region.

11. The method according to claim 10, wherein, The at least one region includes part or all of the cheek, eyes, nose, mouth, and ears.

12. The method according to any one of claims 1-11, wherein, The key point detection results include at least 20 key points of the left ear and 20 key points of the right ear; the 20 key points of the left ear are used to describe the outer contour of the left ear; the 20 key points of the right ear are used to describe the outer contour of the right ear. And / or, The key point detection results include at least 20 key points of the iris in the left eye and 20 key points of the iris in the right eye; the 20 key points of the iris in the left eye are used to describe the iris contour of the left eye; the 20 key points of the iris in the right eye are used to describe the iris contour of the right eye.

13. The method according to claim 12, wherein, The key point detection results also include 16 key points in the middle orbit of the left eye, 16 key points in the middle orbit of the right eye, 4 key points on the left side of the nose, and 4 key points on the right side of the nose. The 16 key points in the eye socket of the left eye are used to describe the outline of the left eye; The 16 key points in the eye socket of the right eye are used to describe the outline of the right eye; The four key points on the left side of the nose are used to describe the left contour of the bridge of the nose. The four key points on the right side of the nose are used to describe the right contour of the bridge of the nose.

14. The method according to any one of claims 1-13, wherein, The process of constructing the three-dimensional mesh includes: The two-dimensional image is subjected to three-dimensional reconstruction processing to obtain the reconstruction result; The reconstruction result is retopologically processed according to the target mesh standard to obtain the three-dimensional mesh, which conforms to the target mesh standard.

15. The method according to any one of claims 1-14, further comprising: Obtain the texture map corresponding to the two-dimensional image under the deformed mesh; Based on the optimization results of the texture mapping, the deformed mesh is rendered to obtain a three-dimensional face model; the optimization results are determined based on at least one pre-constructed texture optimization model, and different texture optimization models have different optimization functions.

16. A three-dimensional mesh determination device, comprising: The acquisition unit is configured to acquire a two-dimensional image, the two-dimensional image being used to describe at least one region of a face in a two-dimensional space; The construction unit is configured to construct a three-dimensional mesh corresponding to the two-dimensional image and to obtain key point detection results of the two-dimensional image. The three-dimensional mesh is used to describe the at least one region in three-dimensional space, and the key point detection results include key points of the at least one region. The deformation unit is configured to deform the three-dimensional mesh based on the key point detection results to obtain a deformed mesh, wherein the state of at least one region described by the deformed mesh in the two-dimensional space is consistent with the state of at least one region described by the key point detection results, or the state of at least one region described by the deformed mesh is consistent with the state of at least one region described by the key point detection results in the three-dimensional space.

17. An electronic device comprising: Processor and memory; The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-15.

18. A computer-readable medium storing instructions or a computer program, wherein, When the instructions or computer program are executed on the device, the device causes the device to perform the method according to any one of claims 1-15.

19. A computer program product comprising a computer program carried on a non-transitory computer-readable medium, wherein, The computer program includes program code for performing the method according to any one of claims 1-15.

Images