Face image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a face image processing method and device, electronic equipment and storage medium. The method comprises: obtaining first lip position information of a target face in a face image corresponding to a first face mesh model, second lip position information of the target face in the face image, and first tooth position information of teeth in the target face in the face image; determining position difference information between the second lip position information and the first tooth position information; determining first weight data corresponding to a preset tooth shape of the target face based on the first lip position information and the position difference information; and driving first mesh vertices associated with the preset tooth shape in a face mesh model corresponding to the target face based on the first weight data to obtain a second face mesh model corresponding to the target face. The present disclosure can realize the representation of real face tooth information by a face mesh model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for processing facial images. Background Technology

[0002] With the continuous development of computer vision, computer graphics, and artificial intelligence technologies, virtual human special effects have been widely used.

[0003] In related technologies, computer devices detect input facial images and drive virtual human facial movements based on the detection results, thereby presenting changes in facial expressions on the virtual human face.

[0004] However, in related technologies, the accuracy of tooth positioning in virtual human faces is low, and the realism of virtual humans is poor. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, and storage medium for processing facial images, to at least solve the problems of low accuracy in the positioning of teeth and poor realism of virtual human faces in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a face image processing method is provided, the method comprising:

[0007] The first lip position information of the target face in the face image corresponding to the first face mesh model, the second lip position information of the target face in the face image, and the first tooth position information of the teeth in the target face in the face image are obtained.

[0008] Determine the positional difference information between the second lip position information and the first tooth position information;

[0009] Based on the first lip position information and the position difference information, a first weight data corresponding to the target face and the preset tooth shape is determined. The first weight data characterizes the degree of deformation of the tooth shape corresponding to the target face relative to the preset tooth shape.

[0010] Based on the first weight data, the first grid vertex in the face mesh model corresponding to the target face, which is associated with the preset tooth shape, is driven to obtain the second face mesh model corresponding to the target face. The first grid vertex is used to represent the addition position corresponding to the tooth effect.

[0011] In some possible designs, the step of driving the first mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset tooth shape, based on the first weight data, to obtain the second face mesh model corresponding to the target face includes:

[0012] Determine the vertex displacement change information corresponding to the preset tooth shape, wherein the vertex displacement change information characterizes the change in the position of the mesh vertex between the face mesh model corresponding to the preset tooth shape and the original face mesh model corresponding to the target face;

[0013] Based on the first weight data and the vertex displacement change information, determine the first vertex displacement data corresponding to the first grid vertex;

[0014] Based on the first vertex displacement data, the first mesh vertex is driven to move, thereby obtaining the second face mesh model.

[0015] In some possible designs, determining the first weighted data corresponding to a preset tooth shape for the target face based on the first lip position information and the position difference information includes:

[0016] Obtain the second tooth position information in the original face mesh model corresponding to the target face. The second tooth position information refers to the position information of the mesh vertex associated with the tooth in the original face mesh model.

[0017] Obtain the vertex displacement change information corresponding to the preset tooth shape;

[0018] Based on the first lip position information and the position difference information, the third tooth position information is determined. The third tooth position information represents the reference position of the tooth in the target three-dimensional coordinate system. The target three-dimensional coordinate system refers to the three-dimensional coordinate system corresponding to the first face mesh model.

[0019] Based on preset constraints, constraint analysis is performed on the target position error corresponding to the third tooth position information to obtain the first weight data;

[0020] The preset constraint condition refers to the condition that constrains the target position error to reach the minimum value. The target position error is the position error between the third tooth position information and the predicted tooth position information. The predicted tooth position information is the tooth position information after offsetting the second tooth position information based on the target vertex displacement information. The target vertex displacement information is obtained by weighting the vertex displacement change information based on the first weight data. The first weight data is determined when the target position error reaches the minimum value.

[0021] In some possible designs, the first lip position information includes the first upper lip position data corresponding to the target face in the first face mesh model and the first lower lip position data corresponding to the target face in the first face mesh model. The position difference information includes first position difference data and second position difference data. The first position difference data represents the position difference between the upper lip position and the upper teeth position in the face image, and the second position difference data represents the position difference between the lower lip position and the lower teeth position in the face image.

[0022] The step of determining the position information of the third tooth based on the first lip position information and the position difference information includes:

[0023] Based on the first upper lip position data and the first position difference data, the reference upper tooth position data corresponding to the tooth in the target three-dimensional coordinate system is determined;

[0024] Based on the first lower lip position data and the second position difference data, the reference lower tooth position data corresponding to the tooth in the target three-dimensional coordinate system is determined, and the third tooth position information includes the reference upper tooth position data and the reference lower tooth position data.

[0025] In some possible designs, the teeth include upper teeth, the preset tooth shape includes a preset upper tooth shape, the second tooth position information includes the original upper tooth position data corresponding to the upper tooth vertex associated with the upper tooth in the original face mesh model, and the vertex displacement change information includes a first displacement matrix corresponding to the upper tooth vertex. The first displacement matrix represents the vertex displacement change relationship between the upper tooth vertex in the original face mesh model and the upper tooth vertex in the face mesh model corresponding to the preset upper tooth shape.

[0026] The first weight data is obtained by performing constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints, including:

[0027] Based on the preset constraints, the upper tooth position error corresponding to the reference upper tooth position data is constrained and analyzed to obtain the second weight data of the target face corresponding to the preset upper tooth shape.

[0028] The target position error includes the upper tooth position error, which is the error between the reference upper tooth position data and the predicted upper tooth position data. The predicted upper tooth position data is the upper tooth position data after offsetting the original upper tooth position data based on the second vertex displacement data. The target vertex displacement information includes the second vertex displacement data, which is obtained by weighting the first displacement matrix based on the second weight data.

[0029] In some possible designs, the step of driving the first mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset tooth shape, based on the first weight data, to obtain the second face mesh model corresponding to the target face includes:

[0030] Based on the second weight data, the second mesh vertex in the face mesh model corresponding to the target face, which is associated with the preset upper teeth shape, is driven to obtain the second face mesh model, wherein the first mesh vertex includes the second mesh vertex.

[0031] In some possible designs, the step of driving the second mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset upper teeth shape, based on the second weight data, to obtain the second face mesh model includes:

[0032] Based on the second weight data and the first displacement matrix, determine the second vertex displacement data corresponding to the second grid vertex;

[0033] Based on the displacement data of the second vertex, the second grid vertex is driven to move, thereby obtaining the second face grid model.

[0034] In some possible designs, the teeth include lower teeth, the preset tooth shape also includes a preset lower tooth shape, the second tooth position information also includes the original lower tooth position data corresponding to the lower tooth vertex associated with the lower tooth in the original face mesh model, the vertex displacement change information also includes the second displacement matrix corresponding to the lower tooth vertex, the second displacement matrix characterizes the vertex displacement change relationship between the lower tooth vertex in the original face mesh model and the lower tooth vertex in the face mesh model corresponding to the preset lower tooth shape;

[0035] The first weight data is obtained by performing constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints, including:

[0036] Based on the preset constraints, the lower tooth position error corresponding to the reference lower tooth position data is constrained and analyzed to obtain the third weight data of the target face corresponding to the preset lower tooth shape.

[0037] The target position error includes the mandibular position error, which is the error between the reference mandibular position data and the predicted mandibular position data. The predicted mandibular position data is the mandibular position data after offsetting the original mandibular position data based on the third vertex displacement data. The target vertex displacement information includes the third vertex displacement data, which is obtained by weighting the second displacement matrix based on the third weight data.

[0038] In some possible designs, the step of driving the first mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset tooth shape, based on the first weight data, to obtain the second face mesh model corresponding to the target face includes:

[0039] Based on the third weight data, the third grid vertex in the face grid model corresponding to the target face, which is associated with the preset lower tooth shape, is driven to obtain the second face grid model, wherein the first grid vertex includes the third grid vertex.

[0040] In some possible designs, the step of driving the third mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset lower tooth shape, based on the third weight data, to obtain the second face mesh model includes:

[0041] Based on the third weight data and the second displacement matrix, the third vertex displacement data corresponding to the third grid vertex is determined;

[0042] Based on the displacement data of the third vertex, the third grid vertex is driven to move, thereby obtaining the second face grid model.

[0043] In some possible designs, the second lip position information includes the second upper lip position data corresponding to the target face in the face image and the second lower lip position data corresponding to the target face in the face image, and the first tooth position information includes the first upper tooth position data corresponding to the target face in the face image and the first lower tooth position data corresponding to the target face in the face image.

[0044] The determination of the positional difference information between the second lip position information and the first tooth position information includes:

[0045] Determine the first positional difference data between the second upper lip position data and the first upper tooth position data;

[0046] A second positional difference data is determined between the second lower lip position data and the first lower tooth position data, wherein the positional difference information includes the first positional difference data and the first positional difference data.

[0047] In some possible designs, obtaining the first lip position information corresponding to the target face in the first face mesh model, the second lip position information corresponding to the target face in the face image, and the first tooth position information corresponding to the teeth in the target face in the face image includes:

[0048] The facial image is subjected to expression detection processing to obtain fourth weight data of the target face corresponding to a preset expression shape. The fourth weight data represents the degree of deformation of the expression shape of the target face relative to the preset expression shape.

[0049] Based on the fourth weight data, the grid vertices in the original face mesh model corresponding to the target face are driven to obtain the first face mesh model, which includes the first lip position information;

[0050] The face image is processed by mouth detection to obtain the second lip position information and the first tooth position information.

[0051] In some possible designs, the face image is obtained by capturing the target face using a monocular camera.

[0052] According to a second aspect of the present disclosure, a face image processing apparatus is provided, the apparatus comprising:

[0053] The location information acquisition module is configured to acquire the first lip position information corresponding to the target face in the first face mesh model, the second lip position information corresponding to the target face in the face image, and the first tooth position information corresponding to the teeth in the target face in the face image.

[0054] The position difference determination module is configured to determine the position difference information between the second lip position information and the first tooth position information;

[0055] The weight determination module is configured to perform a first weight data based on the first lip position information and the position difference information to determine the target face corresponding to a preset tooth shape. The first weight data characterizes the degree of deformation of the tooth shape corresponding to the target face relative to the preset tooth shape.

[0056] The face mesh model driving module is configured to execute, based on the first weight data, drive the first mesh vertex in the face mesh model corresponding to the target face that is associated with the preset tooth shape, to obtain the second face mesh model corresponding to the target face, wherein the first mesh vertex is used to characterize the addition position corresponding to the tooth effect.

[0057] In some possible designs, the face mesh model driving module includes:

[0058] The displacement change information determination unit is configured to determine the vertex displacement change information corresponding to the preset tooth shape, wherein the vertex displacement change information represents the change in the position of the mesh vertex between the face mesh model corresponding to the preset tooth shape and the original face mesh model corresponding to the target face.

[0059] The displacement data determination unit is configured to determine the first vertex displacement data corresponding to the first mesh vertex based on the first weight data and the vertex displacement change information.

[0060] The vertex displacement unit is configured to drive the first mesh vertices to displacement based on the first vertex displacement data to obtain the second face mesh model.

[0061] In some possible designs, the weight determination module includes:

[0062] The tooth position acquisition unit is configured to acquire the second tooth position information in the original face mesh model corresponding to the target face, wherein the second tooth position information refers to the position information of the mesh vertex associated with the tooth in the original face mesh model;

[0063] The displacement change information acquisition unit is configured to acquire vertex displacement change information corresponding to the preset tooth shape;

[0064] The tooth position determination unit is configured to determine third tooth position information based on the first lip position information and the position difference information. The third tooth position information represents the reference position of the tooth in the target three-dimensional coordinate system, which refers to the three-dimensional coordinate system corresponding to the first face mesh model.

[0065] The weight data determination unit is configured to perform constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints to obtain the first weight data.

[0066] The preset constraint condition refers to the condition that constrains the target position error to reach the minimum value. The target position error is the position error between the third tooth position information and the predicted tooth position information. The predicted tooth position information is the tooth position information after offsetting the second tooth position information based on the target vertex displacement information. The target vertex displacement information is obtained by weighting the vertex displacement change information based on the first weight data. The first weight data is determined when the target position error reaches the minimum value.

[0067] In some possible designs, the first lip position information includes the first upper lip position data corresponding to the target face in the first face mesh model and the first lower lip position data corresponding to the target face in the first face mesh model. The position difference information includes first position difference data and second position difference data. The first position difference data represents the position difference between the upper lip position and the upper teeth position in the face image, and the second position difference data represents the position difference between the lower lip position and the lower teeth position in the face image.

[0068] The tooth position determination unit includes:

[0069] The reference upper tooth position determination subunit is configured to perform the determination of reference upper tooth position data corresponding to the tooth in the target three-dimensional coordinate system based on the first upper lip position data and the first position difference data.

[0070] The reference mandibular position determination subunit is configured to perform a process based on the first lower lip position data and the second position difference data to determine the reference mandibular position data corresponding to the tooth in the target three-dimensional coordinate system. The third tooth position information includes the reference upper tooth position data and the reference mandibular position data.

[0071] In some possible designs, the teeth include upper teeth, the preset tooth shape includes a preset upper tooth shape, the second tooth position information includes the original upper tooth position data corresponding to the upper tooth vertex associated with the upper tooth in the original face mesh model, and the vertex displacement change information includes a first displacement matrix corresponding to the upper tooth vertex. The first displacement matrix represents the vertex displacement change relationship between the upper tooth vertex in the original face mesh model and the upper tooth vertex in the face mesh model corresponding to the preset upper tooth shape.

[0072] The weight data determination unit is specifically configured to perform constraint analysis on the upper tooth position error corresponding to the reference upper tooth position data based on the preset constraint conditions, and obtain the second weight data of the target face corresponding to the preset upper tooth shape.

[0073] The target position error includes the upper tooth position error, which is the error between the reference upper tooth position data and the predicted upper tooth position data. The predicted upper tooth position data is the upper tooth position data after offsetting the original upper tooth position data based on the second vertex displacement data. The target vertex displacement information includes the second vertex displacement data, which is obtained by weighting the first displacement matrix based on the second weight data.

[0074] In some possible designs, the face mesh model driving module is specifically configured to execute, based on the second weight data, drive the second mesh vertices in the face mesh model corresponding to the target face that are associated with the preset upper teeth shape, to obtain the second face mesh model, wherein the first mesh vertex includes the second mesh vertex.

[0075] In some possible designs, the displacement data determining unit is specifically configured to determine the second vertex displacement data corresponding to the second mesh vertex based on the second weight data and the first displacement matrix;

[0076] The vertex displacement unit is specifically configured to drive the second mesh vertices to displacement based on the second vertex displacement data, thereby obtaining the second face mesh model.

[0077] In some possible designs, the teeth also include lower teeth, the preset tooth shape also includes a preset lower tooth shape, the second tooth position information also includes the original lower tooth position data corresponding to the lower tooth vertex associated with the lower tooth in the original face mesh model, the vertex displacement change information also includes the second displacement matrix corresponding to the lower tooth vertex, the second displacement matrix characterizes the vertex displacement change relationship between the lower tooth vertex in the original face mesh model and the lower tooth vertex in the face mesh model corresponding to the preset lower tooth shape;

[0078] The weight data determination unit is further configured to perform constraint analysis on the lower tooth position error corresponding to the reference lower tooth position data based on the preset constraint conditions, and obtain the third weight data of the target face corresponding to the preset lower tooth shape.

[0079] The target position error includes the mandibular position error, which is the error between the reference mandibular position data and the predicted mandibular position data. The predicted mandibular position data is the mandibular position data after offsetting the original mandibular position data based on the third vertex displacement data. The target vertex displacement information includes the third vertex displacement data, which is obtained by weighting the second displacement matrix based on the third weight data.

[0080] In some possible designs, the face mesh model driving module is further configured to execute, based on the third weight data, drive the third mesh vertex in the face mesh model corresponding to the target face that is associated with the preset lower tooth shape, to obtain the second face mesh model, wherein the first mesh vertex includes the third mesh vertex.

[0081] In some possible designs, the displacement data determining unit is further configured to perform the determination of the third vertex displacement data corresponding to the third mesh vertex based on the third weight data and the second displacement matrix;

[0082] The vertex displacement unit is further configured to drive the third mesh vertex to displacement based on the third vertex displacement data, thereby obtaining the second face mesh model.

[0083] In some possible designs, the second lip position information includes the second upper lip position data corresponding to the target face in the face image and the second lower lip position data corresponding to the target face in the face image, and the first tooth position information includes the first upper tooth position data corresponding to the target face in the face image and the first lower tooth position data corresponding to the target face in the face image.

[0084] The location difference determination module includes:

[0085] The first position difference determination unit is configured to determine a first position difference data between the second upper lip position data and the first upper tooth position data;

[0086] The second position difference determination unit is configured to determine a second position difference data between the second lower lip position data and the first lower tooth position data, wherein the position difference information includes the first position difference data and the first position difference data.

[0087] In some possible designs, the location information acquisition module includes:

[0088] The expression weight determination unit is configured to perform expression detection processing on the face image to obtain fourth weight data of the target face corresponding to a preset expression form, wherein the fourth weight data characterizes the degree of deformation of the expression form corresponding to the target face relative to the preset expression form.

[0089] The face mesh model driving module is also configured to execute, based on the fourth weight data, drive the mesh vertices in the original face mesh model corresponding to the target face to obtain the first face mesh model, wherein the first face mesh model includes the first lip position information;

[0090] The mouth detection unit is configured to perform mouth detection processing on the face image to obtain the second lip position information and the first tooth position information.

[0091] In some possible designs, the face image is obtained by capturing the target face using a monocular camera.

[0092] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the face image processing method as described in any one of the first aspects above.

[0093] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the face image processing method described in any one of the first aspects of the present disclosure.

[0094] According to a fifth aspect of the present disclosure, a computer program product including instructions is provided, which, when run on a computer, causes the computer to perform the face image processing method described in any one of the first aspects of the present disclosure.

[0095] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0096] By detecting the two-dimensional positions of the lips and teeth in the target face image, the two-dimensional positional differences between the lips and teeth in the face image can be determined. Based on the above positional difference information and the three-dimensional position of the lips corresponding to the target face in the first face mesh model, weight data that can characterize the degree of deformation between the tooth shape of the target face and the preset tooth shape can be determined. By driving the mesh vertices in the face mesh model associated with the preset tooth shape according to the weight data, the face mesh model can be made to represent the tooth information of the real face, which improves the consistency between the tooth shape of the face mesh model and the tooth shape of the real face, and improves the accuracy of the tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0097] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0098] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0099] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment;

[0100] Figure 2 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 1 ;

[0101] Figure 3 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 2 ;

[0102] Figure 4 This is a block diagram of a face image processing apparatus according to an exemplary embodiment;

[0103] Figure 5 This is a block diagram illustrating an electronic device for face image processing according to an exemplary embodiment. Detailed Implementation

[0104] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0105] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0106] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0107] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include terminal 100 and server 200.

[0108] Terminal 100 can be used to provide facial image processing services to any user. Specifically, terminal 100 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, or software running on the aforementioned electronic devices, such as applications. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0109] In an optional embodiment, server 200 can provide background services to terminal 100. Specifically, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0110] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, other application environments may also be included, such as more terminals.

[0111] In the embodiments described in this specification, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication, and this disclosure does not impose any restrictions.

[0112] Figure 2 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 1 Optionally, this face image processing method is used in an electronic device. Optionally, the electronic device can be a terminal or a server. Figure 2 As shown, the method may include the following steps (210-240).

[0113] Step 210: Obtain the first lip position information of the target face in the face image corresponding to the first face mesh model, the second lip position information of the target face in the face image, and the first tooth position information of the teeth in the target face in the face image.

[0114] In one possible implementation, the face image is obtained by capturing the target face using a monocular camera. The face image captured by the monocular camera is a two-dimensional image (non-depth image), not a depth image with three-dimensional depth feature information.

[0115] Related technologies require depth images containing facial depth information for face modeling, and these technologies are highly dependent on the presence of 3D depth features related to the face in the depth images. In contrast, the technical solution provided in this disclosure can achieve 3D face modeling and driving based on 2D face images without depth information captured by a monocular camera, and can represent the teeth information of a realistic face. This reduces the requirements for camera equipment in 3D face modeling and driving scenarios, allowing devices without a depth camera to perform 3D face modeling and driving.

[0116] In another possible implementation, the aforementioned face image is a depth image.

[0117] Optionally, the first lip position information includes the first upper lip position data corresponding to the target face in the first face mesh model and the first lower lip position data corresponding to the target face in the first face mesh model.

[0118] Optionally, the aforementioned first upper lip position data refers to the vertex position data corresponding to the upper lip vertex associated with the upper lip in the first face mesh model. Similarly, the aforementioned first lower lip position data refers to the vertex position data corresponding to the lower lip vertex associated with the lower lip in the first face mesh model.

[0119] Optionally, the aforementioned first face mesh model is a face mesh model obtained by driving the original face mesh model based on the fourth weight data corresponding to the preset expression shape. Optionally, the fourth weight data is a weight coefficient.

[0120] The aforementioned preset facial expressions refer to pre-defined facial expressions, such as the 51 preset blendshapes in Arkit that correspond to 51 different facial expressions. These preset facial expressions are unrelated to tooth shapes. Each preset facial expression has its corresponding preset facial mesh model. The lack of tooth definition in the preset facial expressions results in missing tooth information in the corresponding preset facial mesh model.

[0121] Both the preset face mesh model and the original face mesh model corresponding to the preset expression form include mesh vertices representing preset positions on the face. However, since the positions of key points on the face change under different expression forms, the vertex positions of the mesh vertices in the preset face mesh model and the original face mesh model differ. Therefore, the vertex displacement between the face mesh models under different expression forms can represent the changes in facial expression. Optionally, the aforementioned fourth weight data can represent the degree of deformation of the expression form corresponding to the target face relative to the preset expression form.

[0122] There is a correspondence between the grid vertices in the preset face mesh model corresponding to the preset expression and the grid vertices in the original face mesh model. There may be positional differences between the corresponding grid vertices. This positional difference can represent the displacement of the grid vertices. For example, when the original face mesh model is converted into the preset expression, the displacement required for each grid vertex is the positional difference between the grid vertices in the original face mesh model and the preset face mesh model.

[0123] In real-world scenarios, facial expressions can change freely and to varying degrees, not limited to preset expression forms. Therefore, by performing expression detection on facial images, a fourth weight data that can characterize the degree of deformation of a real face relative to a preset expression form can be determined. Based on this fourth weight data, the actual displacement of mesh vertices can be controlled, thereby enabling the facial mesh model to represent any expression form.

[0124] In an exemplary embodiment, the implementation process of step 210 includes: acquiring a face image; performing expression detection processing on the face image to obtain fourth weight data of the target face corresponding to a preset expression form, wherein the fourth weight data characterizes the degree of deformation of the expression form corresponding to the target face relative to the preset expression form; and, based on the fourth weight data, driving the grid vertices in the original face mesh model corresponding to the target face to obtain a first face mesh model, wherein the first face mesh model includes first lip position information.

[0125] Optionally, preset facial key points are detected in the image to obtain two-dimensional position data of the preset facial key points in the face image. The two-dimensional position data is then transformed into a three-dimensional coordinate system according to preset coordinate system transformation rules, such as coordinate system transformation matrix, to obtain three-dimensional coordinate data. The three-dimensional coordinate data corresponding to the preset facial key points under preset expression forms are compared to obtain the aforementioned fourth weight data. Then, the vertices in the face network model are driven to move according to the fourth weight data.

[0126] Optionally, vertex displacement change information corresponding to a preset facial expression shape is determined. Optionally, the vertex displacement change information corresponding to the preset facial expression shape includes vertex position difference data between the mesh vertices in the preset face mesh model corresponding to the preset facial expression shape and the mesh vertices in the original face mesh model. Optionally, by multiplying the above-mentioned fourth weight data with the above-mentioned vertex position difference data, the vertex displacement data corresponding to the mesh vertices associated with the preset facial expression shape in the face mesh model can be determined. Based on the vertex displacement data, the mesh vertices associated with the preset facial expression shape in the original face mesh model can be driven to displace, thereby obtaining a first face mesh model corresponding to the current facial expression shape of the target face.

[0127] Optionally, the tooth shape and the preset expression shape can be separated, so that when driving the face mesh model, the driving effect of other facial expressions besides the teeth will not be affected, while ensuring the realism of the face mesh model's expression and teeth.

[0128] In an exemplary embodiment, the implementation process of step 210 above further includes: performing mouth detection processing on the face image to obtain second lip position information and first tooth position information.

[0129] Optionally, the second lip position information includes the second upper lip position data corresponding to the target face in the face image and the second lower lip position data corresponding to the target face in the face image.

[0130] Optionally, the aforementioned second upper lip position data refers to the two-dimensional position data of key points detected from the face image that are associated with the upper lip, located in the two-dimensional coordinate system corresponding to the face image. For example, the two-dimensional position data of the upper lip center point in the image coordinate system where the face image is located. Similarly, the aforementioned second lower lip position data refers to the two-dimensional position data of key points detected from the face image that are associated with the lower lip, located in the two-dimensional coordinate system corresponding to the face image. For example, the two-dimensional position data of the lower lip center point in the image coordinate system where the face image is located.

[0131] Optionally, the first tooth position information includes the position data of the first upper teeth of the target face in the face image and the position data of the first lower teeth of the target face in the face image.

[0132] Optionally, the aforementioned first upper tooth position data refers to the two-dimensional position data of key points associated with the upper teeth detected from the face image in the two-dimensional coordinate system corresponding to the face image, such as the two-dimensional position data of the upper tooth apex point in the image coordinate system where the face image is located. Similarly, the aforementioned first lower tooth position data refers to the two-dimensional position data of key points associated with the lower teeth detected from the face image in the two-dimensional coordinate system corresponding to the face image, such as the two-dimensional position data of the lower tooth apex point in the image coordinate system where the face image is located.

[0133] In one possible implementation, mouth detection processing is performed on a facial image to obtain two-dimensional lip position data corresponding to lip keypoints and two-dimensional tooth position data corresponding to tooth keypoints. The second lip information includes two-dimensional lip position data, which includes second upper lip position data corresponding to upper lip keypoints and second lower lip position data corresponding to lower lip keypoints. The first tooth position information includes two-dimensional tooth position data, which includes first upper tooth position data corresponding to upper tooth keypoints and first lower tooth position data corresponding to lower tooth keypoints. This application embodiment does not limit the selection of upper lip keypoints, lower lip keypoints, upper tooth keypoints, and lower tooth keypoints. The detection of the above keypoints can be implemented based on a keypoint detection model, and this application embodiment does not limit this approach.

[0134] Step 220: Determine the positional difference information between the second lip position information and the first tooth position information.

[0135] In one possible implementation, according to a preset correspondence, the lip position data in the second lip position information is compared with the corresponding tooth position data in the first tooth position information to obtain various position difference data in the position difference information.

[0136] In another possible implementation, a first positional difference data is determined between the second upper lip position data and the first upper tooth position data; a second positional difference data is determined between the second lower lip position data and the first lower tooth position data.

[0137] Optionally, the positional difference information includes first positional difference data and second positional difference data. The first positional difference data represents the positional difference between the upper lip and upper teeth in the face image, and the second positional difference data represents the positional difference between the lower lip and lower teeth in the face image.

[0138] In one possible implementation, on the one hand, the two-dimensional position data corresponding to the center point of the upper lip is subtracted from the two-dimensional position data corresponding to the tip point of the upper tooth to obtain a first positional difference data between the center point of the upper lip and the tip point of the upper tooth, such as a first distance between the center point of the upper lip and the tip point of the upper tooth; on the other hand, the two-dimensional position data corresponding to the center point of the lower lip is subtracted from the two-dimensional position data corresponding to the tip point of the lower tooth to obtain a second positional difference data between the center point of the lower lip and the tip point of the lower tooth, such as a second distance between the center point of the lower lip and the tip point of the lower tooth.

[0139] In this embodiment of the present disclosure, by determining the relative positions of the upper lip and upper teeth, and the lower lip and upper teeth in the face image, the relative positions of the upper lip and upper teeth, and the lower lip and upper teeth in the face mesh model can be measured, thereby improving the accuracy of weight data determination.

[0140] Step 230: Based on the first lip position information and position difference information, determine the first weight data corresponding to the preset tooth shape of the target face.

[0141] Optionally, the first weighted data characterizes the degree of deformation of the tooth shape corresponding to the target face relative to the preset tooth shape.

[0142] The aforementioned preset tooth shapes are predefined tooth shapes. Each preset tooth shape has a corresponding preset face mesh model. For example, each preset tooth shape corresponds to a face blendshape, i.e., a face mesh model, to display the corresponding tooth shape. By adding additional preset tooth shapes, the missing tooth information in the preset expression shapes can be compensated for, without affecting other facial expression shapes.

[0143] Both the face mesh model corresponding to the preset tooth shape and the original face mesh model include mesh vertices representing preset positions in the face. However, since the positions of the vertices associated with the teeth in the face will change under different tooth shapes, the vertex positions of the mesh vertices associated with the preset tooth shape will be different in different face mesh models. Therefore, the displacement of the mesh vertices associated with the preset tooth shape can represent the changes in the position and shape of the teeth in the face.

[0144] Furthermore, in real-world scenarios, teeth in a human face can move freely and to varying degrees, not just limited to a preset tooth shape. Therefore, by determining the aforementioned first lip position information and position difference information, a first weight data that can characterize the degree of deformation of the real human face's teeth relative to the preset tooth shape can be further determined. In subsequent processing, the mesh vertices associated with the preset tooth shape can be displaced based on this first weight data, thereby enabling the face mesh model to represent any tooth shape.

[0145] In one possible implementation, the facial mesh model corresponding to the preset tooth shape is added to the virtual human resource library corresponding to the target face. Optionally, the virtual human resource library also includes the facial mesh model corresponding to the preset expression shape.

[0146] In another possible implementation, the aforementioned preset tooth shapes include preset upper tooth shapes and preset lower tooth shapes. Accordingly, the face mesh models corresponding to each of the preset upper tooth shapes and preset lower tooth shapes are added to the virtual human resource library corresponding to the target face. The face mesh model corresponding to the preset upper tooth shape can be a face mesh model obtained by adjusting the positions of the mesh vertices associated with the preset upper tooth shape in the original face mesh model; similarly, the face mesh model corresponding to the preset lower tooth shape can be a face mesh model obtained by adjusting the positions of the mesh vertices associated with the preset lower tooth shape in the original face mesh model. For example, two blendshapes that move the upper and lower teeth in the y-direction can be added to the driven virtual human resource library.

[0147] Specifically, in modeling software such as Maya, you can select the mesh vertices associated with the upper teeth in the original face mesh model and drag them upward along the y-axis, and select the mesh vertices associated with the lower teeth in the original face mesh model and drag them downward along the y-axis. This will give you the face meshes corresponding to the preset upper and lower tooth shapes, respectively. Then, you can add the two newly added face meshes into two blendshapes, which are the face mesh models.

[0148] Optionally, by fusing the positional difference values ​​from the positional difference information with the lip position data corresponding to the target face in the first face mesh model, the predicted position data of the teeth in the first face mesh model can be estimated. By comparing the tooth position data in the face mesh model corresponding to the preset tooth shape with the predicted position data of the teeth, the aforementioned first weight data can be obtained.

[0149] In an exemplary embodiment, such as Figure 3 As shown, the implementation process of step 230 above includes the following steps (231-234). Figure 3 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 2 .

[0150] Step 231: Obtain the position information of the second tooth in the original face mesh model corresponding to the target face.

[0151] Optionally, the second tooth position information refers to the position information of the mesh vertices associated with the teeth in the original face mesh model, that is, the initial position information of the mesh vertices associated with the teeth in the original face mesh model. The second tooth position information includes the initial position data of the mesh vertices associated with the teeth in the original face mesh model, such as coordinate data.

[0152] Optionally, the second tooth position information includes the original upper tooth position data corresponding to the upper tooth vertex associated with the upper teeth in the original face mesh model, and the original lower tooth position data corresponding to the lower tooth vertex associated with the lower teeth in the original face mesh model.

[0153] Optionally, the aforementioned upper tooth vertex is a mesh vertex associated with upper tooth keypoints detected from the face image, and the aforementioned lower tooth vertex is a mesh vertex associated with lower tooth keypoints detected from the face image.

[0154] Step 232: Obtain vertex displacement change information corresponding to the preset tooth shape.

[0155] Optionally, the vertex displacement change information corresponding to the preset tooth shape includes the displacement matrix of the mesh vertex in the preset face mesh model corresponding to the preset tooth shape.

[0156] Optionally, the displacement matrix includes the vertex position difference data between the mesh vertices in the face mesh model corresponding to the preset tooth shape and the corresponding mesh vertices in the original face mesh model.

[0157] Step 233: Determine the position information of the third tooth based on the position information of the first lip and the position difference information.

[0158] Optionally, the third tooth position information represents the reference position of the tooth in the target three-dimensional coordinate system, which refers to the three-dimensional coordinate system corresponding to the first face mesh model.

[0159] In one possible implementation, the lip position data in the first lip position information is fused with the corresponding position difference data in the position difference information, for example, by adding or subtracting, to obtain the reference position data of the tooth in the target three-dimensional coordinate system, which is the aforementioned third tooth position information.

[0160] In another possible implementation, the aforementioned third tooth position information includes reference upper tooth position data and reference lower tooth position data.

[0161] Specifically, based on the first upper lip position data and the first position difference data, the reference upper tooth position data corresponding to the tooth in the target three-dimensional coordinate system is determined. For example, the first upper lip position data and the first position difference data are fused, such as by addition or subtraction, to obtain the above-mentioned reference upper tooth position data.

[0162] Similarly, based on the first lower lip position data and the second position difference data, the reference lower tooth position data corresponding to the tooth in the target three-dimensional coordinate system is determined. For example, the first lower lip position data and the second position difference data are fused together, such as by addition or subtraction, to obtain the above-mentioned reference lower tooth position data.

[0163] The technical solution provided in this disclosure, on the one hand, can determine a reference three-dimensional position of the upper teeth in a three-dimensional coordinate system by using the two-dimensional relative position between the upper lip and upper teeth in the face image and the three-dimensional position of the upper lip in the first face mesh model; on the other hand, can determine a reference three-dimensional position of the lower teeth in a three-dimensional coordinate system by using the two-dimensional relative position between the lower lip and lower teeth in the face image and the three-dimensional position of the lower lip in the first face mesh model. Based on the reference positions of the upper and lower teeth, the weight data corresponding to the morphology of the upper and lower teeth can be determined respectively, which helps to improve the accuracy of the weight data determination.

[0164] Step 234: Based on preset constraints, perform constraint analysis on the target position error corresponding to the third tooth position information to obtain the first weight data.

[0165] The aforementioned preset constraint condition refers to the condition that the target position error reaches the minimum value. The target position error is the position error between the third tooth position information and the predicted tooth position information. The predicted tooth position information is the tooth position information after offsetting the second tooth position information based on the target vertex displacement information. The target vertex displacement information is obtained by weighting the vertex displacement change information based on the first weight data. The first weight data is determined when the target position error reaches the minimum value.

[0166] In one possible implementation, preset transformation relationship information is obtained, which is used to calculate the target position error, specifically representing the logical correspondence between the target position error, the predicted tooth position information, the second tooth position information, the third tooth position information, and the first weight data.

[0167] Specifically, by transforming the relational information, the first weighted data is multiplied by the preset vertex displacement data in the vertex displacement change information to obtain the target vertex displacement data that needs to be actually displaced, i.e., the aforementioned target vertex displacement information. The mesh vertices associated with teeth in the original face mesh model are then displaced from their initial positions according to the aforementioned target vertex displacement data to obtain the predicted positions. That is, the aforementioned target vertex displacement data is fused to the initial position data (i.e., the second tooth position information), for example, by addition or subtraction, to obtain the predicted tooth position data, i.e., the aforementioned predicted tooth position information.

[0168] By constraining the target position error according to the above transformation relationship information, the first weight data can be obtained.

[0169] The technical solution provided in this disclosure can determine a reference three-dimensional position of the teeth in a three-dimensional coordinate system by using the two-dimensional relative position between the lips and teeth in a face image and the three-dimensional position of the lips in a first face mesh model. When the error between the reference position and the predicted position is minimized according to preset constraints, the weight data corresponding to the preset tooth shape can be determined, which helps to improve the accuracy of the weight data determination.

[0170] In another possible implementation, the teeth include upper teeth, the preset tooth shape includes a preset upper tooth shape, the second tooth position information includes the original upper tooth position data corresponding to the upper tooth vertex associated with the upper tooth in the original face mesh model, and the vertex displacement change information includes the first displacement matrix corresponding to the upper tooth vertex. The first displacement matrix represents the vertex displacement change relationship between the upper tooth vertex in the original face mesh model and the upper tooth vertex in the face mesh model corresponding to the preset upper tooth shape.

[0171] Accordingly, based on preset constraints, constraint analysis is performed on the upper tooth position error corresponding to the reference upper tooth position data to obtain the second weight data of the target face corresponding to the preset upper tooth shape.

[0172] The target position error includes the upper tooth position error, which is the error between the reference upper tooth position data and the predicted upper tooth position data. The predicted upper tooth position data is the upper tooth position data after offsetting the original upper tooth position data based on the second vertex displacement data. The target vertex displacement information includes the second vertex displacement data, which is obtained by weighting the first displacement matrix based on the second weight data.

[0173] The aforementioned first weight data includes second weight data, which characterizes the degree of deformation of the upper teeth shape corresponding to the target face relative to the preset upper teeth shape. The second weight data is determined when the upper teeth position error reaches the minimum value.

[0174] Optionally, the first displacement matrix includes the full range of upper tooth vertex position difference data between the upper tooth vertices in the face mesh model corresponding to the preset upper tooth shape and the corresponding upper tooth vertices in the original face mesh model.

[0175] Optionally, based on the original upper tooth position data and the reference upper tooth position data, the actual upper tooth vertex position difference data can be determined. Based on the actual upper tooth vertex position difference data and the full upper tooth vertex position difference data, the degree of deformation of the actual tooth shape of the target face relative to the preset upper tooth shape can be determined, that is, the second weight data mentioned above can be determined.

[0176] Since the first displacement matrix can characterize the total vertex displacement change of the upper tooth vertex between the original face mesh model and the face mesh model corresponding to the preset upper tooth shape, the second weight data can be accurately determined by comparing the relative displacement change between the original upper tooth position data and the reference upper tooth position data with the total vertex displacement change.

[0177] In another possible implementation, the teeth also include lower teeth, the preset tooth shape also includes a preset lower tooth shape, the second tooth position information also includes the original lower tooth position data corresponding to the lower tooth vertex associated with the lower tooth in the original face mesh model, and the vertex displacement change information also includes the second displacement matrix corresponding to the lower tooth vertex. The second displacement matrix represents the vertex displacement change relationship between the lower tooth vertex in the original face mesh model and the lower tooth vertex in the face mesh model corresponding to the preset lower tooth shape.

[0178] Accordingly, based on preset constraints, constraint analysis is performed on the lower tooth position error corresponding to the reference lower tooth position data to obtain the third weight data of the target face corresponding to the preset lower tooth shape;

[0179] The target position error includes the lower tooth position error, which is the error between the reference lower tooth position data and the predicted lower tooth position data. The predicted lower tooth position data is the lower tooth position data after offsetting the original lower tooth position data based on the third vertex displacement data. The target vertex displacement information includes the third vertex displacement data, which is obtained by weighting the second displacement matrix based on the third weight data.

[0180] The aforementioned first weight data includes third weight data, which characterizes the degree of deformation of the lower teeth shape corresponding to the target face relative to the preset lower teeth shape. The third weight data is determined when the lower teeth position error reaches the minimum value.

[0181] Optionally, the second displacement matrix includes the total difference data of the lower tooth vertex positions between the lower tooth vertices in the face mesh model corresponding to the preset lower tooth shape and the corresponding lower tooth vertices in the original face mesh model.

[0182] Optionally, based on the original lower tooth position data and the reference lower tooth position data, the actual lower tooth vertex position difference data can be determined. Based on the actual lower tooth vertex position difference data and the full lower tooth vertex position difference data, the degree of deformation of the actual lower tooth shape of the target face relative to the preset lower tooth shape can be determined, that is, the third weight data mentioned above can be determined.

[0183] Since the second displacement matrix can characterize the total vertex displacement change of the lower tooth vertex between the original face mesh model and the face mesh model corresponding to the preset lower tooth shape, the third weight data can be accurately determined by comparing the relative displacement change between the original lower tooth position data and the reference lower tooth position data with the total vertex displacement change.

[0184] In one example, the weight data corresponding to the preset tooth shape can be determined by the following formulas (1) and (2), and the preset constraints include the following formulas (1) and (2).

[0185]

[0186] P i =μ i +Bs i ×id toot h T Formula (2)

[0187] Among them, id toot h represents the blendshape coefficient of the tooth to be solved (i.e., the weight data of the preset tooth shape), Bs i μ represents the basis matrix (i.e., the displacement matrix) corresponding to the i-th tooth vertex. i P represents the original tooth position of the i-th tooth vertex in the original face mesh model. i P represents the position of the i-th tooth vertex in the second face mesh model; i ′ represents the reference tooth position data corresponding to the i-th tooth vertex, w i represents the weight, and n represents the number of tooth vertices.

[0188] Step 240: Based on the first weight data, drive the first mesh vertices in the face mesh model corresponding to the target face that are associated with the preset tooth shape to obtain the second face mesh model corresponding to the target face.

[0189] Optionally, the first grid vertex is used to represent the position where the tooth effect is added. Based on the first grid vertex, the corresponding effect can be added to the face mesh model. Optionally, the tooth effect can be added based on the position of the first grid vertex after it has been driven. For example, in short video effects, live streaming, and other scenarios, the effect of teeth whitening, dentures, and dental ornaments can be achieved based on the first grid vertex after it has been driven, improving the fit and stability of the tooth effect. Furthermore, accurate driving of the first grid vertex helps to improve the realism of the virtual human's speech.

[0190] Optionally, based on the second weight data, the second grid vertices in the face grid model corresponding to the target face, which are associated with the preset upper teeth shape, are driven to obtain the second face grid model. The first grid vertices include the second grid vertices.

[0191] By driving the displacement of the second grid vertices in the face mesh model associated with the preset upper tooth shape through the second weight data, the face mesh model can be made to represent the upper tooth information of the real face, which improves the consistency between the upper tooth shape of the face mesh model and the upper tooth shape of the real face, and improves the accuracy of the upper tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0192] Optionally, based on the third weight data, the third grid vertices in the face grid model corresponding to the target face, which are associated with the preset lower tooth shape, are driven to obtain the second face grid model, where the first grid vertices include the third grid vertices.

[0193] By driving the displacement of the third mesh vertices in the face mesh model associated with the preset lower tooth shape using the third weight data, the face mesh model can represent the lower tooth information of a real face, improving the consistency between the lower tooth shape of the face mesh model and the lower tooth shape of a real face, and overall improving the accuracy of the lower tooth information in the face mesh model and the realism of the face mesh model.

[0194] In an exemplary embodiment, such as Figure 3 As shown, the implementation process of step 240 above may include the following steps (241 to 243).

[0195] Step 241: Determine the vertex displacement change information corresponding to the preset tooth shape.

[0196] Optionally, the vertex displacement change information represents the change in the position of the mesh vertices between the face mesh model corresponding to the preset tooth shape and the original face mesh model corresponding to the target face.

[0197] Optionally, the vertex displacement change information corresponding to the preset tooth shape includes the displacement matrix of the mesh vertex in the preset face mesh model corresponding to the preset tooth shape.

[0198] Optionally, the displacement matrix corresponding to the mesh vertex includes the vertex position difference data between the vertex position of the mesh vertex in the face mesh model corresponding to the preset tooth shape and the vertex position of the mesh vertex in the original face mesh model.

[0199] Step 242: Based on the first weight data and vertex displacement change information, determine the first vertex displacement data corresponding to the first grid vertex.

[0200] In one possible implementation, the first weight data is multiplied by the vertex position difference data to determine the first vertex displacement data required to move the first grid vertex.

[0201] Optionally, the second vertex displacement data corresponding to the second grid vertex is determined based on the second weight data and the first displacement matrix.

[0202] For the preset upper tooth shape, the first displacement matrix includes the upper tooth vertex position difference data between the upper tooth vertex in the face mesh model corresponding to the preset upper tooth shape and the corresponding upper tooth vertex in the original face mesh model. By multiplying the above second weight data with the upper tooth vertex position difference data, the second vertex displacement data required to move the second mesh vertex can be determined.

[0203] Optionally, the third vertex displacement data corresponding to the third grid vertex is determined based on the third weight data and the second displacement matrix.

[0204] For the preset lower tooth shape, the second displacement matrix includes the lower tooth vertex position difference data between the lower tooth vertex in the face mesh model corresponding to the preset lower tooth shape and the corresponding lower tooth vertex in the original face mesh model. By multiplying the above third weight data with the lower tooth vertex position difference data, the third vertex displacement data required to move the third mesh vertex can be determined.

[0205] Step 243: Based on the first vertex displacement data, drive the first mesh vertex to perform displacement to obtain the second face mesh model.

[0206] By using the aforementioned first weight data and the full vertex displacement change data corresponding to the preset tooth shape, the actual vertex displacement data corresponding to the actual tooth shape of the target face can be determined. Then, the mesh vertices in the face mesh model are driven to move according to the actual vertex displacement data, so that the face mesh model can accurately represent the actual tooth shape of the real face, improve the consistency between the tooth shape of the face mesh model and the tooth shape of the real face, and improve the accuracy of tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0207] Optionally, based on the displacement data of the second vertex, the second mesh vertex is driven to perform displacement to obtain the second face mesh model.

[0208] By using the aforementioned second weight data and the full range of upper tooth vertex displacement change data corresponding to the preset upper tooth shape, the actual tooth vertex displacement data corresponding to the actual tooth shape of the target face can be determined. Then, the mesh vertices in the face mesh model are driven to move according to the actual tooth vertex displacement data, so that the face mesh model can accurately represent the actual tooth shape of the real face, improve the consistency between the upper tooth shape of the face mesh model and the upper tooth shape of the real face, and improve the accuracy of the upper tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0209] Optionally, based on the displacement data of the third vertex, the third mesh vertex is driven to perform displacement to obtain the second face mesh model.

[0210] By using the aforementioned third weight data and the full range of lower tooth vertex displacement change data corresponding to the preset lower tooth shape, the actual lower tooth vertex displacement data corresponding to the actual lower tooth shape of the target face can be determined. Then, the mesh vertices in the face mesh model are driven to move according to the actual lower tooth vertex displacement data, so that the face mesh model can accurately represent the actual lower tooth shape of the real face, improving the consistency between the lower tooth shape of the face mesh model and the lower tooth shape of the real face, and improving the accuracy of the lower tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0211] In summary, the technical solution provided by this disclosure can determine the two-dimensional positional difference information of the lips and teeth in the face image by detecting the corresponding two-dimensional positions of the lips and teeth in the target face. Based on the above positional difference information and the three-dimensional position of the lips corresponding to the target face in the first face mesh model, weight data that can characterize the degree of deformation between the tooth shape of the target face and the preset tooth shape can be determined. By driving the mesh vertices associated with the preset tooth shape in the face mesh model according to the weight data, the face mesh model can be made to represent the tooth information of the real face, thereby improving the consistency between the tooth shape of the face mesh model and the tooth shape of the real face, and improving the accuracy of the tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0212] Figure 4 This is a block diagram of a face image processing apparatus according to an exemplary embodiment. (Refer to...) Figure 4 The device 400 includes:

[0213] The location information acquisition module 410 is configured to acquire the first lip position information corresponding to the target face in the first face mesh model, the second lip position information corresponding to the target face in the face image, and the first tooth position information corresponding to the teeth in the target face in the face image.

[0214] The position difference determination module 420 is configured to determine the position difference information between the second lip position information and the first tooth position information;

[0215] The weight determination module 430 is configured to perform a first weight data based on the first lip position information and the position difference information to determine the target face corresponding to a preset tooth shape. The first weight data characterizes the degree of deformation of the tooth shape corresponding to the target face relative to the preset tooth shape.

[0216] The face mesh model driving module 440 is configured to execute, based on the first weight data, drive the first mesh vertex in the face mesh model corresponding to the target face that is associated with the preset tooth shape, to obtain the second face mesh model corresponding to the target face, wherein the first mesh vertex is used to characterize the addition position corresponding to the tooth effect.

[0217] In some possible designs, the face mesh model driving module includes:

[0218] The displacement change information determination unit is configured to determine the vertex displacement change information corresponding to the preset tooth shape, wherein the vertex displacement change information represents the change in the position of the mesh vertex between the face mesh model corresponding to the preset tooth shape and the original face mesh model corresponding to the target face.

[0219] The displacement data determination unit is configured to determine the first vertex displacement data corresponding to the first mesh vertex based on the first weight data and the vertex displacement change information.

[0220] The vertex displacement unit is configured to drive the first mesh vertices to displacement based on the first vertex displacement data to obtain the second face mesh model.

[0221] In some possible designs, the weight determination module includes:

[0222] The tooth position acquisition unit is configured to acquire the second tooth position information in the original face mesh model corresponding to the target face, wherein the second tooth position information refers to the position information of the mesh vertex associated with the tooth in the original face mesh model;

[0223] The displacement change information acquisition unit is configured to acquire vertex displacement change information corresponding to the preset tooth shape;

[0224] The tooth position determination unit is configured to determine third tooth position information based on the first lip position information and the position difference information. The third tooth position information represents the reference position of the tooth in the target three-dimensional coordinate system, which refers to the three-dimensional coordinate system corresponding to the first face mesh model.

[0225] The weight data determination unit is configured to perform constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints to obtain the first weight data.

[0226] The preset constraint condition refers to the condition that constrains the target position error to reach the minimum value. The target position error is the position error between the third tooth position information and the predicted tooth position information. The predicted tooth position information is the tooth position information after offsetting the second tooth position information based on the target vertex displacement information. The target vertex displacement information is obtained by weighting the vertex displacement change information based on the first weight data. The first weight data is determined when the target position error reaches the minimum value.

[0227] In some possible designs, the first lip position information includes the first upper lip position data corresponding to the target face in the first face mesh model and the first lower lip position data corresponding to the target face in the first face mesh model. The position difference information includes first position difference data and second position difference data. The first position difference data represents the position difference between the upper lip position and the upper teeth position in the face image, and the second position difference data represents the position difference between the lower lip position and the lower teeth position in the face image.

[0228] The tooth position determination unit includes:

[0229] The reference upper tooth position determination subunit is configured to perform the determination of reference upper tooth position data corresponding to the tooth in the target three-dimensional coordinate system based on the first upper lip position data and the first position difference data.

[0230] The reference mandibular position determination subunit is configured to perform a process based on the first lower lip position data and the second position difference data to determine the reference mandibular position data corresponding to the tooth in the target three-dimensional coordinate system. The third tooth position information includes the reference upper tooth position data and the reference mandibular position data.

[0231] In some possible designs, the teeth include upper teeth, the preset tooth shape includes a preset upper tooth shape, the second tooth position information includes the original upper tooth position data corresponding to the upper tooth vertex associated with the upper tooth in the original face mesh model, and the vertex displacement change information includes a first displacement matrix corresponding to the upper tooth vertex. The first displacement matrix represents the vertex displacement change relationship between the upper tooth vertex in the original face mesh model and the upper tooth vertex in the face mesh model corresponding to the preset upper tooth shape.

[0232] The weight data determination unit is specifically configured to perform constraint analysis on the upper tooth position error corresponding to the reference upper tooth position data based on the preset constraint conditions, and obtain the second weight data of the target face corresponding to the preset upper tooth shape.

[0233] The target position error includes the upper tooth position error, which is the error between the reference upper tooth position data and the predicted upper tooth position data. The predicted upper tooth position data is the upper tooth position data after offsetting the original upper tooth position data based on the second vertex displacement data. The target vertex displacement information includes the second vertex displacement data, which is obtained by weighting the first displacement matrix based on the second weight data.

[0234] In some possible designs, the face mesh model driving module is specifically configured to execute, based on the second weight data, drive the second mesh vertices in the face mesh model corresponding to the target face that are associated with the preset upper teeth shape, to obtain the second face mesh model, wherein the first mesh vertex includes the second mesh vertex.

[0235] In some possible designs, the displacement data determining unit is specifically configured to determine the second vertex displacement data corresponding to the second mesh vertex based on the second weight data and the first displacement matrix;

[0236] The vertex displacement unit is specifically configured to drive the second mesh vertices to displacement based on the second vertex displacement data, thereby obtaining the second face mesh model.

[0237] In some possible designs, the teeth also include lower teeth, the preset tooth shape also includes a preset lower tooth shape, the second tooth position information also includes the original lower tooth position data corresponding to the lower tooth vertex associated with the lower tooth in the original face mesh model, the vertex displacement change information also includes the second displacement matrix corresponding to the lower tooth vertex, the second displacement matrix characterizes the vertex displacement change relationship between the lower tooth vertex in the original face mesh model and the lower tooth vertex in the face mesh model corresponding to the preset lower tooth shape;

[0238] The weight data determination unit is further configured to perform constraint analysis on the lower tooth position error corresponding to the reference lower tooth position data based on the preset constraint conditions, and obtain the third weight data of the target face corresponding to the preset lower tooth shape.

[0239] The target position error includes the mandibular position error, which is the error between the reference mandibular position data and the predicted mandibular position data. The predicted mandibular position data is the mandibular position data after offsetting the original mandibular position data based on the third vertex displacement data. The target vertex displacement information includes the third vertex displacement data, which is obtained by weighting the second displacement matrix based on the third weight data.

[0240] In some possible designs, the face mesh model driving module is further configured to execute, based on the third weight data, drive the third mesh vertex in the face mesh model corresponding to the target face that is associated with the preset lower tooth shape, to obtain the second face mesh model, wherein the first mesh vertex includes the third mesh vertex.

[0241] In some possible designs, the displacement data determining unit is further configured to perform the determination of the third vertex displacement data corresponding to the third mesh vertex based on the third weight data and the second displacement matrix;

[0242] The vertex displacement unit is further configured to drive the third mesh vertex to displacement based on the third vertex displacement data, thereby obtaining the second face mesh model.

[0243] In some possible designs, the second lip position information includes the second upper lip position data corresponding to the target face in the face image and the second lower lip position data corresponding to the target face in the face image, and the first tooth position information includes the first upper tooth position data corresponding to the target face in the face image and the first lower tooth position data corresponding to the target face in the face image.

[0244] The location difference determination module includes:

[0245] The first position difference determination unit is configured to determine a first position difference data between the second upper lip position data and the first upper tooth position data;

[0246] The second position difference determination unit is configured to determine a second position difference data between the second lower lip position data and the first lower tooth position data, wherein the position difference information includes the first position difference data and the first position difference data.

[0247] In some possible designs, the location information acquisition module includes:

[0248] The expression weight determination unit is configured to perform expression detection processing on the face image to obtain fourth weight data of the target face corresponding to a preset expression form, wherein the fourth weight data characterizes the degree of deformation of the expression form corresponding to the target face relative to the preset expression form.

[0249] The face mesh model driving module is also configured to execute, based on the fourth weight data, drive the mesh vertices in the original face mesh model corresponding to the target face to obtain the first face mesh model, wherein the first face mesh model includes the first lip position information;

[0250] The mouth detection unit is configured to perform mouth detection processing on the face image to obtain the second lip position information and the first tooth position information.

[0251] In some possible designs, the face image is obtained by capturing the target face using a monocular camera.

[0252] In summary, the technical solution provided by this disclosure can determine the two-dimensional positional difference information of the lips and teeth in the face image by detecting the corresponding two-dimensional positions of the lips and teeth in the target face. Based on the above positional difference information and the three-dimensional position of the lips corresponding to the target face in the first face mesh model, weight data that can characterize the degree of deformation between the tooth shape of the target face and the preset tooth shape can be determined. By driving the mesh vertices associated with the preset tooth shape in the face mesh model according to the weight data, the face mesh model can be made to represent the tooth information of the real face, thereby improving the consistency between the tooth shape of the face mesh model and the tooth shape of the real face, and improving the accuracy of the tooth information in the face mesh model and the realism of the face mesh model as a whole.

[0253] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0254] Figure 5 This is a block diagram illustrating an electronic device for face image processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a facial image processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0255] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0256] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the face image processing method as described in the embodiments of this disclosure.

[0257] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the face image processing method of the present disclosure embodiments.

[0258] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the face image processing method of the present disclosure embodiments.

[0259] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0260] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0261] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A face image processing method, characterized in that, The method includes: The first lip position information corresponding to the target face in the first face mesh model, the second lip position information corresponding to the target face in the face image, and the first tooth position information corresponding to the teeth in the target face in the face image are obtained. Determine the positional difference information between the second lip position information and the first tooth position information; Obtain the second tooth position information in the original face mesh model corresponding to the target face. The second tooth position information refers to the position information of the mesh vertex associated with the tooth in the original face mesh model. Obtain the vertex displacement change information corresponding to the preset tooth shape; Based on the first lip position information and the position difference information, the third tooth position information is determined, and the third tooth position information represents the reference position of the tooth in the three-dimensional coordinate system corresponding to the first face mesh model. Based on preset constraints, the target position error corresponding to the third tooth position information is subjected to constraint analysis to obtain the first weight data; The preset constraint condition refers to the condition that the positional error between the third tooth position information and the predicted tooth position information reaches the minimum value. The predicted tooth position information is the tooth position information after offsetting the second tooth position information based on the target vertex displacement information. The first weight data represents the degree of deformation of the tooth shape corresponding to the target face relative to the preset tooth shape. Based on the first weight data, the first grid vertex in the face mesh model corresponding to the target face, which is associated with the preset tooth shape, is driven to obtain the second face mesh model corresponding to the target face. The first grid vertex is used to represent the addition position corresponding to the tooth effect.

2. The method according to claim 1, characterized in that, The step of driving the first mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset tooth shape, based on the first weight data, to obtain the second face mesh model corresponding to the target face includes: Determine the vertex displacement change information corresponding to the preset tooth shape, wherein the vertex displacement change information characterizes the change in the position of the mesh vertex between the face mesh model corresponding to the preset tooth shape and the original face mesh model corresponding to the target face; Based on the first weight data and the vertex displacement change information, determine the first vertex displacement data corresponding to the first grid vertex; Based on the first vertex displacement data, the first mesh vertex is driven to move, thereby obtaining the second face mesh model.

3. The method according to claim 1, characterized in that, The first lip position information includes the first upper lip position data corresponding to the target face in the first face mesh model and the first lower lip position data corresponding to the target face in the first face mesh model. The position difference information includes first position difference data and second position difference data. The first position difference data represents the position difference between the upper lip position and the upper teeth position in the face image, and the second position difference data represents the position difference between the lower lip position and the lower teeth position in the face image. The step of determining the position information of the third tooth based on the first lip position information and the position difference information includes: Based on the first upper lip position data and the first position difference data, the reference upper tooth position data corresponding to the tooth in the target three-dimensional coordinate system is determined; Based on the first lower lip position data and the second position difference data, the reference lower tooth position data corresponding to the tooth in the target three-dimensional coordinate system is determined, and the third tooth position information includes the reference upper tooth position data and the reference lower tooth position data.

4. The method according to claim 3, characterized in that, The teeth include upper teeth, the preset tooth shape includes a preset upper tooth shape, the second tooth position information includes the original upper tooth position data corresponding to the upper tooth vertex associated with the upper tooth in the original face mesh model, and the vertex displacement change information includes a first displacement matrix corresponding to the upper tooth vertex. The first displacement matrix represents the vertex displacement change relationship between the upper tooth vertex in the original face mesh model and the upper tooth vertex in the face mesh model corresponding to the preset upper tooth shape. The first weight data is obtained by performing constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints, including: Based on the preset constraints, the upper tooth position error corresponding to the reference upper tooth position data is constrained and analyzed to obtain the second weight data of the target face corresponding to the preset upper tooth shape. The target position error includes the upper tooth position error, which is the error between the reference upper tooth position data and the predicted upper tooth position data. The predicted upper tooth position data is the upper tooth position data after offsetting the original upper tooth position data based on the second vertex displacement data. The target vertex displacement information includes the second vertex displacement data, which is obtained by weighting the first displacement matrix based on the second weight data.

5. The method according to claim 4, characterized in that, The step of driving the first mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset tooth shape, based on the first weight data, to obtain the second face mesh model corresponding to the target face includes: Based on the second weight data, the second mesh vertex in the face mesh model corresponding to the target face, which is associated with the preset upper teeth shape, is driven to obtain the second face mesh model, wherein the first mesh vertex includes the second mesh vertex.

6. The method according to claim 5, characterized in that, The step of driving the second mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset upper teeth shape, based on the second weight data, to obtain the second face mesh model includes: Based on the second weight data and the first displacement matrix, determine the second vertex displacement data corresponding to the second grid vertex; Based on the displacement data of the second vertex, the second grid vertex is driven to move, thereby obtaining the second face grid model.

7. The method according to claim 3, characterized in that, The teeth include lower teeth, the preset tooth shape also includes a preset lower tooth shape, the second tooth position information also includes the original lower tooth position data corresponding to the lower tooth vertex associated with the lower tooth in the original face mesh model, the vertex displacement change information also includes the second displacement matrix corresponding to the lower tooth vertex, the second displacement matrix characterizes the vertex displacement change relationship between the lower tooth vertex in the original face mesh model and the lower tooth vertex in the face mesh model corresponding to the preset lower tooth shape; The first weight data is obtained by performing constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints, including: Based on the preset constraints, the lower tooth position error corresponding to the reference lower tooth position data is constrained and analyzed to obtain the third weight data of the target face corresponding to the preset lower tooth shape. The target position error includes the mandibular position error, which is the error between the reference mandibular position data and the predicted mandibular position data. The predicted mandibular position data is the mandibular position data after offsetting the original mandibular position data based on the third vertex displacement data. The target vertex displacement information includes the third vertex displacement data, which is obtained by weighting the second displacement matrix based on the third weight data.

8. The method according to claim 7, characterized in that, The step of driving the first mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset tooth shape, based on the first weight data, to obtain the second face mesh model corresponding to the target face includes: Based on the third weight data, the third grid vertex in the face grid model corresponding to the target face, which is associated with the preset lower tooth shape, is driven to obtain the second face grid model, wherein the first grid vertex includes the third grid vertex.

9. The method according to claim 8, characterized in that, The step of driving the third mesh vertices in the face mesh model corresponding to the target face, which are associated with the preset lower tooth shape, based on the third weight data, to obtain the second face mesh model includes: Based on the third weight data and the second displacement matrix, the displacement data of the third vertex corresponding to the third grid vertex is determined; Based on the displacement data of the third vertex, the third grid vertex is driven to move, thereby obtaining the second face grid model.

10. The method according to any one of claims 1 to 9, characterized in that, The second lip position information includes the second upper lip position data corresponding to the target face in the face image and the second lower lip position data corresponding to the target face in the face image; the first tooth position information includes the first upper tooth position data corresponding to the target face in the face image and the first lower tooth position data corresponding to the target face in the face image. The determination of the positional difference information between the second lip position information and the first tooth position information includes: Determine the first positional difference data between the second upper lip position data and the first upper tooth position data; A second positional difference data is determined between the second lower lip position data and the first lower tooth position data, wherein the positional difference information includes the first positional difference data and the first positional difference data.

11. The method according to any one of claims 1 to 9, characterized in that, The step of acquiring the first lip position information corresponding to the target face in the first face mesh model, the second lip position information corresponding to the target face in the face image, and the first tooth position information corresponding to the teeth in the target face in the face image includes: The facial image is subjected to expression detection processing to obtain fourth weight data of the target face corresponding to a preset expression shape. The fourth weight data represents the degree of deformation of the expression shape of the target face relative to the preset expression shape. Based on the fourth weight data, the grid vertices in the original face mesh model corresponding to the target face are driven to obtain the first face mesh model, which includes the first lip position information; The face image is processed by mouth detection to obtain the second lip position information and the first tooth position information.

12. A face image processing device, characterized in that, The device includes: The location information acquisition module is configured to acquire the first lip position information corresponding to the target face in the first face mesh model, the second lip position information corresponding to the target face in the face image, and the first tooth position information corresponding to the teeth in the target face in the face image. The position difference determination module is configured to determine the position difference information between the second lip position information and the first tooth position information; The tooth position acquisition unit is configured to acquire the second tooth position information in the original face mesh model corresponding to the target face, wherein the second tooth position information refers to the position information of the mesh vertex associated with the tooth in the original face mesh model; The displacement change information acquisition unit is configured to acquire vertex displacement change information corresponding to the preset tooth shape; The tooth position determination unit is configured to determine third tooth position information based on the first lip position information and the position difference information, wherein the third tooth position information represents the reference position of the tooth in the three-dimensional coordinate system corresponding to the first face mesh model. The weight data determination unit is configured to perform constraint analysis on the target position error corresponding to the third tooth position information based on preset constraints to obtain first weight data. The preset constraint condition refers to the condition that the positional error between the third tooth position information and the predicted tooth position information reaches the minimum value. The predicted tooth position information is the tooth position information after offsetting the second tooth position information based on the target vertex displacement information. The first weight data represents the degree of deformation of the tooth shape corresponding to the target face relative to the preset tooth shape. The face mesh model driving module is configured to execute, based on the first weight data, drive the first mesh vertex in the face mesh model corresponding to the target face that is associated with the preset tooth shape, to obtain the second face mesh model corresponding to the target face, wherein the first mesh vertex is used to characterize the addition position corresponding to the tooth effect.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the face image processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the face image processing method as described in any one of claims 1 to 11.

15. A computer program product comprising instructions that, when executed on a computer, cause the computer to perform the face image processing method as described in any one of claims 1 to 11.

Images