Face image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a face image processing method and device, electronic equipment and storage medium. The method comprises: obtaining spatial transformation information, face pose change information corresponding to a face image, tooth position information in the face image corresponding to teeth in the face image, and initial tooth pose information in a three-dimensional coordinate system corresponding to a three-dimensional tooth model corresponding to the teeth; based on a first preset constraint condition, the target position error corresponding to the tooth position information is analyzed to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model; and based on the first tooth pose change information and the face pose change information, the initial tooth pose information is offset to obtain the target tooth pose information. The present disclosure can realize the reconstruction of the three-dimensional face to represent the real tooth information of the face.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for processing facial images. Background Technology

[0002] With the continuous development of computer vision, computer graphics, and artificial intelligence technologies, facial image effects have been widely used, such as virtual humans and facial expression effects.

[0003] In related technologies, computer equipment detects input facial images and reconstructs a 3D face based on the detection results, thereby enabling the addition of facial expression effects. For example, in short videos and live streaming scenarios, it can add relevant headwear and other effects to the broadcaster.

[0004] However, the reconstructed 3D faces in these technologies lack teeth information. Summary of the Invention

[0005] This disclosure provides a face image processing method, apparatus, electronic device, and storage medium to at least solve the problem of missing tooth information in reconstructed three-dimensional faces in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a face image processing method is provided, the method comprising:

[0007] The system acquires spatial transformation information, facial pose change information corresponding to a facial image, tooth position information corresponding to teeth in the facial image, and initial tooth pose information of the three-dimensional tooth model corresponding to the teeth in a three-dimensional coordinate system. The spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system corresponding to the facial image, and the facial pose change information represents the pose change of the three-dimensional facial model corresponding to the facial image in the three-dimensional coordinate system.

[0008] Based on the first preset constraint condition, a constraint analysis is performed on the target position error corresponding to the tooth position information to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model; wherein, the first preset constraint condition refers to the condition that constrains the target position error to reach the minimum value, the target position error is the error between the tooth position information and the first predicted tooth position information, the first predicted tooth position information is the two-dimensional tooth position information corresponding to the first tooth pose information in the three-dimensional coordinate system after transforming the first tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the first tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information, and the first tooth pose change information is determined under the condition that the target position error reaches the minimum value;

[0009] Based on the first tooth pose change information and the face pose change information, the initial tooth pose information is offset to obtain the target tooth pose information, which is used to characterize the addition position corresponding to the tooth effect.

[0010] In some possible designs, the teeth include upper teeth, the three-dimensional tooth model includes the maxillary tooth model corresponding to the upper teeth, the initial tooth pose information includes initial upper tooth pose information, the tooth position information includes upper tooth position information, and the constraint analysis is performed on the target position error corresponding to the tooth position information based on the first preset constraint condition to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model, including:

[0011] Based on the first preset constraint, constraint analysis is performed on the upper tooth position error corresponding to the upper tooth position information to obtain the maxillary bone pose change information of the maxillary tooth model relative to the three-dimensional face model.

[0012] Wherein, the first tooth pose change information includes the maxillary pose change information, the target position error includes the upper tooth position error, the upper tooth position error is the error between the upper tooth position information and the predicted upper tooth position information, the predicted upper tooth position information is the two-dimensional upper tooth position information corresponding to the target upper tooth pose information in the three-dimensional coordinate system after transforming the target upper tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the target upper tooth pose information is the upper tooth pose information after offsetting the initial upper tooth pose information based on the maxillary pose change information and the face pose change information, and the maxillary pose change information is determined when the upper tooth position error reaches the minimum value.

[0013] In some possible designs, the maxillary pose change information includes maxillary rotation information and maxillary displacement information of the maxillary tooth model relative to the three-dimensional face model. The step of offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information includes:

[0014] Based on the facial pose change information, the maxillary bone rotation information, and the maxillary bone displacement information, the initial maxillary tooth pose information is offset to obtain the target maxillary tooth pose information.

[0015] In some possible designs, the teeth also include lower teeth, the three-dimensional tooth model also includes a mandibular tooth model corresponding to the lower teeth, the initial tooth pose information also includes initial lower tooth pose information, the tooth position information also includes lower tooth position information, and the step of performing constraint analysis on the target position error corresponding to the tooth position information based on a first preset constraint condition to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model includes:

[0016] Based on the first preset constraint, the lower tooth position error corresponding to the lower tooth position information is constrained and analyzed to obtain the mandibular pose change information of the mandibular tooth model relative to the three-dimensional face model.

[0017] Wherein, the first tooth pose change information includes the mandibular pose change information, the target position error includes the mandibular position error, the mandibular position error is the error between the mandibular position information and the predicted mandibular position information, the predicted mandibular position information is the two-dimensional mandibular position information corresponding to the target mandibular pose information in the three-dimensional coordinate system after transforming the target mandibular pose information of the mandibular tooth into the two-dimensional coordinate system based on the spatial transformation information, the target mandibular pose information is the mandibular pose information after offsetting the initial mandibular pose information based on the mandibular pose change information and the face pose change information, and the mandibular pose change information is determined when the mandibular position error reaches the minimum value.

[0018] In some possible designs, the mandibular pose change information includes mandibular rotation information and mandibular displacement information of the mandibular tooth model relative to the three-dimensional face model. The step of offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information includes:

[0019] Based on the facial pose change information, the mandibular rotation information, and the mandibular displacement information, the initial mandibular pose information is offset to obtain the target mandibular pose information.

[0020] In some possible designs, the method further includes:

[0021] Obtain the tooth pose variation range information corresponding to the three-dimensional tooth model, and the tooth pose variation range information is used to characterize the pose variation range of the three-dimensional tooth model relative to the three-dimensional face model;

[0022] The step of performing constraint analysis on the target position error corresponding to the tooth position information based on the first preset constraint condition to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model includes:

[0023] Based on the second preset constraint condition, the first fusion position error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth pose change information.

[0024] The second preset constraint condition refers to the condition that constrains the first fusion position error to reach the minimum value. The fusion position error is obtained by fusing the target position error and the pose change range deviation. The pose change range deviation refers to the deviation between the first tooth pose change information and the tooth pose change range information. The first tooth pose change information is determined when the first fusion position error reaches the minimum value.

[0025] In some possible designs, the method further includes:

[0026] Determine the previous face image corresponding to the given face image;

[0027] Obtain the second tooth pose change information corresponding to the previous face image;

[0028] The first fusion position error corresponding to the tooth position information is constrained and analyzed based on the second preset constraint condition to obtain the first tooth pose change information, including:

[0029] Based on the third preset constraint condition, the second fusion position error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth pose change information.

[0030] The third preset constraint condition refers to the condition that constrains the second fusion position error to reach the minimum value. The second fusion position error is obtained by fusing the first fusion position error and the pose motion range deviation. The pose motion range deviation refers to the deviation between the first tooth pose change information and the second tooth pose change information. The first tooth pose change information is determined when the second fusion position error reaches the minimum value.

[0031] In some possible designs, before shifting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information, the method further includes:

[0032] Based on the fourth preset constraint condition, the positional morphology error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth morphology change information corresponding to the tooth; wherein, the fourth preset constraint condition refers to the condition that constrains the positional morphology error to reach the minimum value, the first tooth morphology change information represents the morphological change between the tooth morphology and the tooth morphology corresponding to the tooth in the three-dimensional tooth model, the positional morphology error is the error between the tooth position information and the second predicted tooth position information, the second predicted tooth position information is the two-dimensional tooth position information corresponding to the second tooth pose information corresponding to the tooth in the three-dimensional coordinate system after transforming it to the two-dimensional coordinate system based on the spatial transformation information, the second tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the face pose change information, the first tooth pose change information and the first tooth morphology change information, and the first tooth morphology change information is determined under the condition that the positional morphology error reaches the minimum value;

[0033] The step of shifting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information includes:

[0034] Based on the facial pose change information, the first tooth pose change information, and the first tooth morphology change information, the initial tooth pose information is offset to obtain the target tooth pose information.

[0035] In some possible designs, the method further includes:

[0036] Determine the previous face image corresponding to the given face image;

[0037] Obtain the second tooth morphology change information corresponding to the previous face image;

[0038] The step of performing constraint analysis on the positional morphology error corresponding to the tooth position information based on the fourth preset constraint condition to obtain the first tooth morphology change information corresponding to the tooth includes:

[0039] Based on the fifth preset constraint condition, the position morphology fusion error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth morphology change information.

[0040] The fifth preset constraint condition refers to the condition that constrains the positional morphology fusion error to reach the minimum value. The positional morphology fusion error is obtained by fusing the positional morphology error and the morphological change range deviation. The morphological change range deviation refers to the deviation between the first tooth morphological change information and the second tooth morphological change information. The first tooth morphological change information is determined when the positional morphology fusion error reaches the minimum value.

[0041] According to a second aspect of the present disclosure, a face image processing apparatus is provided, the apparatus comprising:

[0042] The target information acquisition module is configured to acquire spatial transformation information, facial pose change information corresponding to a facial image, tooth position information of the teeth in the facial image, and initial tooth pose information of the three-dimensional tooth model corresponding to the teeth in the three-dimensional coordinate system. The spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system corresponding to the facial image, and the facial pose change information represents the pose change of the three-dimensional facial model corresponding to the facial image in the three-dimensional coordinate system.

[0043] The pose change determination module is configured to perform constraint analysis on the target position error corresponding to the tooth position information based on a first preset constraint condition, to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model; wherein, the first preset constraint condition refers to the condition that constrains the target position error to reach a minimum value, the target position error is the error between the tooth position information and the first predicted tooth position information, the first predicted tooth position information is the two-dimensional tooth position information corresponding to the first tooth pose information in the three-dimensional coordinate system after transforming the first tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the first tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information, and the first tooth pose change information is determined under the condition that the target position error reaches a minimum value;

[0044] The tooth pose determination module is configured to perform an offset on the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain target tooth pose information, wherein the target tooth pose information is used to characterize the addition position corresponding to the tooth effect.

[0045] In some possible designs, the teeth include upper teeth, the three-dimensional tooth model includes the maxillary tooth model corresponding to the upper teeth, the initial tooth pose information includes initial upper tooth pose information, the tooth position information includes upper tooth position information, and the pose change determination module is specifically configured to perform constraint analysis on the upper tooth position error corresponding to the upper tooth position information based on the first preset constraint conditions, so as to obtain the maxillary pose change information of the maxillary tooth model relative to the three-dimensional face model;

[0046] Wherein, the first tooth pose change information includes the maxillary pose change information, the target position error includes the upper tooth position error, the upper tooth position error is the error between the upper tooth position information and the predicted upper tooth position information, the predicted upper tooth position information is the two-dimensional upper tooth position information corresponding to the target upper tooth pose information in the three-dimensional coordinate system after transforming the target upper tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the target upper tooth pose information is the upper tooth pose information after offsetting the initial upper tooth pose information based on the maxillary pose change information and the face pose change information, and the maxillary pose change information is determined when the upper tooth position error reaches the minimum value.

[0047] In some possible designs, the maxillary pose change information includes maxillary rotation information and maxillary displacement information of the maxillary tooth model relative to the three-dimensional face model. The tooth pose determination module is specifically configured to perform offsetting of the initial maxillary pose information based on the face pose change information, the maxillary rotation information, and the maxillary displacement information to obtain the target maxillary pose information.

[0048] In some possible designs, the teeth also include lower teeth, the three-dimensional tooth model also includes a mandibular tooth model corresponding to the lower teeth, the initial tooth pose information also includes initial lower tooth pose information, the tooth position information also includes lower tooth position information, and the pose change determination module is specifically configured to perform constraint analysis on the lower tooth position error corresponding to the lower tooth position information based on the first preset constraint conditions, so as to obtain the mandibular pose change information of the mandibular tooth model relative to the three-dimensional face model;

[0049] Wherein, the first tooth pose change information includes the mandibular pose change information, the target position error includes the mandibular position error, the mandibular position error is the error between the mandibular position information and the predicted mandibular position information, the predicted mandibular position information is the two-dimensional mandibular position information corresponding to the target mandibular pose information in the three-dimensional coordinate system after transforming the target mandibular pose information of the mandibular tooth into the two-dimensional coordinate system based on the spatial transformation information, the target mandibular pose information is the mandibular pose information after offsetting the initial mandibular pose information based on the mandibular pose change information and the face pose change information, and the mandibular pose change information is determined when the mandibular position error reaches the minimum value.

[0050] In some possible designs, the mandibular pose change information includes mandibular rotation information and mandibular displacement information of the mandibular tooth model relative to the three-dimensional face model. The tooth pose determination module is further configured to perform offsetting of the initial mandibular pose information based on the face pose change information, the mandibular rotation information, and the mandibular displacement information to obtain the target mandibular pose information.

[0051] In some possible designs, the device further includes:

[0052] The pose change range acquisition module is configured to acquire the pose change range information of the teeth corresponding to the three-dimensional tooth model. The tooth pose change range information is used to characterize the pose change range of the three-dimensional tooth model relative to the three-dimensional face model.

[0053] The pose change determination module is further configured to perform constraint analysis on the first fusion position error corresponding to the tooth position information based on the second preset constraint conditions, so as to obtain the first tooth pose change information.

[0054] The second preset constraint condition refers to the condition that constrains the first fusion position error to reach the minimum value. The fusion position error is obtained by fusing the target position error and the pose change range deviation. The pose change range deviation refers to the deviation between the first tooth pose change information and the tooth pose change range information. The first tooth pose change information is determined when the first fusion position error reaches the minimum value.

[0055] In some possible designs, the device further includes:

[0056] The face image determination module is configured to determine the previous face image corresponding to the face image;

[0057] The target information acquisition module is also configured to acquire the second tooth pose change information corresponding to the previous face image;

[0058] The pose change determination module is further configured to perform constraint analysis on the second fusion position error corresponding to the tooth position information based on a third preset constraint condition, so as to obtain the first tooth pose change information.

[0059] The third preset constraint condition refers to the condition that constrains the second fusion position error to reach the minimum value. The second fusion position error is obtained by fusing the first fusion position error and the pose motion range deviation. The pose motion range deviation refers to the deviation between the first tooth pose change information and the second tooth pose change information. The first tooth pose change information is determined when the second fusion position error reaches the minimum value.

[0060] In some possible designs, the device further includes:

[0061] The morphological change determination module is configured to perform constraint analysis on the positional morphological error corresponding to the tooth position information based on a fourth preset constraint condition, to obtain the first tooth morphological change information corresponding to the tooth; wherein, the fourth preset constraint condition refers to the condition that constrains the positional morphological error to reach a minimum value, the first tooth morphological change information represents the morphological change between the tooth morphology and the tooth morphology corresponding to the tooth in the three-dimensional tooth model, the positional morphological error is the error between the tooth position information and the second predicted tooth position information, the second predicted tooth position information is the two-dimensional tooth position information corresponding to the second tooth pose information corresponding to the tooth in the three-dimensional coordinate system after transforming it to the two-dimensional coordinate system based on the spatial transformation information, the second tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the face pose change information, the first tooth pose change information and the first tooth morphological change information, and the first tooth morphological change information is determined under the condition that the positional morphological error reaches a minimum value;

[0062] The tooth pose determination module is further configured to perform an offset on the initial tooth pose information based on the face pose change information, the first tooth pose change information, and the first tooth morphology change information to obtain the target tooth pose information.

[0063] In some possible designs, the device further includes:

[0064] The face image determination module is configured to determine the previous face image corresponding to the face image;

[0065] The target information acquisition module is also configured to acquire the second tooth morphology change information corresponding to the previous face image;

[0066] The morphological change determination module is specifically configured to perform constraint analysis on the positional morphological fusion error corresponding to the tooth position information based on the fifth preset constraint condition, so as to obtain the first tooth morphological change information.

[0067] The fifth preset constraint condition refers to the condition that constrains the positional morphology fusion error to reach the minimum value. The positional morphology fusion error is obtained by fusing the positional morphology error and the morphological change range deviation. The morphological change range deviation refers to the deviation between the first tooth morphological change information and the second tooth morphological change information. The first tooth morphological change information is determined when the positional morphology fusion error reaches the minimum value.

[0068] In some possible designs, the face image is obtained by capturing the target face using a monocular camera.

[0069] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the face image processing method as described in any one of the first aspects above.

[0070] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the face image processing method described in any one of the first aspects of the present disclosure.

[0071] According to a fifth aspect of the present disclosure, a computer program product including instructions is provided, which, when run on a computer, causes the computer to perform the face image processing method described in any one of the first aspects of the present disclosure.

[0072] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0073] By minimizing the positional error between the actual and predicted positions of the teeth through preset constraints, the corresponding tooth pose change can be determined when the positional error is minimized. Then, based on the tooth pose change and the initial tooth pose, the target pose of the teeth can be determined, so that the reconstructed 3D face can represent the tooth information of the real face. This improves the consistency between the tooth pose of the 3D face and the tooth pose of the real face, and improves the accuracy of tooth information and the realism of the 3D face overall.

[0074] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0075] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0076] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment;

[0077] Figure 2 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 1 ;

[0078] Figure 3(a) illustrates a schematic diagram of a three-dimensional skull model with the mouth closed;

[0079] Figure 3(b) illustrates a schematic diagram of a three-dimensional skull model with the mouth open;

[0080] Figure 4 An exemplary schematic diagram of a three-dimensional coordinate system is shown;

[0081] Figure 5 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 2 ;

[0082] Figure 6 This is flowchart three illustrating a face image processing method according to an exemplary embodiment;

[0083] Figure 7 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 4 ;

[0084] Figure 8 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 5 ;

[0085] Figure 9 This is a block diagram of a face image processing apparatus according to an exemplary embodiment;

[0086] Figure 10 This is a block diagram illustrating an electronic device for face image processing according to an exemplary embodiment. Detailed Implementation

[0087] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0088] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0089] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0090] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include terminal 100 and server 200.

[0091] Terminal 100 can be used to provide facial image processing services to any user. Specifically, terminal 100 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, or software running on the aforementioned electronic devices, such as applications. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0092] In an optional embodiment, server 200 can provide background services to terminal 100. Specifically, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0093] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, other application environments may also be included, such as more terminals.

[0094] In the embodiments described in this specification, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication, and this disclosure does not impose any restrictions.

[0095] Figure 2 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 1 Optionally, this face image processing method is used in an electronic device. Optionally, the electronic device can be a terminal or a server. Figure 2 As shown, the method may include the following steps (210-230).

[0096] Step 210: Obtain spatial transformation information, facial pose change information corresponding to the facial image, tooth position information corresponding to the teeth in the facial image, and initial tooth pose information corresponding to the three-dimensional tooth model in the three-dimensional coordinate system.

[0097] Specifically, spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system corresponding to the face image, while face pose change information represents the pose change of the three-dimensional face model corresponding to the face image in the three-dimensional coordinate system. Optionally, the three-dimensional face model can be a point cloud model or a skeletal animation model.

[0098] Optionally, a face image and the corresponding spatial transformation information can be obtained.

[0099] Optionally, the aforementioned face image is obtained by capturing the target face using a monocular camera. The face image captured by the monocular camera is a two-dimensional image (non-depth image), not a depth image with three-dimensional depth feature information.

[0100] Related technologies require depth images containing facial depth information for face modeling, and these technologies are highly dependent on the presence of 3D depth features related to the face in the depth images. In contrast, the technical solution provided in this disclosure can achieve 3D face modeling and driving based on 2D face images without depth information captured by a monocular camera, and can represent the teeth information of a realistic face. This reduces the requirements for camera equipment in 3D face modeling and driving scenarios, allowing devices without a depth camera to perform 3D face modeling and driving.

[0101] Optionally, the spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system corresponding to the face image and the three-dimensional coordinate system of the three-dimensional face model corresponding to the face image. This spatial transformation information can be a coordinate system transformation matrix between the two-dimensional and three-dimensional coordinate systems, or a coordinate system transformation function between the two-dimensional and three-dimensional coordinate systems.

[0102] Specifically, the aforementioned spatial transformation information is the camera intrinsic parameter matrix of the camera device corresponding to the aforementioned face image. The device can obtain the aforementioned camera intrinsic parameter matrix from the configuration information corresponding to the camera device.

[0103] Optionally, face detection processing is performed on the above face image to obtain face pose change information corresponding to the face image, and tooth position information corresponding to the teeth in the face image.

[0104] Optionally, the face pose change information represents the pose change of the 3D face model in the 3D coordinate system. Optionally, the face pose change information includes a face pose change matrix. Optionally, the face pose change matrix includes the rotation data and displacement data of the 3D face model in the 3D coordinate system.

[0105] The 3D face model has an initial position in the aforementioned 3D coordinate system. During face modeling, the 3D face model rotates and translates within the 3D coordinate system to reach the target position, ensuring that the pose of the 3D face model at the target position matches the pose of the target face in the face image. The aforementioned face pose change information characterizes the pose change of the 3D face model at the target position relative to its initial position. Accordingly, the face pose change information includes rotation and translation data of the 3D face model from its initial position to its target position in the aforementioned 3D coordinate system. Specifically, the face pose change information can be a face pose change matrix.

[0106] Optionally, the teeth mentioned above include at least one of upper teeth and lower teeth. Accordingly, the tooth position information mentioned above includes at least one of upper tooth position information and lower tooth position information.

[0107] Optionally, the aforementioned tooth location information includes two-dimensional tooth location data corresponding to the tooth's key points in the face image. Optionally, the aforementioned tooth key points correspond to a three-dimensional tooth model.

[0108] Optionally, teeth in a face image can be represented by dental keypoints. For example, each tooth corresponds to a tooth vertex and a tooth center point, and adjacent teeth can reuse the tooth vertices between adjacent teeth. The face detection described above can detect exposed teeth in a face image and obtain the two-dimensional position data of the dental keypoints corresponding to the exposed teeth in the face image.

[0109] In one possible implementation, the face image is input into a trained convolutional neural network model or object detection model to perform two-dimensional key point detection, thereby obtaining two-dimensional position data (such as two-dimensional coordinate data) corresponding to preset face key points and two-dimensional position data (such as two-dimensional coordinate data) corresponding to tooth key points; based on the two-dimensional position data corresponding to the preset face key points, the face pose change matrix can be determined.

[0110] In another possible implementation, the two-dimensional position data corresponding to the above-mentioned facial key points can be mapped to a three-dimensional coordinate system through a coordinate system transformation relationship matrix to obtain a three-dimensional facial model corresponding to the current facial image. By comparing the position difference between the three-dimensional facial model corresponding to the current facial image and the three-dimensional facial model at the initial position, the above-mentioned facial pose change matrix can be obtained.

[0111] Optionally, 51 tooth key points are used to represent teeth in a human face.

[0112] In an exemplary embodiment, the initial tooth pose information of the three-dimensional tooth model corresponding to the tooth in the three-dimensional coordinate system is obtained.

[0113] In one possible implementation, the aforementioned three-dimensional tooth model can be an overall tooth model, and the aforementioned initial tooth pose information is the initial position data (such as three-dimensional coordinate data) of the three-dimensional tooth model in the three-dimensional coordinate system.

[0114] In another possible implementation, the three-dimensional tooth model can be divided into a maxillary tooth model corresponding to the upper teeth and a mandibular tooth model corresponding to the lower teeth. That is, the aforementioned three-dimensional tooth model includes a maxillary tooth model corresponding to the upper teeth and a mandibular tooth model corresponding to the lower teeth. Accordingly, the initial tooth pose information includes initial upper tooth pose information and initial lower tooth pose information. Optionally, the aforementioned maxillary tooth model is a tooth model constructed based on the maxilla containing the upper teeth, and the aforementioned mandibular tooth model is a tooth model constructed based on the mandible containing the lower teeth.

[0115] The aforementioned initial upper tooth pose information can be the initial position data of the maxillary tooth model in the three-dimensional coordinate system, and the aforementioned initial lower tooth pose information can be the initial position data of the lower tooth model in the three-dimensional coordinate system.

[0116] In one example, as shown in Figure 3(a), a schematic diagram of a three-dimensional skull model with the mouth closed is illustrated. The three-dimensional skull model 30 includes a three-dimensional face model 31, a maxillary tooth model 32, and a mandibular tooth model 33. The maxillary tooth model 32 and the mandibular tooth model 33 can move relative to the three-dimensional face model 31 to mimic the movement of teeth in a real human face. Optionally, the pose changes of the maxillary tooth model 32 and the mandibular tooth model 33 relative to the three-dimensional face model 31 are measured using two rotational degrees of freedom (pitch and yaw) and three translational degrees of freedom (x, y, and z) in the three-dimensional coordinate system.

[0117] In one example, as shown in Figure 3(b), a schematic diagram of a three-dimensional skull model with the mouth open is illustrated. With the mouth open, the mandibular tooth model 33 exhibits significant pose changes, while the maxillary tooth model 32 shows smaller pose changes. As shown in Figure 3(b), the mandibular tooth model 33 undergoes significant downward rotation and forward displacement relative to the three-dimensional face model 31.

[0118] In one example, such as Figure 4 As shown, it exemplarily illustrates a schematic diagram of a three-dimensional coordinate system. Figure 4 The image shows the three-dimensional coordinate system in which the three-dimensional skull model 40 is located. In this three-dimensional coordinate system, the X-axis points from the origin to one side of the skull, the Y-axis points from the origin to the top of the skull, and the Z-axis points from the origin to the outside of the mouth. The X-axis, Y-axis and Z-axis are orthogonal to each other.

[0119] Step 220: Based on the first preset constraint conditions, perform constraint analysis on the target position error corresponding to the tooth position information to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model.

[0120] The first preset constraint condition refers to the condition that the target position error reaches the minimum value. The target position error is the error between the tooth position information and the first predicted tooth position information. The first predicted tooth position information is the two-dimensional tooth position information after the tooth's pose information in the three-dimensional coordinate system is transformed to the two-dimensional coordinate system based on the spatial transformation information. The first tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information. The first tooth pose change information is determined when the target position error reaches the minimum value.

[0121] Therefore, based on spatial transformation information, facial pose change information, initial tooth pose information, and tooth position information, the first tooth pose change information of the 3D tooth model relative to the 3D facial model can be determined.

[0122] Optionally, the first tooth pose change information represents the pose change of the 3D tooth model relative to the 3D face model in the 3D coordinate system. Initially, the relative position between the 3D tooth model and the 3D face model is fixed.

[0123] In real-world scenarios, teeth in a real human face are movable. Accordingly, the technical solution provided in this disclosure represents the movement of teeth in a real human face through the relative positional changes between a 3D tooth model and a 3D face model. During face modeling, on one hand, the 3D face model rotates and translates in the 3D coordinate system to reach a first target position, thereby ensuring that the pose of the 3D face model at the first target position is consistent with the pose of the target face in the face image. On the other hand, the 3D tooth model also rotates and translates in the 3D coordinate system to reach a second target position, thereby ensuring that the pose of the 3D tooth model at the second target position is consistent with the pose of the teeth in the face image. The aforementioned first tooth pose change information can characterize the positional change of the 3D tooth model at the second target position relative to the 3D face model.

[0124] Since the aforementioned spatial transformation information can characterize the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, the face pose change information can characterize the pose change of the three-dimensional face model in the three-dimensional coordinate system, the initial tooth pose information can characterize the tooth position of the three-dimensional tooth model in its initial position, and the tooth position information can characterize the position data of the teeth in the two-dimensional coordinate system, and the three-dimensional tooth model and the three-dimensional face model have an initial relative positional relationship in the initial state, the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model can be determined based on the above data.

[0125] In one possible implementation, after obtaining the actual two-dimensional position data corresponding to the teeth in the aforementioned face image, based on the aforementioned first preset constraint condition, a constraint analysis is performed on the target position error between the actual two-dimensional position data corresponding to the teeth (i.e., the aforementioned tooth position information) and the predicted two-dimensional position data (i.e., the aforementioned first predicted tooth position information), thereby obtaining the tooth pose change matrix of the three-dimensional tooth model relative to the three-dimensional face model, i.e., the aforementioned first tooth pose change information. The predicted two-dimensional position data corresponding to the teeth is the two-dimensional position data corresponding to the teeth after the offset position data (i.e., the first tooth pose information) in the three-dimensional coordinate system is transformed to the two-dimensional coordinate system based on the coordinate system transformation matrix. The offset position data corresponding to the teeth in the three-dimensional coordinate system is the three-dimensional position data corresponding to the teeth after offsetting the initial three-dimensional position data (i.e., the initial tooth pose information) based on the tooth pose change matrix and the face pose change matrix. The aforementioned tooth pose change matrix is ​​determined when the target position error reaches its minimum value.

[0126] In another possible implementation, the above-mentioned three-dimensional tooth model can be divided into a maxillary tooth model corresponding to the upper teeth and a mandibular tooth model corresponding to the lower teeth.

[0127] For the upper teeth, the teeth mentioned above include the upper teeth, the three-dimensional tooth model includes the maxillary tooth model corresponding to the upper teeth, the initial tooth pose information includes the initial upper tooth pose information, and the tooth position information includes the upper tooth position information; correspondingly, the first tooth pose change information mentioned above includes the maxillary bone pose change information.

[0128] Optionally, the aforementioned initial upper tooth pose information includes the initial upper tooth pose data corresponding to the upper teeth in the three-dimensional coordinate system in the maxillary tooth model, such as the initial three-dimensional position data corresponding to the maxillary tooth model; the upper tooth position information includes the two-dimensional position data of the upper teeth in the two-dimensional coordinate system of the face image.

[0129] Accordingly, such as Figure 5 As shown, the implementation process of step 220 above may include the following step 221, based on which the maxillary pose change information can be determined. Figure 5 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 2 .

[0130] Step 221: Based on the first preset constraint condition, perform constraint analysis on the upper tooth position error corresponding to the upper tooth position information to obtain the maxillary bone pose change information of the maxillary tooth model relative to the three-dimensional face model.

[0131] The first tooth pose change information includes maxillary pose change information, the target position error includes maxillary tooth position error, the maxillary tooth position error is the error between the maxillary tooth position information and the predicted maxillary tooth position information, the predicted maxillary tooth position information is the two-dimensional maxillary tooth position information after the target maxillary tooth pose information in the three-dimensional coordinate system is transformed to the two-dimensional coordinate system based on the spatial transformation information, and the target maxillary tooth pose information is the maxillary tooth pose information after offsetting the initial maxillary tooth pose information based on the maxillary pose change information and the face pose change information. The maxillary pose change information is determined when the maxillary tooth position error reaches the minimum value.

[0132] Optionally, after obtaining the actual two-dimensional position data (i.e., upper tooth position information) of the upper teeth in the aforementioned face image, based on the aforementioned first preset constraint condition, a constraint analysis is performed on the position error (i.e., upper tooth position error) between the actual two-dimensional position data of the upper teeth and the predicted two-dimensional position data (i.e., the aforementioned predicted upper tooth position information) of the upper teeth, thereby obtaining the upper tooth pose change matrix of the maxillary tooth model relative to the three-dimensional face model, i.e., the maxillary pose change information. The predicted two-dimensional position data of the upper teeth is the two-dimensional position data corresponding to the upper teeth after the offset position data (i.e., target upper tooth pose information) in the three-dimensional coordinate system is transformed to the two-dimensional coordinate system based on the coordinate system transformation matrix. The offset position data of the upper teeth in the three-dimensional coordinate system is the three-dimensional position data after offsetting the initial three-dimensional position data (i.e., initial upper tooth pose information) of the upper teeth based on the maxillary pose change matrix (i.e., maxillary pose change information) and the face pose change matrix. The maxillary pose change matrix is ​​determined when the upper tooth position error reaches its minimum value.

[0133] Based on the above transformation relationship, it can be seen that after obtaining the above spatial transformation information, face pose change information, initial upper tooth pose information, and upper tooth position information, the maxillary bone pose change information of the maxillary tooth model relative to the three-dimensional face model can be determined.

[0134] Optionally, the maxillary pose change information represents the pose change of the maxillary tooth model relative to the 3D face model in a 3D coordinate system. Initially, the relative position between the maxillary tooth model and the 3D face model is fixed.

[0135] In real-world scenarios, teeth on a real human face are movable, and the upper teeth naturally exhibit corresponding movements. The technical solution provided in this disclosure represents the movement of the upper teeth in a real human face through the relative positional changes between a maxillary tooth model and a 3D face model. During face modeling, on one hand, the 3D face model rotates and shifts in the 3D coordinate system to reach a first target position, ensuring that the pose of the 3D face model at the first target position is consistent with the pose of the target face in the face image. On the other hand, the maxillary tooth model also rotates and shifts in the 3D coordinate system to reach a third target position, ensuring that the pose of the maxillary tooth model at the third target position is consistent with the pose of the upper teeth in the face image. The aforementioned maxillary pose change information characterizes the pose change of the maxillary tooth model at the third target position relative to the 3D face model.

[0136] Since the aforementioned spatial transformation information can characterize the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, the facial pose change information can characterize the pose change of the three-dimensional face model in the three-dimensional coordinate system, the initial upper tooth pose information can characterize the initial three-dimensional position of the upper teeth corresponding to the initial position of the maxillary tooth model, and the upper tooth position information can characterize the actual two-dimensional position data of the upper teeth in the two-dimensional coordinate system, and the maxillary tooth model and the three-dimensional face model have an initial relative positional relationship in the initial state, based on the above data, the maxillary pose change information of the maxillary tooth model relative to the three-dimensional face model can be accurately determined, improving the accuracy of determining the upper tooth pose change, and thus determining the accurate pose of the upper teeth in the three-dimensional coordinate system.

[0137] In one possible implementation, the pose change of the maxillary tooth model can be measured from two perspectives: rotation and displacement; correspondingly, the maxillary pose change information can include maxillary rotation information and maxillary displacement information.

[0138] Optionally, based on spatial transformation information, facial pose change information, initial upper tooth pose information, and upper tooth position information, the maxillary rotation information of the maxillary tooth model relative to the three-dimensional face model and the maxillary displacement information of the maxillary tooth model relative to the three-dimensional face model are determined.

[0139] Optionally, the maxillary bone rotation information includes rotation data of the maxillary tooth model relative to the 3D face model in a three-dimensional coordinate system. Optionally, the maxillary bone displacement information includes displacement data of the maxillary tooth model relative to the 3D face model in a three-dimensional coordinate system.

[0140] Optionally, the maxillary pose change information includes a maxillary tooth pose change matrix, which includes the aforementioned maxillary tooth rotation data and the aforementioned maxillary tooth displacement data.

[0141] Specifically, the upper tooth rotation data can include the pitch angle of the maxillary tooth model relative to the 3D face model and the yaw angle of the maxillary tooth model relative to the 3D face model; the upper tooth displacement data can include the displacement data of the maxillary tooth model relative to the 3D face model in each direction of the 3D coordinate system.

[0142] By determining the rotation and displacement of the maxillary tooth model relative to the 3D face model, the pose change of the maxillary tooth model relative to the 3D face model can be accurately measured, thus improving the accuracy of determining the pose change of the maxillary teeth.

[0143] For the lower teeth, the teeth also include the lower teeth, the three-dimensional tooth model also includes the mandibular tooth model corresponding to the lower teeth, the initial tooth pose information also includes the initial lower tooth pose information, the tooth position information also includes the lower tooth position information; correspondingly, the first tooth pose change information includes the mandibular pose change information.

[0144] Optionally, the aforementioned initial mandibular tooth pose information includes the initial mandibular tooth pose data corresponding to the mandibular teeth in the three-dimensional coordinate system in the mandibular tooth model; the mandibular tooth position information includes the two-dimensional position data of the mandibular teeth in the face image corresponding to the mandibular teeth in the two-dimensional coordinate system where the face image is located.

[0145] Accordingly, such as Figure 5 As shown, the implementation process of step 220 above may also include the following step 222, based on which the above-mentioned mandibular pose change information can be determined.

[0146] Step 222: Based on the first preset constraint condition, perform constraint analysis on the lower tooth position error corresponding to the lower tooth position information to obtain the mandibular pose change information of the mandibular tooth model relative to the three-dimensional face model.

[0147] The first tooth pose change information includes mandibular pose change information, the target position error includes mandibular position error, the mandibular position error is the error between the mandibular position information and the predicted mandibular position information, the predicted mandibular position information is the two-dimensional mandibular position information after the target mandibular pose information in the three-dimensional coordinate system is transformed to the two-dimensional coordinate system based on spatial transformation information, and the target mandibular pose information is the mandibular pose information after offsetting the initial mandibular pose information based on mandibular pose change information and face pose change information. The mandibular pose change information is determined when the mandibular position error reaches the minimum value.

[0148] Optionally, after obtaining the actual two-dimensional position data (i.e., lower tooth position information) of the lower teeth in the aforementioned face image, based on the aforementioned first preset constraint condition, a constraint analysis is performed on the position error (i.e., lower tooth position error) between the actual two-dimensional position data of the lower teeth and the predicted two-dimensional position data (i.e., the aforementioned predicted lower tooth position information) of the lower teeth, thereby obtaining the lower tooth pose change matrix of the mandibular tooth model relative to the three-dimensional face model, i.e., the mandibular pose change information. The predicted two-dimensional position data of the lower teeth is the two-dimensional position data corresponding to the offset position data (i.e., target lower tooth pose information) of the lower teeth in the three-dimensional coordinate system after being transformed to the two-dimensional coordinate system based on the coordinate system transformation matrix. The offset position data of the lower teeth in the three-dimensional coordinate system is the three-dimensional position data after offsetting the initial three-dimensional position data (i.e., initial lower tooth pose information) of the lower teeth based on the mandibular pose change matrix (i.e., mandibular pose change information) and the face pose change matrix. The mandibular pose change matrix is ​​determined when the upper tooth position error reaches its minimum value.

[0149] Based on the above transformation relationship, it can be seen that after obtaining spatial transformation information, face pose change information, initial mandibular tooth pose information, and mandibular tooth position information, the mandibular pose change information of the mandibular tooth model relative to the three-dimensional face model can be determined.

[0150] Optionally, the mandibular pose change information represents the pose change of the mandibular tooth model relative to the 3D face model in a 3D coordinate system. Initially, the relative position between the mandibular tooth model and the 3D face model is fixed.

[0151] In real-world scenarios, teeth on a real human face are movable, and the lower teeth naturally exhibit corresponding movement. The technical solution provided in this disclosure represents the movement of the lower teeth on a real human face through the relative positional changes between a mandibular tooth model and a 3D face model. During face modeling, on one hand, the 3D face model rotates and shifts in the 3D coordinate system to reach a first target position, ensuring that the pose of the 3D face model at the first target position is consistent with the pose of the target face in the face image. On the other hand, the mandibular tooth model also rotates and shifts in the 3D coordinate system to reach a fourth target position, ensuring that the pose of the mandibular tooth model at the fourth target position is consistent with the pose of the lower teeth in the face image. The aforementioned mandibular pose change information characterizes the positional change of the mandibular tooth model at the fourth target position relative to the 3D face model at the first target position.

[0152] Since the aforementioned spatial transformation information can characterize the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, the facial pose change information can characterize the pose change of the three-dimensional face model in the three-dimensional coordinate system, the initial mandibular tooth pose information can characterize the initial three-dimensional position of the mandibular teeth model in its initial position, and the mandibular tooth position information can characterize the actual two-dimensional position data of the mandibular teeth in the two-dimensional coordinate system, and the mandibular teeth model and the three-dimensional face model have an initial relative positional relationship in the initial state, based on the above data, the mandibular pose change information of the mandibular teeth model relative to the three-dimensional face model can be accurately determined, improving the accuracy of determining the mandibular tooth pose change, and thus the accurate pose of the mandibular teeth in the three-dimensional coordinate system can be determined.

[0153] In one possible implementation, similar to the angle used to measure the pose change of the maxillary tooth model, the pose change of the mandibular tooth model can also be measured from two angles: rotation and displacement; correspondingly, the mandibular pose change information includes mandibular rotation information and mandibular displacement information.

[0154] Optionally, based on spatial transformation information, facial pose change information, initial mandibular pose information, and mandibular position information, the mandibular rotation information of the mandibular tooth model relative to the three-dimensional face model and the mandibular displacement information of the mandibular tooth model relative to the three-dimensional face model are determined.

[0155] Optionally, the aforementioned mandibular rotation information includes rotation data of the mandibular tooth model relative to the three-dimensional face model in a three-dimensional coordinate system. Optionally, the mandibular displacement information includes displacement data of the mandibular tooth model relative to the three-dimensional face model in a three-dimensional coordinate system.

[0156] Optionally, the mandibular pose change information includes a mandibular pose change matrix, which includes the aforementioned mandibular rotation data and the aforementioned mandibular displacement data.

[0157] Specifically, the mandibular tooth rotation data can include the pitch angle of the mandibular tooth model relative to the 3D face model and the yaw angle of the mandibular tooth model relative to the 3D face model; the mandibular tooth displacement data can include the displacement data of the mandibular tooth model relative to the 3D face model in each direction of the 3D coordinate system.

[0158] By determining the rotation and displacement of the mandibular tooth model relative to the 3D face model, the pose change of the mandibular tooth model relative to the 3D face model can be accurately measured, thus improving the accuracy of determining the pose change of the upper teeth.

[0159] In real-world dental activity scenarios, the range of pose changes of teeth relative to the skull is limited. This means that the range of pose changes of the aforementioned 3D tooth model relative to the 3D face model is also limited. Accordingly, as... Figure 6 As shown, the above method also includes the following step 240, Figure 6 This is flowchart three illustrating a face image processing method according to an exemplary embodiment.

[0160] Step 240: Obtain the range of tooth pose changes corresponding to the three-dimensional tooth model.

[0161] Optionally, the tooth pose variation range information is used to characterize the pose variation range of the 3D tooth model relative to the 3D face model.

[0162] Optionally, the information on the range of tooth pose changes includes pose change threshold data of the three-dimensional tooth model relative to the three-dimensional face model.

[0163] Optionally, the pose change threshold data includes rotation threshold data and displacement threshold data of the 3D tooth model relative to the 3D face model.

[0164] Optionally, the rotation threshold data includes pitch and yaw angle thresholds for the 3D tooth model relative to the 3D face model.

[0165] Optionally, the displacement threshold data includes the displacement thresholds of the 3D tooth model relative to the 3D face model in each dimension of the 3D coordinate system.

[0166] Accordingly, such as Figure 6 As shown, step 220 above can be replaced by step 22a below.

[0167] Step 22a: Based on the second preset constraint conditions, perform constraint analysis on the first fusion position error corresponding to the tooth position information to obtain the first tooth pose change information.

[0168] The second preset constraint condition refers to the condition that the first fusion position error reaches the minimum value. The fusion position error is obtained by fusing the target position error and the pose change range deviation. The pose change range deviation refers to the deviation between the first tooth pose change information and the tooth pose change range information. The first tooth pose change information is determined when the first fusion position error reaches the minimum value.

[0169] Optionally, the tooth pose change range information includes the range thresholds of various pose change data in the first tooth pose change information; the various pose change data in the first tooth pose change information, such as the rotation data and displacement data of the three-dimensional tooth model relative to the three-dimensional face model, are compared with the range thresholds of various pose change data, such as the rotation range threshold and the displacement range threshold, to obtain the above-mentioned pose change range deviation.

[0170] Based on the relationships between the above data, it can be seen that after obtaining the above spatial transformation information, face pose change information, tooth pose change range information, initial tooth pose information, and tooth position information, the first tooth pose change information can be determined.

[0171] In the technical solution provided by this disclosure, it is also necessary to obtain the tooth pose change range information corresponding to the three-dimensional tooth model, and then combine the tooth pose change range information to determine the pose change of the three-dimensional tooth model relative to the three-dimensional face model. This can ensure that the output tooth pose change is within the biologically reasonable range, avoid obtaining results that exceed the biological pose change range, and further improve the accuracy of determining the tooth pose change.

[0172] Similarly, in real-world dental activity scenarios, tooth movement is continuous. Therefore, the changes in tooth pose between adjacent facial images in the video captured by the device will not be too large. This means that the pose change of the teeth relative to the skull in the current image frame is relatively close to the pose change of the teeth relative to the skull in the previous image frame. Accordingly, such as Figure 7 As shown, the above method also includes the following steps (250-260), Figure 7 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 4 .

[0173] Step 250: Determine the previous face image corresponding to the current face image.

[0174] Optionally, the face image sequence to which the face image belongs can be determined, and the face image sequence can be a video. The previous face image corresponding to the current face image can then be found from the face image sequence.

[0175] Step 260: Obtain the second tooth pose change information corresponding to the previous face image.

[0176] The second set of tooth pose change information is the tooth pose change information corresponding to the previous face image. When determining the tooth pose corresponding to the previous face image, the tooth pose change information corresponding to the previous face image is already determined, so it can be obtained directly here.

[0177] Accordingly, such as Figure 7 As shown, step 220 above can be replaced by step 22b below.

[0178] Step 22b: Based on the third preset constraint condition, perform constraint analysis on the second fusion position error corresponding to the tooth position information to obtain the first tooth pose change information.

[0179] The third preset constraint condition refers to the condition that the second fusion position error reaches the minimum value. The second fusion position error is obtained by fusing the first fusion position error and the pose motion range deviation. The pose motion range deviation refers to the deviation between the first tooth pose change information and the second tooth pose change information. The first tooth pose change information is determined when the second fusion position error reaches the minimum value.

[0180] Optionally, the pose change data in the first tooth pose change information can be compared with the pose change data in the second tooth pose change information to obtain the pose motion difference data. The pose motion difference data can then be compared with the corresponding preset pose motion change range threshold to obtain the pose motion range deviation. Specifically, the pose motion range deviation is the deviation of the tooth's pose motion range between adjacent frames.

[0181] Optionally, the pose change data in the first tooth pose change information are the tooth pose change data of the three-dimensional tooth model corresponding to the tooth in the current image relative to the initial position, such as rotation data, displacement data, etc. Correspondingly, the pose change data in the second tooth pose change information are the tooth pose change data of the three-dimensional tooth model corresponding to the tooth in the next image of the current image relative to the initial position, such as rotation data, displacement data, etc.

[0182] Based on the relationships between the above data, it can be seen that after obtaining spatial transformation information, face pose change information, second tooth pose change information, initial tooth pose information, and tooth position information, the first tooth pose change information can be determined.

[0183] In the technical solution provided by this disclosure, it is also necessary to obtain the tooth pose change information corresponding to the previous face image, and then combine the tooth pose change corresponding to the previous face image to accurately determine the pose change of the teeth relative to the skull in the current image frame. This can ensure that the output tooth pose change is within the reasonable range of kinematics, avoid obtaining results that exceed the range of kinematic pose change, and further improve the accuracy of determining the tooth pose change.

[0184] Of course, the information on the range of tooth pose changes and the information on the second tooth pose changes corresponding to the previous face image can be combined to determine the information on the first tooth pose changes corresponding to the current face image.

[0185] That is, based on spatial transformation information, face pose change information, tooth pose change range information, second tooth pose change information, initial tooth pose information, and tooth position information, the first tooth pose change information is determined.

[0186] In one example, the first tooth pose change information can be determined according to the following formulas (1) and (2), and the third constraint conditions include the following formulas (1) and (2).

[0187]

[0188]

[0189] In formula (1), n ​​represents the number of key points of the tooth, i represents the sequence number of the key point of the tooth, and the first term "w" represents the number of key points of the tooth. i ||u′ i -u i || 2 " is the constraint term for each key point of the tooth, representing the tooth position error, u′ i It is the actual two-dimensional coordinate data of the key points of the teeth in the face image, w i It is the weight of the constraint term, w i Influenced by the confidence level of the dental key points and the facial angle (higher confidence level and more positive facial angle result in greater weight), u i It is the predicted two-dimensional coordinate data of the tooth points in the tooth model in the face image (i.e., the first predicted tooth position information mentioned above); the second item "w range ||Rt-Rt range || 2 "This is the tooth position range constraint, representing the deviation of the position change range, w" range This is the weight of this term, Rt range Based on the range of pose changes in skeletal physiology, Rt is the tooth pose change matrix to be determined, where R represents tooth rotation and t represents tooth displacement. R can be decomposed into two rotational degrees of freedom: pitch and yaw, and t can be decomposed into three displacement degrees of freedom: X, Y, and Z, where X, Y, and Z represent the three directional dimensions in a three-dimensional coordinate system. The third term "w" smooth ||Rt-Rt pre || 2 " is an inter-frame regularization term, representing the deviation in pose range of motion, used to improve stability, w smooth This is the weight of this item, w smooth Affected by frame rate (the higher the frame rate, the greater the weight), Rt pre It is the tooth pose change matrix corresponding to the teeth in the previous frame of the face image.

[0190] Formula (2) is u i The corresponding calculation formula, where, It is the camera intrinsic parameter matrix (i.e., spatial transformation information), M head It is the head extrinsic parameter matrix (i.e., the face pose transformation matrix), p iIt refers to the initial three-dimensional coordinates (i.e., initial tooth pose information) of the key points of the teeth in the three-dimensional tooth model.

[0191] When the above formula (1) reaches its minimum value, the above tooth pose change matrix can be solved. The above formula (1) introduces the constraint of the previous frame, which improves the stability of the reconstructed face model and tooth model in multi-frame video scenes. It also introduces the range of tooth pose changes determined based on the facial skeleton anatomy data as a constraint, which improves the rationality of the tooth pose.

[0192] Step 230: Based on the first tooth pose change information and the face pose change information, the initial tooth pose information is offset to obtain the target tooth pose information.

[0193] Optionally, the target tooth pose information mentioned above is used to characterize the location where the tooth effect is added.

[0194] Since the aforementioned first tooth pose change information can characterize the pose change of the 3D tooth model relative to the 3D face model in the 3D coordinate system, and the initial tooth pose information can characterize the tooth position corresponding to the 3D tooth model in the initial position, the target tooth pose corresponding to the tooth in the 3D coordinate system can be determined by offsetting the aforementioned pose change amount based on the initial tooth pose. The aforementioned target tooth pose can then be kept consistent with the tooth pose in the face image.

[0195] In an exemplary embodiment, the target tooth pose information includes the target maxillary tooth pose information. The maxillary pose change information determined in the above steps may include maxillary rotation information and maxillary displacement information of the maxillary tooth model relative to the three-dimensional face model; then correspondingly, as... Figure 5 As shown, the implementation process of step 230 above may include the following step 231, based on which the target maxillary tooth pose information can be determined.

[0196] Step 231: Based on the facial pose change information, maxillary rotation information, and maxillary displacement information, the initial maxillary pose information is offset to obtain the target maxillary pose information.

[0197] Since the above-mentioned maxillary pose change information can characterize the pose change of the maxillary tooth model relative to the three-dimensional face model in the three-dimensional coordinate system, and the initial maxillary tooth pose information can characterize the tooth position corresponding to the maxillary tooth model in the initial position, the target maxillary tooth pose in the three-dimensional coordinate system can be determined based on the maxillary tooth pose change amount and the initial maxillary tooth pose. The target maxillary tooth pose can then be kept consistent with the maxillary tooth pose in the face image.

[0198] Specifically, the three-dimensional face model can be offset according to the aforementioned face pose change information; at the same time, the maxillary tooth model can be rotated relative to the three-dimensional face model according to the aforementioned maxillary rotation information, and the maxillary tooth model can be displaced relative to the three-dimensional face model according to the aforementioned maxillary displacement information, thereby obtaining accurate target maxillary tooth pose information.

[0199] In an exemplary embodiment, the target tooth pose information further includes target mandibular tooth pose information. The mandibular pose change information determined in the above steps includes mandibular rotation information and mandibular displacement information relative to the three-dimensional face model; therefore, accordingly, as Figure 5 As shown, the implementation process of step 230 above can also be carried out as follows: step 232. Based on step 232, the target mandibular tooth pose information can be determined.

[0200] Step 232: Based on the facial pose change information, mandibular rotation information, and mandibular displacement information, the initial mandibular pose information is offset to obtain the target mandibular pose information.

[0201] Since the aforementioned mandibular pose change information can characterize the pose change of the mandibular tooth model relative to the three-dimensional face model in the three-dimensional coordinate system, and the initial mandibular tooth pose information can characterize the tooth position corresponding to the mandibular tooth model in the initial position, the target mandibular tooth pose in the three-dimensional coordinate system can be determined based on the mandibular tooth pose change and the initial mandibular tooth pose. The target mandibular tooth pose can then be kept consistent with the mandibular tooth pose in the face image.

[0202] Specifically, the 3D face model can be offset according to the aforementioned face pose change information; at the same time, the mandibular tooth model can be rotated relative to the 3D face model according to the aforementioned mandibular rotation information, and the mandibular tooth model can be displaced relative to the 3D face model according to the aforementioned mandibular displacement information, thereby obtaining accurate target mandibular tooth pose information.

[0203] In the exemplary embodiment, determining the tooth pose requires not only considering the changes in tooth pose caused by tooth movement, but also the differences in tooth morphology between individuals. In real-world scenarios, the morphology of teeth varies among individuals. Therefore, when determining the tooth pose in three-dimensional space, it is also necessary to determine the changes in tooth morphology between individuals. In the exemplary embodiment, as... Figure 8 As shown, the above method also includes the following step 270. Figure 8 This is a flowchart of a face image processing method according to an exemplary embodiment. Figure 5 .

[0204] Step 270: Based on the fourth preset constraint condition, perform constraint analysis on the positional morphology error corresponding to the tooth position information to obtain the first tooth morphology change information corresponding to the tooth.

[0205] Among them, the fourth preset constraint condition refers to the condition that the positional shape error reaches the minimum value. The first tooth shape change information represents the shape change between the tooth shape and the tooth shape corresponding to the tooth in the three-dimensional tooth model. The positional shape error is the error between the tooth position information and the second predicted tooth position information. The second predicted tooth position information is the two-dimensional tooth position information corresponding to the second tooth pose information in the three-dimensional coordinate system after converting the second tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information. The second tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the face pose change information, the first tooth pose change information, and the first tooth shape change information. The first tooth shape change information is determined when the positional shape error reaches the minimum value.

[0206] In one possible implementation, after acquiring the actual two-dimensional position data of the teeth in the aforementioned face image and determining the aforementioned tooth pose change matrix (i.e., the aforementioned first tooth pose change information), based on the aforementioned fourth preset constraint condition, a constraint analysis is performed on the difference data (i.e., the aforementioned positional morphology error) between the actual two-dimensional position data of the teeth (i.e., the aforementioned tooth position information) and the predicted two-dimensional position data of the teeth at this time (i.e., the aforementioned second predicted tooth position information) (the constraint analysis aims to minimize the difference data), thereby obtaining the tooth morphology change matrix (i.e., the aforementioned first tooth morphology change information) that represents the change from the tooth model morphology in the three-dimensional tooth model to the actual tooth morphology. The predicted two-dimensional position data of the teeth is the two-dimensional position data corresponding to the teeth after the offset position data (i.e., the second tooth pose information) in the three-dimensional coordinate system is transformed to the two-dimensional coordinate system based on the coordinate system transformation matrix. The offset position data of the teeth in the three-dimensional coordinate system is the three-dimensional position data after offsetting the initial three-dimensional position data (i.e., the initial tooth pose information) of the teeth based on the tooth pose change matrix, the face pose change matrix, and the tooth morphology change matrix. The aforementioned tooth morphology change matrix is ​​determined when the positional morphology error reaches its minimum value.

[0207] Based on the above transformation relationship, it can be seen that after obtaining spatial transformation information, face pose change information, first tooth pose change information, initial tooth pose information, and tooth position information, the first tooth morphological change information corresponding to the tooth can be determined.

[0208] Optionally, the first tooth morphological change information represents the morphological change between the tooth's actual shape and its corresponding shape in the 3D tooth model. For example, it represents the difference between the actual tooth shape, such as its actual contour or the actual location of key points, and the tooth model shape in the 3D tooth model.

[0209] Optionally, the first tooth morphological change information includes scaling data and displacement data corresponding to the tooth in the 3D tooth model. Optionally, the scaling data represents the scaling amount of the tooth in the face image relative to the corresponding model tooth in the 3D tooth model. Optionally, the displacement data represents the displacement amount of the tooth in the face image relative to the corresponding model tooth in the 3D tooth model.

[0210] In real-world dental activity scenarios, not only are tooth movements continuous, but the morphology of each individual's teeth also doesn't easily change. Therefore, the morphological changes in teeth between adjacent face images in the video captured by the device are not significant. This means that the morphology of teeth in the current image frame is relatively close to that in the previous image frame. Therefore, in the exemplary embodiment, tooth morphological change information corresponding to the previous face image can also be obtained, and then the tooth morphological change corresponding to the current image frame can be determined by combining the tooth morphology corresponding to the previous face image. Accordingly, after determining the previous image corresponding to the face image, the above method further includes: obtaining second tooth morphological change information corresponding to the previous face image.

[0211] The aforementioned second tooth morphology change information refers to the tooth morphology change information corresponding to the previous face image. When determining the tooth pose corresponding to the previous face image, not only is the tooth pose change information corresponding to the previous face image determined, but also the tooth morphology change information corresponding to the previous face image, which can be obtained directly here.

[0212] Accordingly, the specific implementation process of step 270 above can be: based on the fifth preset constraint condition, perform constraint analysis on the position morphology fusion error corresponding to the tooth position information to obtain the first tooth morphology change information;

[0213] Among them, the fifth preset constraint condition refers to the condition that the position and shape fusion error reaches the minimum value. The position and shape fusion error is obtained by fusing the position and shape error and the deviation of the shape change range. The deviation of the shape change range refers to the deviation between the first tooth shape change information and the second tooth shape change information. The first tooth shape change information is determined under the condition that the position and shape fusion error reaches the minimum value.

[0214] Optionally, comparing the morphological change data in the first tooth morphological change information with the morphological change data in the second tooth morphological change information yields the morphological change difference data, i.e., the aforementioned morphological change range deviation. Optionally, comparing the morphological change difference data with a corresponding preset morphological change range threshold yields another type of morphological change range deviation. Specifically, the morphological change range deviation is the morphological deviation of the tooth between adjacent frames.

[0215] Optionally, the morphological change data in the first tooth morphological change information are the tooth morphological change data that occurs when the tooth modal morphology in the 3D tooth model is transformed into the actual tooth morphology in the current image, such as contour scaling data and displacement data. Correspondingly, the morphological change data in the second tooth morphological change information are the tooth morphological change data that occurs when the tooth modal morphology in the 3D tooth model is transformed into the actual tooth morphology in the next image, such as contour scaling data and displacement data.

[0216] In the technical solution provided by this embodiment, it is also necessary to obtain the tooth morphology change information corresponding to the previous face image, and then combine the tooth morphology change corresponding to the previous face image to accurately determine the tooth morphology change in the current image frame relative to the model teeth. This can ensure that the output tooth morphology change is reasonable and stable, and further improve the accuracy of determining the tooth pose change.

[0217] In one example, information on changes in tooth morphology can be determined by the following formulas (3) and (4), and the fifth constraint mentioned above can include the following formulas (3) and (4).

[0218]

[0219]

[0220] In formula (3), n represents the number of key points of the tooth, i represents the sequence number of the key point of the tooth, and the first term "w" represents the number of key points of the tooth. i ||u′ i -u i || 2 “ is the constraint term for each tooth's key point, representing the positional and morphological error, u′ i It is the actual two-dimensional coordinate data of the key points of the teeth in the face image, w i It is the weight of the constraint term, w i Influenced by the confidence level of the dental key points and the facial angle (higher confidence level and more positive facial angle result in greater weight), u i It is the predicted two-dimensional coordinate data of the tooth points in the tooth model in the face image; the second item "w′ smooth ||st′-st′ pre ||2 " is the inter-frame tooth morphology regularization term, representing the deviation in the range of morphological changes, w′ smooth This is the weight of this item, w′ smooth Affected by frame rate (higher frame rate, higher weight), st′ represents the morphological change information of the first tooth, s represents the scaling data of the tooth's tangent vector space, and t′ represents the displacement data of the tooth's tangent vector space. pre It is the scaling and displacement data of the teeth in the previous frame of the face image, that is, the second tooth shape change information.

[0221] Formula (4) is u i The corresponding calculation formula is obtained by improving upon the above formula (2). Wherein, It is the camera intrinsic parameter matrix (i.e., spatial transformation information), M head It is the head extrinsic parameter matrix (i.e., the face pose transformation matrix), M coarse Here, [s|t′] is the tooth pose change matrix determined above, and [s|t′] is the tooth morphology change matrix to be solved. i These are the initial 3D coordinates of the key points of the teeth in the 3D tooth model.

[0222] When the above formula (3) reaches its minimum value, the above tooth morphology change matrix can be solved.

[0223] Accordingly, after determining the morphological change information of the first tooth, step 230 above can be replaced by step 23a below.

[0224] Step 23a: Based on the face pose change information, the first tooth pose change information, and the first tooth shape change information, the initial tooth pose information is offset to obtain the target tooth pose information.

[0225] Since the aforementioned first tooth pose change information can characterize the pose change of the 3D tooth model relative to the 3D face model in the 3D coordinate system, the first tooth morphology change information can characterize the morphology change of the teeth in the target face relative to the morphology of the model teeth in the 3D tooth model, and the initial tooth pose information can characterize the tooth position corresponding to the 3D tooth model in the initial position, the target tooth pose corresponding to the tooth in the 3D coordinate system can be accurately determined based on the tooth pose change amount, tooth morphology change amount, and initial tooth pose. The aforementioned target tooth pose can then be kept consistent with the pose and morphology of the teeth in the face image.

[0226] Specifically, the 3D face model can be offset according to the aforementioned face pose change information; at the same time, the 3D tooth model can be rotated relative to the 3D face model according to the aforementioned first tooth pose change information; and the teeth in the 3D tooth model can be scaled and displaced according to the aforementioned first tooth morphology change information, thereby obtaining more accurate target lower tooth pose information.

[0227] Optionally, the aforementioned target tooth pose information represents the location where tooth effects can be added. Tooth effects can be added based on this target tooth pose information. For example, in short video effects and live streaming scenarios, effects such as teeth whitening, dentures, and dental ornaments can be achieved based on the target tooth pose, improving the fit and stability of the tooth effects. Furthermore, accurate tooth pose determination helps improve the realism of virtual human speech.

[0228] The technical solution provided in this disclosure, after determining the tooth pose change information, can further determine the tooth morphology change information, and calibrate the tooth morphology based on the tooth morphology change, so that the three-dimensional tooth model can not only accurately display the tooth pose of the target face, but also accurately display the morphology of each tooth in the target face, effectively improving the accuracy of tooth pose determination.

[0229] In summary, the technical solution provided by the embodiments of this disclosure minimizes the positional error between the actual and predicted positions of the teeth by setting preset constraints. When the positional error is minimized, the corresponding tooth pose change can be determined. Then, based on the tooth pose change and the initial tooth pose, the tooth can be determined in the corresponding target pose. This allows the reconstructed 3D face to represent the tooth information of the real face, improves the consistency between the tooth pose of the 3D face and the tooth pose of the real face, and enhances the accuracy of tooth information and the realism of the 3D face overall.

[0230] Figure 9 This is a block diagram of a face image processing apparatus according to an exemplary embodiment. (Refer to...) Figure 9 The device 900 includes:

[0231] The target information acquisition module 910 is configured to acquire spatial transformation information, facial pose change information corresponding to a facial image, tooth position information of the teeth in the facial image, and initial tooth pose information of the three-dimensional tooth model corresponding to the teeth in the three-dimensional coordinate system; the spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system corresponding to the facial image, and the facial pose change information represents the pose change of the three-dimensional facial model corresponding to the facial image in the three-dimensional coordinate system;

[0232] The pose change determination module 920 is configured to perform constraint analysis on the target position error corresponding to the tooth position information based on a first preset constraint condition, to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model; wherein, the first preset constraint condition refers to the condition that constrains the target position error to reach a minimum value, the target position error is the error between the tooth position information and the first predicted tooth position information, the first predicted tooth position information is the two-dimensional tooth position information corresponding to the first tooth pose information in the three-dimensional coordinate system after transforming the first tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the first tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information, and the first tooth pose change information is determined under the condition that the target position error reaches a minimum value;

[0233] The tooth pose determination module 930 is configured to perform an offset on the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain target tooth pose information, wherein the target tooth pose information is used to characterize the addition position corresponding to the tooth effect.

[0234] In some possible designs, the teeth include upper teeth, the three-dimensional tooth model includes the maxillary tooth model corresponding to the upper teeth, the initial tooth pose information includes initial upper tooth pose information, the tooth position information includes upper tooth position information, and the pose change determination module is specifically configured to perform constraint analysis on the upper tooth position error corresponding to the upper tooth position information based on the first preset constraint conditions, so as to obtain the maxillary pose change information of the maxillary tooth model relative to the three-dimensional face model;

[0235] Wherein, the first tooth pose change information includes the maxillary pose change information, the target position error includes the upper tooth position error, the upper tooth position error is the error between the upper tooth position information and the predicted upper tooth position information, the predicted upper tooth position information is the two-dimensional upper tooth position information corresponding to the target upper tooth pose information in the three-dimensional coordinate system after transforming the target upper tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the target upper tooth pose information is the upper tooth pose information after offsetting the initial upper tooth pose information based on the maxillary pose change information and the face pose change information, and the maxillary pose change information is determined when the upper tooth position error reaches the minimum value.

[0236] In some possible designs, the maxillary pose change information includes maxillary rotation information and maxillary displacement information of the maxillary tooth model relative to the three-dimensional face model. The tooth pose determination module is specifically configured to perform offsetting of the initial maxillary pose information based on the face pose change information, the maxillary rotation information, and the maxillary displacement information to obtain the target maxillary pose information.

[0237] In some possible designs, the teeth also include lower teeth, the three-dimensional tooth model also includes a mandibular tooth model corresponding to the lower teeth, the initial tooth pose information also includes initial lower tooth pose information, the tooth position information also includes lower tooth position information, and the pose change determination module is specifically configured to perform constraint analysis on the lower tooth position error corresponding to the lower tooth position information based on the first preset constraint conditions, so as to obtain the mandibular pose change information of the mandibular tooth model relative to the three-dimensional face model;

[0238] Wherein, the first tooth pose change information includes the mandibular pose change information, the target position error includes the mandibular position error, the mandibular position error is the error between the mandibular position information and the predicted mandibular position information, the predicted mandibular position information is the two-dimensional mandibular position information corresponding to the target mandibular pose information in the three-dimensional coordinate system after transforming the target mandibular pose information of the mandibular tooth into the two-dimensional coordinate system based on the spatial transformation information, the target mandibular pose information is the mandibular pose information after offsetting the initial mandibular pose information based on the mandibular pose change information and the face pose change information, and the mandibular pose change information is determined when the mandibular position error reaches the minimum value.

[0239] In some possible designs, the mandibular pose change information includes mandibular rotation information and mandibular displacement information of the mandibular tooth model relative to the three-dimensional face model. The tooth pose determination module is further configured to perform offsetting of the initial mandibular pose information based on the face pose change information, the mandibular rotation information, and the mandibular displacement information to obtain the target mandibular pose information.

[0240] In some possible designs, the device further includes:

[0241] The pose change range acquisition module is configured to acquire the pose change range information of the teeth corresponding to the three-dimensional tooth model. The tooth pose change range information is used to characterize the pose change range of the three-dimensional tooth model relative to the three-dimensional face model.

[0242] The pose change determination module is further configured to perform constraint analysis on the first fusion position error corresponding to the tooth position information based on the second preset constraint conditions, so as to obtain the first tooth pose change information.

[0243] The second preset constraint condition refers to the condition that constrains the first fusion position error to reach the minimum value. The fusion position error is obtained by fusing the target position error and the pose change range deviation. The pose change range deviation refers to the deviation between the first tooth pose change information and the tooth pose change range information. The first tooth pose change information is determined when the first fusion position error reaches the minimum value.

[0244] In some possible designs, the device further includes:

[0245] The face image determination module is configured to determine the previous face image corresponding to the face image;

[0246] The target information acquisition module is also configured to acquire the second tooth pose change information corresponding to the previous face image;

[0247] The pose change determination module is further configured to perform constraint analysis on the second fusion position error corresponding to the tooth position information based on a third preset constraint condition, so as to obtain the first tooth pose change information.

[0248] The third preset constraint condition refers to the condition that constrains the second fusion position error to reach the minimum value. The second fusion position error is obtained by fusing the first fusion position error and the pose motion range deviation. The pose motion range deviation refers to the deviation between the first tooth pose change information and the second tooth pose change information. The first tooth pose change information is determined when the second fusion position error reaches the minimum value.

[0249] In some possible designs, the device further includes:

[0250] The morphological change determination module is configured to perform constraint analysis on the positional morphological error corresponding to the tooth position information based on a fourth preset constraint condition, to obtain the first tooth morphological change information corresponding to the tooth; wherein, the fourth preset constraint condition refers to the condition that constrains the positional morphological error to reach a minimum value, the first tooth morphological change information represents the morphological change between the tooth morphology and the tooth morphology corresponding to the tooth in the three-dimensional tooth model, the positional morphological error is the error between the tooth position information and the second predicted tooth position information, the second predicted tooth position information is the two-dimensional tooth position information corresponding to the second tooth pose information corresponding to the tooth in the three-dimensional coordinate system after transforming it to the two-dimensional coordinate system based on the spatial transformation information, the second tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the face pose change information, the first tooth pose change information and the first tooth morphological change information, and the first tooth morphological change information is determined under the condition that the positional morphological error reaches a minimum value;

[0251] The tooth pose determination module is further configured to perform an offset on the initial tooth pose information based on the face pose change information, the first tooth pose change information, and the first tooth morphology change information to obtain the target tooth pose information.

[0252] In some possible designs, the device further includes:

[0253] The face image determination module is configured to determine the previous face image corresponding to the face image;

[0254] The target information acquisition module is also configured to acquire the second tooth morphology change information corresponding to the previous face image;

[0255] The morphological change determination module is specifically configured to perform constraint analysis on the positional morphological fusion error corresponding to the tooth position information based on the fifth preset constraint condition, so as to obtain the first tooth morphological change information.

[0256] The fifth preset constraint condition refers to the condition that constrains the positional morphology fusion error to reach the minimum value. The positional morphology fusion error is obtained by fusing the positional morphology error and the morphological change range deviation. The morphological change range deviation refers to the deviation between the first tooth morphological change information and the second tooth morphological change information. The first tooth morphological change information is determined when the positional morphology fusion error reaches the minimum value.

[0257] In some possible designs, the face image is obtained by capturing the target face using a monocular camera.

[0258] In summary, the technical solution provided by the embodiments of this disclosure minimizes the positional error between the actual and predicted positions of the teeth by setting preset constraints. When the positional error is minimized, the corresponding tooth pose change can be determined. Then, based on the tooth pose change and the initial tooth pose, the tooth can be determined in the corresponding target pose. This allows the reconstructed 3D face to represent the tooth information of the real face, improves the consistency between the tooth pose of the 3D face and the tooth pose of the real face, and enhances the accuracy of tooth information and the realism of the 3D face overall.

[0259] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0260] Figure 10 This is a block diagram illustrating an electronic device for face image processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a facial image processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0261] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0262] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the face image processing method as described in the embodiments of this disclosure.

[0263] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the face image processing method of the present disclosure embodiments.

[0264] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the face image processing method of the present disclosure embodiments.

[0265] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0266] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0267] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A face image processing method, characterized in that, The method includes: The system acquires spatial transformation information, facial pose change information corresponding to a facial image, tooth position information corresponding to teeth in the facial image, and initial tooth pose information of the three-dimensional tooth model corresponding to the teeth in a three-dimensional coordinate system. The spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system corresponding to the facial image, and the facial pose change information represents the pose change of the three-dimensional facial model corresponding to the facial image in the three-dimensional coordinate system. Based on the first preset constraint condition, a constraint analysis is performed on the target position error corresponding to the tooth position information to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model; wherein, the first preset constraint condition refers to the condition that constrains the target position error to reach the minimum value, the target position error is the error between the tooth position information and the first predicted tooth position information, the first predicted tooth position information is the two-dimensional tooth position information corresponding to the first tooth pose information in the three-dimensional coordinate system after transforming the first tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the first tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information, and the first tooth pose change information is determined under the condition that the target position error reaches the minimum value; Based on the first tooth pose change information and the face pose change information, the initial tooth pose information is offset to obtain the target tooth pose information, which is used to characterize the addition position corresponding to the tooth effect.

2. The method according to claim 1, characterized in that, The teeth include upper teeth, the three-dimensional tooth model includes a maxillary tooth model corresponding to the upper teeth, the initial tooth pose information includes initial upper tooth pose information, the tooth position information includes upper tooth position information, and the step of performing constraint analysis on the target position error corresponding to the tooth position information based on a first preset constraint condition to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model includes: Based on the first preset constraint, constraint analysis is performed on the upper tooth position error corresponding to the upper tooth position information to obtain the maxillary bone pose change information of the maxillary tooth model relative to the three-dimensional face model. Wherein, the first tooth pose change information includes the maxillary pose change information, the target position error includes the upper tooth position error, the upper tooth position error is the error between the upper tooth position information and the predicted upper tooth position information, the predicted upper tooth position information is the two-dimensional upper tooth position information corresponding to the target upper tooth pose information in the three-dimensional coordinate system after transforming the target upper tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the target upper tooth pose information is the upper tooth pose information after offsetting the initial upper tooth pose information based on the maxillary pose change information and the face pose change information, and the maxillary pose change information is determined when the upper tooth position error reaches the minimum value.

3. The method according to claim 2, characterized in that, The maxillary pose change information includes maxillary rotation information and maxillary displacement information of the maxillary tooth model relative to the three-dimensional face model. The step of offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information includes: Based on the facial pose change information, the maxillary bone rotation information, and the maxillary bone displacement information, the initial maxillary tooth pose information is offset to obtain the target maxillary tooth pose information.

4. The method according to claim 3, characterized in that, The teeth also include lower teeth, and the three-dimensional tooth model also includes a mandibular tooth model corresponding to the lower teeth. The initial tooth pose information also includes initial lower tooth pose information, and the tooth position information also includes lower tooth position information. Based on a first preset constraint condition, constraint analysis is performed on the target position error corresponding to the tooth position information to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model, including: Based on the first preset constraint, the lower tooth position error corresponding to the lower tooth position information is constrained and analyzed to obtain the mandibular pose change information of the mandibular tooth model relative to the three-dimensional face model. Wherein, the first tooth pose change information includes the mandibular pose change information, the target position error includes the mandibular position error, the mandibular position error is the error between the mandibular position information and the predicted mandibular position information, the predicted mandibular position information is the two-dimensional mandibular position information corresponding to the target mandibular pose information in the three-dimensional coordinate system after transforming the target mandibular pose information of the mandibular tooth into the two-dimensional coordinate system based on the spatial transformation information, the target mandibular pose information is the mandibular pose information after offsetting the initial mandibular pose information based on the mandibular pose change information and the face pose change information, and the mandibular pose change information is determined when the mandibular position error reaches the minimum value.

5. The method according to claim 4, characterized in that, The mandibular pose change information includes mandibular rotation information and mandibular displacement information of the mandibular tooth model relative to the three-dimensional face model. The step of offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information includes: Based on the facial pose change information, the mandibular rotation information, and the mandibular displacement information, the initial mandibular pose information is offset to obtain the target mandibular pose information.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the tooth pose change range information corresponding to the three-dimensional tooth model, and the tooth pose change range information is used to characterize the pose change range of the three-dimensional tooth model relative to the three-dimensional face model; The step of performing constraint analysis on the target position error corresponding to the tooth position information based on the first preset constraint condition to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model includes: Based on the second preset constraint condition, the first fusion position error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth pose change information. The second preset constraint condition refers to the condition that constrains the first fusion position error to reach the minimum value. The fusion position error is obtained by fusing the target position error and the pose change range deviation. The pose change range deviation refers to the deviation between the first tooth pose change information and the tooth pose change range information. The first tooth pose change information is determined when the first fusion position error reaches the minimum value.

7. The method according to claim 6, characterized in that, The method further includes: Determine the previous face image corresponding to the given face image; Obtain the second tooth pose change information corresponding to the previous face image; The first fusion position error corresponding to the tooth position information is constrained and analyzed based on the second preset constraint condition to obtain the first tooth pose change information, including: Based on the third preset constraint condition, the second fusion position error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth pose change information. The third preset constraint condition refers to the condition that constrains the second fusion position error to reach the minimum value. The second fusion position error is obtained by fusing the first fusion position error and the pose motion range deviation. The pose motion range deviation refers to the deviation between the first tooth pose change information and the second tooth pose change information. The first tooth pose change information is determined when the second fusion position error reaches the minimum value.

8. The method according to any one of claims 1 to 7, characterized in that, Before obtaining the target tooth pose information by shifting the initial tooth pose information based on the first tooth pose change information and the face pose change information, the method further includes: Based on the fourth preset constraint condition, the positional morphology error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth morphology change information corresponding to the tooth; wherein, the fourth preset constraint condition refers to the condition that constrains the positional morphology error to reach the minimum value, the first tooth morphology change information represents the morphological change between the tooth morphology and the tooth morphology corresponding to the tooth in the three-dimensional tooth model, the positional morphology error is the error between the tooth position information and the second predicted tooth position information, the second predicted tooth position information is the two-dimensional tooth position information corresponding to the second tooth pose information corresponding to the tooth in the three-dimensional coordinate system after transforming it to the two-dimensional coordinate system based on the spatial transformation information, the second tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the face pose change information, the first tooth pose change information and the first tooth morphology change information, and the first tooth morphology change information is determined under the condition that the positional morphology error reaches the minimum value; The step of shifting the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain the target tooth pose information includes: Based on the facial pose change information, the first tooth pose change information, and the first tooth morphology change information, the initial tooth pose information is offset to obtain the target tooth pose information.

9. The method according to claim 8, characterized in that, The method further includes: Determine the previous face image corresponding to the given face image; Obtain the second tooth morphology change information corresponding to the previous face image; The step of performing constraint analysis on the positional morphology error corresponding to the tooth position information based on the fourth preset constraint condition to obtain the first tooth morphology change information corresponding to the tooth includes: Based on the fifth preset constraint condition, the position morphology fusion error corresponding to the tooth position information is constrained and analyzed to obtain the first tooth morphology change information. The fifth preset constraint condition refers to the condition that constrains the positional morphology fusion error to reach the minimum value. The positional morphology fusion error is obtained by fusing the positional morphology error and the morphological change range deviation. The morphological change range deviation refers to the deviation between the first tooth morphological change information and the second tooth morphological change information. The first tooth morphological change information is determined when the positional morphology fusion error reaches the minimum value.

10. A face image processing device, characterized in that, The device includes: The target information acquisition module is configured to acquire spatial transformation information, facial pose change information corresponding to a facial image, tooth position information of the teeth in the facial image, and initial tooth pose information of the three-dimensional tooth model corresponding to the teeth in the three-dimensional coordinate system. The spatial transformation information represents the coordinate system transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system corresponding to the facial image, and the facial pose change information represents the pose change of the three-dimensional facial model corresponding to the facial image in the three-dimensional coordinate system. The pose change determination module is configured to perform constraint analysis on the target position error corresponding to the tooth position information based on a first preset constraint condition, to obtain the first tooth pose change information of the three-dimensional tooth model relative to the three-dimensional face model; wherein, the first preset constraint condition refers to the condition that constrains the target position error to reach a minimum value, the target position error is the error between the tooth position information and the first predicted tooth position information, the first predicted tooth position information is the two-dimensional tooth position information corresponding to the first tooth pose information in the three-dimensional coordinate system after transforming the first tooth pose information in the three-dimensional coordinate system to the two-dimensional coordinate system based on the spatial transformation information, the first tooth pose information is the tooth pose information after offsetting the initial tooth pose information based on the first tooth pose change information and the face pose change information, and the first tooth pose change information is determined under the condition that the target position error reaches a minimum value; The tooth pose determination module is configured to perform an offset on the initial tooth pose information based on the first tooth pose change information and the face pose change information to obtain target tooth pose information, wherein the target tooth pose information is used to characterize the addition position corresponding to the tooth effect.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the face image processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the face image processing method as described in any one of claims 1 to 9.

Images