Tooth design effect previewing method and device, electronic equipment and storage medium

Abstract

The invention provides a tooth design effect previewing method and device, electronic equipment and a storage medium. The method comprises the steps that user data and a reference tooth model are acquired, the user data comprise an original tooth model, a face model, an initial face image and a face point cloud corresponding to the initial face image of a user, and the original tooth model and the face model are both aligned with the reference tooth model; determining a target conversion relation of the original tooth model relative to the initial face image based on alignment processing of the face model and the face point cloud and the original tooth model; and rendering a preset tooth model in the reference tooth model to a preset tooth area in the initial face image by using the target conversion relationship to obtain a target face image displaying a design effect of the preset tooth model. By means of the method, the tooth state can be stably tracked, a user can accurately know the repairing effect of the corresponding tooth before repairing is carried out, and therefore the preview effect of the tooth design effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional scanning technology, and in particular to a method, device, electronic device and storage medium for previewing tooth design effects. Background Art

[0002] As people pay more and more attention to oral health and dental aesthetics, more and more patients hope to improve their imperfect tooth shape through aesthetic restorations. Before performing dental aesthetic restorations, showing patients the restored results in advance is crucial to improving patient satisfaction and ensuring the accuracy of the restoration plan.

[0003] In the preview scheme of the tooth design effect in the related art, the designed tooth model is often directly superimposed on the patient's facial image to simulate the effect after restoration. However, this direct superposition and replacement method has certain limitations. On the one hand, the directly superimposed tooth model may lack coordination with the surrounding teeth, gums and facial features, resulting in a distorted preview effect and a lack of realism. On the other hand, this simple superposition method is difficult to ensure the tracking stability of the tooth state, especially when the patient's facial image only reveals a small part of the teeth. The restored tooth model may not be naturally integrated into the patient's actual facial dynamic expressions or observations from different angles, thus affecting the preview effect. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method, device, electronic device and storage medium for previewing tooth design effects, which can solve the technical problem of poor preview effect of tooth design effects.

[0005] On the one hand, the present application provides a method for previewing dental design effects, the method comprising: obtaining user data and a reference dental model, wherein the user data comprises the user's original dental model, facial model, initial facial image and a facial point cloud corresponding to the initial facial image, the original dental model and the facial model are both aligned with the reference dental model, based on the alignment processing of the facial model and the facial point cloud and the original dental model, determining a target transformation relationship of the original dental model relative to the initial facial image, and using the target transformation relationship, rendering the preset dental model in the reference dental model to the preset dental area in the initial facial image, to obtain a target facial image showing the design effect of the preset dental model.

[0006] In some embodiments of the present application, the target transformation relationship of the original dental model relative to the initial facial image is determined based on the alignment processing of the facial model and the facial point cloud and the original dental model, including: aligning the rigid area between the facial model and the facial point cloud to obtain the initial transformation relationship of the original dental model relative to the initial facial image, using the initial transformation relationship to align the original dental model to the facial point cloud to obtain the aligned dental model, and based on the aligned dental model and the facial point cloud, executing the update process of the initial transformation relationship until the update meets the preset conditions, and then determining the target transformation relationship based on the obtained updated initial transformation relationship.

[0007] In some embodiments of the present application, the update process includes: constructing a point cloud constraint based on the facial point cloud and the aligned dental model, wherein the point cloud constraint includes a first point cloud constraint corresponding to the aligned maxillary model in the aligned dental model and a second point cloud constraint corresponding to the aligned mandibular model in the aligned dental model; constructing a tooth contour constraint based on the initial facial image, the aligned dental model and the shooting internal parameters corresponding to the initial facial image, wherein the tooth contour constraint includes a first tooth contour constraint corresponding to the aligned maxillary model and a second tooth contour constraint corresponding to the aligned mandibular model; and determining the updated initial conversion relationship based on the point cloud constraint and the tooth contour constraint.

[0008] In some embodiments of the present application, the method for constructing the first point cloud constraint includes: determining the second coordinate point in the facial point cloud corresponding to the first coordinate point in the aligned maxillary model based on the matching of the aligned maxillary model and the facial point cloud, and constructing the first point cloud constraint based on the first coordinate point, the second coordinate point and the normal vector of the second coordinate point.

[0009] In some embodiments of the present application, the method for constructing the first tooth contour constraint includes: using the model pose of the aligned maxillary model and the shooting internal parameters, rendering the aligned maxillary model to a two-dimensional pixel plane to obtain a first rendered image, extracting the first tooth contour in the first rendered image, extracting the second tooth contour in the initial facial image, determining the second contour point in the second tooth contour corresponding to the first contour point in the first tooth contour by matching the first tooth contour with the second tooth contour, and constructing the first tooth contour constraint according to the first coordinate point corresponding to the second contour point and the first contour point in the aligned maxillary model.

[0010] In some embodiments of the present application, the target transformation relationship includes a first target transformation relationship corresponding to the aligned maxillary model and a second target transformation relationship corresponding to the aligned mandibular model, and the target transformation relationship is used to render the preset tooth model in the reference tooth model to the preset tooth area in the initial facial image to obtain a target facial image showing the design effect of the preset tooth model, including: determining the intraoral area in the initial facial image according to the second tooth contour, adjusting the pixel value of the preset tooth area in the intraoral area to obtain the adjusted intraoral area, rendering the preset tooth model in the maxillary model of the reference tooth model to the two-dimensional pixel plane using the first target transformation relationship and the shooting internal reference, and / or rendering the preset tooth model in the mandibular model of the reference tooth model to the two-dimensional pixel plane using the second target transformation relationship and the shooting internal reference to obtain a second rendered image, and generating the target facial image according to the adjusted intraoral area and the second rendered image.

[0011] In some embodiments of the present application, during the rendering process of the second rendered image, the first part of the teeth in the preset tooth model that is not obscured by the original tooth model corresponding to the non-preset tooth model and the second part of the teeth that are obscured by the corresponding original tooth model are determined from the reference tooth model, and according to the target conversion relationship corresponding to the preset tooth model, the first area corresponding to the first part of the teeth in the two-dimensional pixel plane is colored, and the pixel values ​​of the second area corresponding to the second part of the teeth in the two-dimensional pixel plane and the area corresponding to the non-preset tooth model in the two-dimensional pixel plane are adjusted to obtain the second rendered image.

[0012] In some embodiments of the present application, the method further includes: generating an updated target facial image based on the target facial image using a preset generation model, or generating a video based on multiple updated target facial images.

[0013] On the other hand, the present application provides a tooth design effect device, which includes: an acquisition module for acquiring user data and a reference tooth model, wherein the user data includes the user's original tooth model, facial model, initial facial image and the facial point cloud corresponding to the initial facial image, and the original tooth model and the facial model are both aligned with the reference tooth model; a determination module for determining the target transformation relationship of the original tooth model relative to the initial facial image based on the alignment processing of the facial model and the facial point cloud and the original tooth model; a rendering module for using the target transformation relationship to render the preset tooth model in the reference tooth model to the preset tooth area in the initial facial image, and obtain a target facial image showing the design effect of the preset tooth model. On the other hand, the present application provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the tooth design effect preview method.

[0014] On the other hand, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor in an electronic device, the method for previewing the tooth design effect is implemented.

[0015] In the dental design effect preview solution provided in the embodiment of the present application, since the facial point cloud and the facial model have the same or similar user facial information, and the original dental model is aligned with the facial model, the conversion relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be accurately obtained as the initial conversion relationship by aligning the facial model and the facial point cloud, so that the original dental model can be aligned to the facial point cloud according to the initial conversion relationship. Since the aligned dental model and the facial point cloud are in the same camera coordinate system, the point cloud constraint is introduced by aligning the dental model and the facial point cloud, so that a more stable and accurate dental posture tracking effect can be achieved, and then the conversion relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be accurately determined to determine the target conversion relationship. Because the target transformation accurately represents the transformation between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud, the reference dental model is aligned with the original dental model, and the transformation between the camera coordinate system and the pixel coordinate system corresponding to the initial facial image is established based on the internal camera parameters of the initial facial image (also known as "camera intrinsic parameters"). Therefore, using this target transformation and the internal camera parameters, the preset dental model in the reference dental model can be accurately rendered or projected onto the preset dental area of ​​the initial facial image. This allows users to accurately understand the restoration effect before performing the restoration, thereby improving the preview of the dental design. Furthermore, by customizing the preset dental model in the reference dental model for display of aesthetic restorations and other design effects, as well as the preset dental areas in the initial facial image for modification such as replacement or updating, the display flexibility of the dental design can be increased. When the preset dental area is part of the dental area in the initial facial image, the original dental information of the non-preset dental area of ​​the initial facial image can be retained and reused in the target facial image, thereby improving the authenticity of the target facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a method for previewing tooth design effects provided in one embodiment of the present application.

[0017] Figure 2 Schematic diagram of the original tooth model provided in one embodiment of the present application.

[0018] Figure 3 This is a front view of a facial model provided in one embodiment of the present application.

[0019] Figure 4 Schematic diagram of a facial point cloud provided in one embodiment of the present application.

[0020] Figure 5 is a schematic diagram of a misaligned reference tooth model provided in one embodiment of the present application.

[0021] Figure 6 This is a schematic diagram of the alignment effect between the reference tooth model and the original tooth model provided in one embodiment of the present application.

[0022] Figure 7 This is a schematic diagram of an original tooth model provided in another embodiment of the present application.

[0023] Figure 8 This is a schematic diagram of the alignment effect of the original tooth model and the facial model provided in one embodiment of the present application.

[0024] Figure 9 Schematic diagram of a rigid body region provided in one embodiment of the present application.

[0025] Figure 10 Schematic diagram of an initial facial image provided in one embodiment of the present application.

[0026] Figure 11 is a schematic diagram of a target facial image provided in one embodiment of the present application.

[0027] Figure 12 This is a flowchart of a method for updating an initial conversion relationship provided in an embodiment of the present application.

[0028] Figure 13 Schematic diagram of tooth contour provided in one embodiment of the present application.

[0029] Figure 14 is a schematic diagram of a first rendered image provided in an embodiment of the present application.

[0030] Figure 15 This is a correspondence diagram between the first contour points and the second contour points provided in an embodiment of the present application.

[0031] Figure 16 This is a flowchart of a method for generating a target facial image provided by an embodiment of the present application.

[0032] Figure 17 Schematic diagram of the mouth area of ​​a target facial image provided in one embodiment of the present application.

[0033] Figure 18 This is a functional module diagram of a tooth design effect preview device provided in one embodiment of the present application.

[0034] Figure 19 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] It should be noted that, in this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A alone, A and B together, and B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," and so on (if any) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or precedence.

[0036] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner. The following embodiments and features in the embodiments may be combined with each other unless there is a conflict.

[0037] As people pay more and more attention to oral health and dental aesthetics, more and more patients hope to improve their imperfect tooth shape through aesthetic restorations. Before performing dental aesthetic restorations, showing patients the restored results in advance is crucial to improving patient satisfaction and ensuring the accuracy of the restoration plan.

[0038] In the preview scheme of the tooth design effect in the related art, the designed tooth model is often directly superimposed on the patient's facial image to simulate the effect after restoration. However, this direct superposition and replacement method has certain limitations. On the one hand, the directly superimposed tooth model may lack coordination with the surrounding teeth, gums and facial features, resulting in a distorted preview effect and a lack of realism. On the other hand, this simple superposition or only using 2D image information as a constraint method is difficult to ensure the tracking stability of the tooth state, especially when the patient's facial image only reveals a small part of the teeth. The restored tooth model may not be naturally integrated into the patient's actual facial dynamic expressions or observations from different angles, thereby affecting the preview effect.

[0039] To solve this technical problem, an embodiment of the present application provides a method for previewing tooth design effects, which can stably track the status of teeth, so that users can accurately understand the restoration effect of the corresponding teeth before implementing the restoration, thereby providing a preview effect of the tooth design and improving the accuracy of tooth restoration, thereby enhancing the user experience.

[0040] The tooth design effect preview method provided in the embodiment of the present application can be applied to one or more electronic devices. The electronic devices can be laptops, mobile phones, computers, servers and other devices. The present application does not limit the type of electronic devices.

[0041] like Figure 1 The figure is a flow chart of a method for previewing tooth design effects provided by an embodiment of the present application. According to different requirements, the order of each step in the flow chart can be adjusted according to actual requirements, and some steps can be omitted. The method is applied to electronic devices, such as Figure 19 An electronic device 10 is shown.

[0042] S11, obtaining user data and a reference tooth model, wherein the user data includes the user's original tooth model, facial model, initial facial image, and facial point cloud corresponding to the initial facial image.

[0043] In some embodiments of the present application, the original tooth model and the facial model can be a three-dimensional mesh model of the same user. The original tooth model can reflect the user's dental state in three-dimensional space, the facial model can reflect the user's facial state in three-dimensional space, the initial facial image can reflect the user's facial state in a two-dimensional pixel plane, and the facial point cloud can reflect the spatial coordinates of the user's face in three-dimensional space. Figure 2 As shown in FIG, it is a schematic diagram of the original tooth model provided by an embodiment of the present application. Figure 3 As shown in FIG, it is a front view of the facial model provided by an embodiment of the present application. Figure 4 , which is a schematic diagram of a facial point cloud provided in one embodiment of the present application. Figure 3 The facial model shown is a stereoscopic model in three-dimensional space. Figure 3 This is the face model under the normal view.

[0044] In some embodiments, an electronic device can obtain user data through various methods. For example, the electronic device can scan or photograph the user's oral cavity using an oral scanning device or camera (e.g., an intraoral scanner, an extraoral scanner, a tablet computer, or a mobile phone), and reconstruct the oral scan or photographic data to obtain an original dental model. Alternatively, the electronic device can scan the user's face using a facial scanning device (e.g., a scanner), and reconstruct the facial scan data to obtain a facial model. A facial scanning device or camera (e.g., a facial scanner, a tablet computer, or a mobile phone) can scan or photograph the user's face and reconstruct a point cloud, or use a lidar to collect a point cloud. Both RGB images and point cloud data, namely, the initial facial image and facial point cloud, can be obtained from the same device. This allows for the acquisition of a video of the user's current free expression and a 3D facial point cloud sequence, namely, an RGBD sequence. Each frame contains an RGB image and a point cloud file (depth image file or point cloud file). In this sequence, the user's expression can be a series of any actions, such as smiling, opening their mouth, or speaking. For example, an electronic device scans and captures a user's face using a sensor for collecting spatial information (such as a color camera or a monochrome camera) or other three-dimensional visual sensor, thereby obtaining a facial image and a corresponding facial point cloud. Because the facial point cloud and the facial image are acquired by the same device, the coordinate system corresponding to the facial point cloud is the device's camera coordinate system.

[0045] The above examples of user data acquisition methods are merely illustrative and are not intended to be limiting in practical applications. For example, the original dental model can be constructed using a simulation algorithm, modified and adjusted from a previously scanned dental model, or reconstructed from corresponding point cloud data or a series of image frames. The facial model can also be constructed using a simulation algorithm, modified and adjusted from a previously scanned facial model, or reconstructed from corresponding point cloud data or a series of image frames.

[0046] In some embodiments, the reference tooth model can be a model designed for the user that can display the design effects of the restored tooth shape, tooth size, tooth color, etc., and can correspond to one or more teeth. Each tooth model in the reference tooth model has a corresponding area in the original tooth model. Since the reference tooth model corresponds to the restored tooth model, the shape, size, etc. of each tooth model in the reference tooth model may be different from the tooth model in the corresponding area in the original tooth model. The present application does not limit the method for obtaining the reference tooth model. For example, the electronic device can read the three-dimensional data under a preset path as a reference tooth model. Each tooth model in the original tooth model and the reference tooth model can have a corresponding identifier (such as a number or serial number) to facilitate the distinction between the original tooth model and the various tooth models in the reference tooth model.

[0047] In some embodiments, the original dental model and the facial model are aligned with the reference dental model. This alignment can be achieved by placing the original dental model, the facial model, and the reference dental model in the same coordinate system, with the dental bases of the original dental model and the reference dental model aligned. The electronic device can align the original dental model, the facial model, and the reference dental model in a variety of ways, and this application does not limit the method of model alignment.

[0048] For example, the electronic device can use a preset registration algorithm, such as the Iterative Closest Point (ICP) algorithm. For example, if teeth are exposed in the facial model, the exposed teeth can be used to align them with the original dental model. Furthermore, the reference dental model is aligned with the original dental model's coordinate system, i.e., the base of the jaw is aligned, with only the dentition morphology (e.g., digital smile design) or posture (e.g., orthodontic design) being changed. This allows alignment of the original dental model, the facial model, and the reference dental model. For example, the electronic device can obtain a three-dimensional model containing dental and facial information, such as a computed tomography (CT) model, and align the three-dimensional model with the original dental model using the ICP algorithm, aligning the three-dimensional model with the original dental model's coordinate system. The facial model can then be aligned with the three-dimensional model. Alignment of the original dental model with the facial model is achieved by aligning the three-dimensional model with the original dental model and then with the facial model. Furthermore, the reference dental model is aligned with the original dental model's coordinate system, i.e., the base of the jaw is aligned, with only the dentition morphology (e.g., digital smile design) or posture (e.g., orthodontic design) being changed. In this way, the original tooth model, the facial model, and the reference tooth model are aligned by using the three-dimensional model with the tooth information and the facial information as the intermediate data. The above-mentioned example of the alignment method of the model is only an example and is not limited to this in actual application.

[0049] like Figure 5 As shown in FIG, it is a schematic diagram of an unaligned reference tooth model provided by an embodiment of the present application. Figure 6 As shown in FIG, it is a schematic diagram of the alignment effect of the reference tooth model and the original tooth model provided by an embodiment of the present application. Figure 7 As shown in FIG, it is a schematic diagram of the original tooth model provided by another embodiment of the present application. Figure 8 , which is a schematic diagram of the alignment effect of the original tooth model and the facial model provided in one embodiment of the present application. Figure 5 The reference tooth model in Figure 2 The original tooth models shown correspond one to one, Figure 6 The tooth model in is the superposition of the original tooth model and the reference tooth model. Figure 5 , Figure 6The tooth morphology of the reference tooth model after alignment has changed. Figure 8 For the general Figure 7 The original tooth model shown is the same as Figure 3 The face model shown is aligned with the model, Figure 8 The model shown has Figure 7 The morphology of the original tooth model shown and Figure 3 The morphology of the facial model shown.

[0050] S12, based on the alignment processing of the facial model and the facial point cloud and the original tooth model, determining a target transformation relationship of the original tooth model relative to the initial facial image.

[0051] In some embodiments of the present application, the electronic device determines the target transformation relationship of the original dental model relative to the initial facial image based on the alignment processing of the facial model and the facial point cloud and the original dental model, including: aligning the rigid body area between the facial model and the facial point cloud, obtaining the initial transformation relationship of the original dental model relative to the initial facial image, using the initial transformation relationship to align the original dental model to the facial point cloud, obtaining the aligned dental model, and executing an update process of the initial transformation relationship based on the aligned dental model and the facial point cloud until the update meets the preset conditions, and then determining the target transformation relationship based on the obtained updated initial transformation relationship.

[0052] The rigid area can be an area of ​​the user that is not easily affected by facial expressions, for example, the rigid area can be the nose bridge and forehead area. For example, the electronic device can use a feature detection algorithm to determine the feature points of the facial model and the facial point cloud (for example, the nose vertex, the eye corner point, etc.), and based on the feature points, segment the nose bridge area and the forehead area from the facial model and the facial point cloud as the rigid area. For example, Figure 9 , which is a schematic diagram of a rigid body area provided in one embodiment of the present application. Figure 9 The rigid body area shown is the rigid body area of ​​the facial point cloud, including the forehead area and the nose area.

[0053] The electronic device can use an ICP algorithm or the like to align the rigid body region of the facial model with the rigid body region of the facial point cloud to obtain a transformation relationship between the world coordinate system corresponding to the facial model and the camera coordinate system corresponding to the facial point cloud. Since the original dental model and the facial model are in the same coordinate system after alignment, a transformation relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be obtained. The electronic device can determine the obtained transformation relationship as the initial transformation relationship of the original dental model relative to the initial facial image. The initial transformation relationship may include a rotation matrix and a translation vector, and may be a 4×4 matrix.

[0054] The electronic device can use the initial transformation relationship to calculate the coordinates of each 3D point of the original tooth model in the original world coordinate system, and obtain the coordinates of each 3D point in the camera coordinate system corresponding to the facial point cloud, thereby completing the alignment of the original tooth model with the facial point cloud. For example, the electronic device can calculate each 3D point in the original tooth model using the following formula (1): ; (1) in, Represents the coordinates of each 3D point in the original tooth model, Represents the coordinates of the 3D point in the camera coordinate system corresponding to the facial point cloud, Represents the rotation matrix in the initial transformation relationship, represents the translation vector in the initial transformation relationship, Indicates the serial index of the initial facial image, for example, counting starts from 1, and the initial facial image is the second image obtained, It can be 2.

[0055] The preset conditions can be customized, and this application does not impose any restrictions on this. For example, the preset condition could be that the number of updates reaches a preset number, such as 3 or 5 updates. The above-mentioned number of updates reaching the preset number is merely an example condition for stopping the update of the initial conversion relationship and is not limited to this in actual applications. For example, when the difference between the updated initial conversion relationship and the previously updated initial conversion relationship falls within a preset range or when other convergence conditions are met, the update of the initial conversion relationship can be stopped.

[0056] When multiple updates are performed, multiple updated initial conversion relationships may be obtained, and the electronic device may determine one of the multiple updated initial conversion relationships as the target conversion relationship. For example, the electronic device may determine the final updated initial conversion relationship as the target conversion relationship.

[0057] In this embodiment, since the facial point cloud and the facial model have the same or similar user facial information, and the original dental model is aligned with the facial model, the transformation relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be accurately obtained by aligning the facial model with the facial point cloud. This transformation relationship is then used as the initial transformation relationship, thereby aligning the original dental model to the facial point cloud. Since the aligned dental model and the facial point cloud are in the same camera coordinate system, the point cloud constraints introduced by the aligned dental model and facial point cloud can achieve a more stable and accurate dental and jaw posture tracking effect, and thus the transformation relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be accurately determined to determine the target transformation relationship.

[0058] S13, using the target conversion relationship, rendering the preset tooth model in the reference tooth model to the preset tooth area in the initial facial image, to obtain a target facial image showing the design effect of the preset tooth model.

[0059] In some embodiments of the present application, the preset tooth model may be a partial tooth model in the reference tooth model, or may be all the tooth models in the reference tooth model, and the present application does not impose any restrictions on this. The preset tooth model may be obtained by the user flexibly selecting the corresponding tooth corresponding model from the reference tooth model based on the identifier of one or more tooth corresponding models in the reference tooth model. For example, the preset tooth model may include the complete maxillary model and mandibular model in the reference tooth model, or the preset tooth model may only include the maxillary model in the reference tooth model, or the preset tooth model may only include part of the teeth in the maxillary model of the reference tooth model, or the preset tooth model may only include the mandibular model in the reference tooth model, or the preset tooth model may only include part of the teeth in the mandibular model of the reference tooth model. The preset tooth region may be a part of the tooth region in the initial facial image, or may be all the tooth regions in the initial facial image. The preset tooth region may be determined by a user's custom setting.

[0060] In this embodiment, because the target transformation relationship accurately represents the transformation relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud, the reference dental model is aligned with the original dental model, and the transformation relationship between the camera coordinate system and the pixel coordinate system corresponding to the initial facial image can be established based on the internal shooting parameters of the initial facial image (also known as "camera internal parameters"). Therefore, using the target transformation relationship and the internal shooting parameters, the preset dental model in the reference dental model can be accurately rendered or projected onto the preset dental area of ​​the initial facial image, allowing the user to visualize the restoration effect before performing the restoration, thus achieving a preview of the dental design. By customizing the preset dental model in the reference dental model for design effects such as aesthetic restorations and the preset dental areas in the initial facial image for modification such as replacement or updating, the display flexibility of the dental design can be enhanced. Furthermore, when the preset dental area is part of the dental area in the initial facial image, the original dental information of the non-preset dental area of ​​the initial facial image can be retained in the target facial image, thereby enhancing the authenticity of the target facial image.

[0061] In other embodiments of the present application, the electronic device can generate an updated target facial image based on the target facial image using a preset generation model, and display the updated target facial image, or generate a video based on multiple updated target facial images, and play the generated video.

[0062] The present application does not limit the type of the preset generation model. For example, the preset generation model may be a Generative Adversarial Network (GAN) or a Diffusion Model.

[0063] In some embodiments, the tooth texture rendered in the target facial image may differ significantly from the tooth texture in the initial facial image, and may even include unknown texture holes in certain areas. Because the target facial image retains the original tooth information from the initial facial image's non-predetermined tooth regions, generating an updated target facial image using the predefined generative model provides an accurate foundation for the predefined generative model, further enhancing the authenticity of the updated target facial image.

[0064] In other embodiments of the present application, a target facial image showing the design effect of a preset dental model and an initial facial image of the original dental model before the dental design is performed can be simultaneously presented for comparative preview in the form of a single-frame image or video. Similarly, each updated target facial image can be previewed individually or compressed into a continuous video for preview. The updated target facial image can be displayed continuously or simultaneously with the target facial image before the update for comparative display, previewed in the form of a single frame or video.

[0065] like Figure 10 As shown in FIG, it is a schematic diagram of an initial facial image provided by an embodiment of the present application. Figure 11 , which is a schematic diagram of a target facial image provided by an embodiment of the present application. Figure 10 and Figure 11 By comparison, we can see that Figure 11 The teeth shown are compared to Figure 10 The teeth shown are more neat and beautiful.

[0066] In the dental design effect preview solution provided in the embodiment of the present application, since the facial point cloud and the facial model have the same or similar user facial information, and the original dental model is aligned with the facial model, the conversion relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be accurately obtained as the initial conversion relationship by aligning the facial model and the facial point cloud, so that the original dental model can be aligned to the facial point cloud according to the initial conversion relationship. Since the aligned dental model and the facial point cloud are in the same camera coordinate system, the point cloud constraint is introduced by aligning the dental model and the facial point cloud, so that a more stable and accurate dental posture tracking effect can be achieved, and then the conversion relationship between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud can be accurately determined to determine the target conversion relationship. Because the target transformation accurately represents the transformation between the world coordinate system corresponding to the original dental model and the camera coordinate system corresponding to the facial point cloud, the reference dental model is aligned with the original dental model, and the transformation between the camera coordinate system and the pixel coordinate system corresponding to the initial facial image is established based on the internal camera parameters of the initial facial image (also known as "camera intrinsic parameters"). Therefore, using this target transformation and the internal camera parameters, the preset dental model in the reference dental model can be accurately rendered or projected onto the preset dental area of ​​the initial facial image. This allows users to accurately understand the restoration effect before performing the restoration, thereby improving the preview of the dental design. Furthermore, by customizing the preset dental model in the reference dental model for display of aesthetic restorations and other design effects, as well as the preset dental areas in the initial facial image for modification such as replacement or updating, the display flexibility of the dental design can be increased. When the preset dental area is part of the dental area in the initial facial image, the original dental information of the non-preset dental area of ​​the initial facial image can be retained and reused in the target facial image, thereby improving the authenticity of the target facial image.

[0067] In some embodiments of the present application, Figure 12 FIG. 1 is a flow chart of a method for updating an initial conversion relationship provided by an embodiment of the present application, comprising the following steps: S121, constructing a point cloud constraint based on the facial point cloud and the aligned dental model, wherein the point cloud constraint includes a first point cloud constraint corresponding to the aligned maxillary model in the aligned dental model and a second point cloud constraint corresponding to the aligned mandibular model in the aligned dental model.

[0068] In some embodiments of the present application, the method for constructing the first point cloud constraint includes: determining the second coordinate point in the facial point cloud corresponding to the first coordinate point in the aligned maxillary model based on the matching of the aligned maxillary model and the facial point cloud, and constructing the first point cloud constraint based on the first coordinate point, the second coordinate point and the normal vector of the second coordinate point.

[0069] Among them, the first coordinate point and the second coordinate point of the embodiment of the present application are both points in three-dimensional space and have corresponding three-dimensional coordinates. The electronic device can match the aligned maxillary model with the facial point cloud through a variety of methods, thereby determining the second coordinate point corresponding to the first coordinate point in the aligned maxillary model in the facial point cloud. For example, the first coordinate point and the second coordinate point that match each other can be determined by calculating the distance (for example, Euclidean distance) between the first coordinate point in the aligned maxillary model and the second coordinate point corresponding to the facial point cloud, wherein the distance between the first coordinate point and the second coordinate point that match each other can be the minimum distance. The normal vector of the second coordinate point can be a vector perpendicular to the plane where the coordinate point is located.

[0070] The first point cloud constraint and the second point cloud constraint can be expressions related to the optimized pose. The optimized pose can represent the pose of the aligned tooth model in the converted initial coordinate system, which is updated based on the point cloud constraint and the tooth contour constraint. The transformation relationship corresponding to the optimized pose can be used to adjust the position and posture of the first coordinate point in the initial coordinate system, so that the adjusted first coordinate point approaches or approaches the corresponding second coordinate point in the facial point cloud. The initial coordinate system can be the camera coordinate system corresponding to the facial point cloud. According to the initial transformation relationship and the transformation relationship corresponding to the optimized pose, the updated initial transformation relationship of the original tooth model and the reference tooth model from the original world coordinate system to the camera coordinate system where the corresponding facial point cloud is located, that is, the target transformation relationship, can be determined. The transformation relationship corresponding to the optimized posture may include the transformation relationship corresponding to the first optimized posture corresponding to the aligned maxillary model and the second optimized posture corresponding to the aligned mandibular model, respectively. The transformation relationship corresponding to the first optimized posture and the transformation relationship corresponding to the second optimized posture both include a rotation matrix and a translation vector. The transformation relationship corresponding to the first optimized posture is used to determine the updated first initial transformation relationship corresponding to the aligned maxillary model below, and the transformation relationship corresponding to the second optimized posture is used to determine the updated second initial transformation relationship corresponding to the aligned mandibular model. Since the method for determining the transformation relationship corresponding to the second optimized posture is basically the same as the method for determining the transformation relationship corresponding to the first optimized posture, in order to clearly illustrate the method for generating the transformation relationship corresponding to the optimized posture, the transformation relationship corresponding to the first optimized posture will be illustrated below using the aligned maxillary model as an example.

[0071] For example, the first point cloud constraint can refer to the following formula (2): ; (2) in, represents the first point cloud constraint, Represents the rotation matrix in the transformation relationship corresponding to the first optimized pose, Indicates the coordinates of the first coordinate point, that is, the maxillary 3D model converted to the initial coordinate system through the initial transformation relationship The coordinates of the first point in , Represents the translation vector in the transformation relationship corresponding to the first optimized pose, represents the coordinates of the second coordinate point corresponding to the first coordinate point, Represents the normal vector of the second coordinate point, Indicates that the vector With normal vector Perform inner product operation.

[0072] The method for constructing the second point cloud constraint may refer to the above description of the method for constructing the first point cloud constraint.

[0073] In this embodiment, since the aligned maxillary model and the aligned mandibular model may move relative to each other, resulting in different updated initial transformation relationships of the aligned maxillary model and the aligned mandibular model relative to the initial facial image, a first point cloud constraint corresponding to the aligned maxillary model and a second point cloud constraint corresponding to the aligned mandibular model are calculated to respectively track the updated initial transformation relationships of the aligned maxillary model and the aligned mandibular model relative to the initial facial image. The residual between the first coordinate point in the aligned maxillary model and the corresponding second coordinate point in the facial point cloud is optimized by the first point cloud constraint. , it can update the transformation relationship corresponding to the first optimized posture, and reduce the differences in position, angle, etc. between the first coordinate point of the aligned maxillary model and the corresponding second coordinate point on the facial point cloud, and then determine the target transformation relationship to achieve a real, accurate and stable tracking alignment effect. S122, constructing tooth contour constraints based on the initial facial image, the aligned tooth model and the shooting internal parameters corresponding to the initial facial image, the tooth contour constraints including a first tooth contour constraint corresponding to the aligned maxillary model and a second tooth contour constraint corresponding to the aligned mandibular model.

[0074] In some embodiments of the present application, a method for constructing a first tooth contour constraint includes: using the model pose and shooting internal parameters of the aligned maxillary model to render the aligned maxillary model to a two-dimensional pixel plane to obtain a first rendered image, extracting the first tooth contour in the first rendered image, extracting the second tooth contour in the initial facial image, and determining the second contour point in the second tooth contour corresponding to the first contour point in the first tooth contour by matching the first tooth contour with the second tooth contour, and constructing the first tooth contour constraint based on the first coordinate point corresponding to the second contour point and the first contour point in the aligned maxillary model.

[0075] Among them, the model pose can be the position and posture of the aligned maxillary model in the initial coordinate system. The initial transformation relationship can include a rotation matrix and a translation vector, which can be a 4×4 matrix. The method for generating the first rendered image can refer to the description of the method for generating the second rendered image below. The electronic device can extract the first tooth contour and the second tooth contour through a variety of methods. This application does not limit the method for extracting the tooth contour. For example, the electronic device can use a detection tool to detect and extract the first tooth contour in the first rendered image and the second tooth contour in the initial facial image. The detection tool can be an mmdetection target detection tool, etc.

[0076] The first tooth profile constraint and the second tooth profile constraint may be expressions related to the transformation relationship corresponding to the optimized pose. For example, the calculation method of the first tooth profile constraint may refer to the following formula (3): ; (3) in, represents the first tooth profile constraint, Refers to the three-dimensional point coordinates in the current camera coordinate system , using the shooting internal parameters (camera internal parameters) to project to the two-dimensional pixel plane (the pixel coordinate system corresponding to the initial facial image), Indicates the coordinates of the first coordinate point corresponding to the first contour point in the aligned maxillary model, Represents the rotation matrix in the transformation relationship corresponding to the first optimized pose, Represents the translation vector in the transformation relationship corresponding to the first optimized pose, Indicates the coordinates of the second contour point corresponding to the first contour point.

[0077] The method for constructing the second tooth profile constraint may refer to the above description of the method for constructing the first tooth profile constraint.

[0078] For example, Figure 13 As shown in FIG, it is a schematic diagram of the tooth contour provided by an embodiment of the present application. Figure 14 As shown in FIG, it is a schematic diagram of a first rendered image provided by an embodiment of the present application. Figure 15 The figure shows the corresponding relationship between the first contour point and the second contour point provided by an embodiment of the present application. Figure 13 In the tooth contour, multiple colors are used to represent the tooth contour, and each tooth has a corresponding tooth contour. Figure 15 The lines of different colors are used to construct Figure 14 The corresponding relationship between the first contour points of the tooth area in the first rendered image and the second contour points of the tooth area in the initial facial image is shown.

[0079] In this embodiment, the residual between the first coordinate point corresponding to the first contour point of the tooth area in the first rendered image and the second contour point corresponding to the tooth area in the initial facial image after alignment is optimized by the first tooth contour constraint. , which can update the transformation relationship corresponding to the first optimized pose, and then determine the target transformation relationship to achieve a true and accurate alignment effect.

[0080] S123: Determine an updated initial transformation relationship based on the point cloud constraint and the tooth contour constraint.

[0081] In some embodiments of the present application, the electronic device may determine a first objective function based on the first point cloud constraint and the first tooth contour constraint, determine a second objective function based on the second point cloud constraint and the second tooth contour constraint, determine the transformation relationship corresponding to each of the first optimized pose and the second optimized pose by minimizing the first objective function and the second objective function, and determine an updated initial transformation relationship based on the initial transformation relationship, the transformation relationship corresponding to each of the first optimized pose and the second optimized pose. The updated initial transformation relationship may include an updated first initial transformation relationship corresponding to the aligned maxillary model and an updated second initial transformation relationship corresponding to the aligned mandibular model.

[0082] For example, the calculation method of the updated initial conversion relationship can refer to the following formulas (4) to (7): ; (4) ; (5) ; (6) ; (7) in, represents the first point cloud constraint, represents the first tooth profile constraint, represents the first objective function that minimizes the first point cloud constraint and the first tooth profile constraint, represents the second point cloud constraint, Indicates the second tooth profile constraint, represents the second objective function that minimizes the second point cloud constraint and the second tooth profile constraint, represents the rotation matrix in the updated first initial transformation relationship, represents the translation vector in the updated first initial transformation relationship, represents the rotation matrix in the updated second initial transformation relationship, represents the translation vector in the updated second initial transformation relationship, Represents the rotation matrix in the transformation relationship corresponding to the first optimized pose, Represents the translation vector in the transformation relationship corresponding to the first optimized pose, Represents the rotation matrix in the transformation relationship corresponding to the second optimized pose, Represents the translation vector in the transformation relationship corresponding to the second optimized pose.

[0083] In this embodiment, since the initial transformation relationship is the transformation relationship between the world coordinate system corresponding to the original tooth model and the world coordinate system corresponding to the facial point cloud, the first optimized pose can represent the pose of the aligned maxillary model in the camera coordinate system corresponding to the facial point cloud, updated based on the point cloud constraint and the tooth contour constraint, and the transformation relationship corresponding to the first optimized pose is used to adjust the position and posture of the first coordinate point in the aligned maxillary model in the initial coordinate system, so that the adjusted first coordinate point approaches or approaches the corresponding second coordinate point in the facial point cloud, and the second optimized pose can represent the pose of the aligned mandibular model in the camera coordinate system corresponding to the facial point cloud, updated based on the point cloud constraint and the tooth contour constraint, and the transformation relationship corresponding to the second ... The transformation relationship is used to adjust the position and posture of the first coordinate point in the mandibular model after alignment in the initial coordinate system, so that the adjusted first coordinate point tends to or approaches the corresponding second coordinate point in the facial point cloud. Therefore, the updated first initial transformation relationship is determined through the initial transformation relationship and the transformation relationship corresponding to the first optimized posture, and the updated second initial transformation relationship is determined through the initial transformation relationship and the transformation relationship corresponding to the second optimized posture, so that the updated first initial transformation relationship can accurately represent the transformation relationship between the world coordinate system where the original maxillary model is located and the camera coordinate system corresponding to the facial point cloud, and the updated second initial transformation relationship can accurately represent the transformation relationship between the world coordinate system where the original mandibular model is located and the camera coordinate system corresponding to the facial point cloud.

[0084] Since the first point cloud constraint can optimize the residual between the first coordinate point in the aligned maxillary model and the corresponding second coordinate point in the facial point cloud, and reduce the difference in position, angle, etc. between the first coordinate point of the aligned maxillary model and the corresponding second coordinate point on the facial point cloud, the second point cloud constraint can optimize the residual between the third coordinate point in the aligned mandibular model and the corresponding second coordinate point in the facial point cloud, and reduce the difference in position, angle, etc. between the third coordinate point of the aligned mandibular model and the corresponding second coordinate point on the facial point cloud, the first tooth contour constraint can optimize the first contour point corresponding to the tooth area in the first rendered image between the first coordinate point corresponding to the aligned maxillary model and the initial facial point cloud. The residual between the second contour points corresponding to the tooth area in the image, the second tooth contour constraint can optimize the residual between the first contour point corresponding to the tooth area in the first rendered image and the third coordinate point corresponding to the aligned mandibular model and the second contour point corresponding to the tooth area in the initial facial image. Therefore, the first objective function is determined according to the first point cloud constraint and the first tooth contour constraint, and the second objective function is determined according to the second point cloud constraint and the second tooth contour constraint, so that the first objective function can reflect the comprehensive registration accuracy of the aligned maxillary model, and the second objective function can reflect the comprehensive registration accuracy of the aligned mandibular model. By minimizing the first objective function and the second objective function, the optimized posture can be accurately determined.

[0085] In some embodiments of the present application, the target transformation relationship may include a first target transformation relationship corresponding to the aligned maxillary model and a second target transformation relationship corresponding to the aligned mandibular model, and the electronic device may generate a target facial image based on the first target transformation relationship and the second target transformation relationship. Figure 16 FIG. 1 is a flow chart of a method for generating a target facial image according to an embodiment of the present application, comprising the following steps: S131: Determine an intraoral area in the initial facial image according to the second tooth contour.

[0086] In some embodiments, the intraoral area may be an area within a graph formed by each second tooth contour. Figure 13 In the embodiment, the intraoral region is the region within the white teeth outline, which may also be referred to as the intralip region. The electronic device may segment the intraoral region from the initial facial image.

[0087] S132, in the intraoral area, adjusting the pixel value of the preset tooth area to obtain an adjusted intraoral area.

[0088] In some embodiments, the electronic device may adjust the pixel value of the preset tooth area to a preset value, for example, the preset value may be zero.

[0089] S133, using the first target conversion relationship and the shooting internal reference, the preset tooth model in the maxillary model of the reference tooth model is rendered to a two-dimensional pixel plane, and / or, using the second target conversion relationship and the shooting internal reference, the preset tooth model in the mandibular model of the reference tooth model is rendered to a two-dimensional pixel plane to obtain a second rendered image.

[0090] In some embodiments, during the rendering process of the second rendered image, the first part of the teeth in the preset tooth model that is not obscured by the original tooth model corresponding to the non-preset tooth model and the second part of the teeth that are obscured by the corresponding original tooth model are determined from the reference tooth model, and according to the target conversion relationship corresponding to the preset tooth model, the first area corresponding to the first part of the teeth in the two-dimensional pixel plane is colored, and the pixel values ​​of the second area corresponding to the second part of the teeth in the two-dimensional pixel plane and the area corresponding to the non-preset tooth model in the two-dimensional pixel plane are adjusted to obtain the second rendered image.

[0091] Among them, the maxillary model and the mandibular model of the reference dental model can respectively have corresponding preset dental models, and the dental models in the reference dental model and the original dental model can both have corresponding numbers or serial numbers. The electronic device determines the preset dental models in the maxillary model and the mandibular model of the reference dental model by receiving user input or automatically identified tooth numbers or serial numbers or selection areas.

[0092] The non-preset tooth models may be the remaining tooth models in the reference tooth model excluding the preset tooth models. For example, if the original tooth model and the reference tooth model include teeth 1 to 24, and the preset tooth model includes teeth 3 to 10, then the non-preset tooth models may be teeth 1, 2, and 11 to 24 in the reference tooth model, and the original tooth models corresponding to the non-preset tooth models are teeth 1, 2, and 11 to 24 in the original tooth model. Since each tooth model in the reference tooth model has a corresponding tooth model in the original tooth model, the electronic device can determine the spatial positions of teeth 3 to 10 in the preset tooth model and the spatial positions of teeth 1, 2, and 11 to 24 in the original tooth model, and determine whether teeth 3 to 10 in the preset tooth model are obscured by teeth 1, 2, and 11 to 24 in the original tooth model.

[0093] A first region corresponding to the first portion of teeth in the two-dimensional pixel plane can be colored according to the color of the first portion of teeth, thereby coloring the first region. Pixel values ​​of a second region corresponding to the second portion of teeth and a region corresponding to a non-preset tooth model in the two-dimensional pixel plane can be adjusted to a preset value (e.g., zero), thereby adjusting the pixel values.

[0094] The target transformation relationship corresponding to the preset tooth model in the maxillary model of the reference tooth model can be a first target transformation relationship, and the target transformation relationship corresponding to the preset tooth model in the mandibular model of the reference tooth model can be a second target transformation relationship. Since the first target transformation relationship and the second target transformation relationship are both transformation relationships between the world coordinate system corresponding to the original tooth model and the camera coordinate system corresponding to the facial point cloud, the original tooth model is aligned with the reference tooth model, and the transformation relationship between the camera coordinate system and the pixel coordinate system corresponding to the initial facial image can be established based on the shooting internal parameters of the initial facial image. Therefore, through the first target transformation relationship and the shooting internal parameters, the two-dimensional coordinates of the preset tooth model in the maxillary model of the reference tooth model in the two-dimensional pixel plane can be determined, so that the mapping area corresponding to the preset tooth model in the maxillary model of the reference tooth model composed of two-dimensional coordinates can be quickly determined. Similarly, through the second target transformation relationship and the shooting internal parameters, the mapping area corresponding to the preset tooth model in the mandibular model of the reference tooth model can be quickly determined.

[0095] In this embodiment, in a two-dimensional pixel plane, a preset tooth model is used to color the first area corresponding to the first part of the teeth in the mapping area, and the pixel values ​​of the second area corresponding to the second part of the teeth in the mapping area and other areas in the two-dimensional pixel plane except the mapping area are adjusted to obtain a second rendered image, so that the second rendered image not only retains the original facial information, but also retains the original tooth texture in the initial facial image, thereby improving the authenticity of the second rendered image.

[0096] S134: Generate a target facial image based on the adjusted intraoral area and the second rendered image.

[0097] In some embodiments, the electronic device may superimpose the intra-oral area with the second rendered image according to a preset rule to obtain a target facial image. The preset rule may include determining a target pixel point in the intra-oral area whose pixel value is a preset value, a target pixel value corresponding to the target pixel point in the second rendered image, superimposing the second rendered image on the intra-oral area, and adjusting the pixel value of the target pixel point in the superimposed intra-oral area from the preset value to the target pixel value, thereby obtaining the target facial image.

[0098] For example, Figure 17 , which is a schematic diagram of the mouth area of ​​the target facial image provided by an embodiment of the present application. Figure 17 In the figure, the gray area represents the area rendered by the preset tooth model, and the other areas retain the original tooth texture in the initial facial image.

[0099] like Figure 18The figure shows a functional module diagram of a tooth design effect preview device provided by an embodiment of the present application. The tooth design effect preview device 11 includes an acquisition module 110, a determination module 111 and a rendering module 112. The module / unit referred to in this application refers to a Figure 19 A series of computer readable instruction segments acquired by the processor 103 in the embodiment of the present invention and capable of completing fixed functions are stored in Figure 19 In the memory 102. In this embodiment, the function of each module / unit will be described in detail in subsequent embodiments.

[0100] The acquisition module 110 is used to acquire user data and a reference dental model, wherein the user data includes the user's original dental model, facial model, initial facial image, and facial point cloud corresponding to the initial facial image, and the original dental model and facial model are aligned with the reference dental model.

[0101] The determination module 111 is configured to determine a target transformation relationship of the original tooth model relative to the initial facial image based on the alignment processing of the facial model and the facial point cloud and the original tooth model.

[0102] In some embodiments, the determination module 111 is also used to align the rigid body area between the facial model and the facial point cloud, obtain the initial transformation relationship of the original dental model relative to the initial facial image, and use the initial transformation relationship to align the original dental model to the facial point cloud to obtain the aligned dental model. Based on the model posture, the aligned dental model and the facial point cloud, the update process of the initial transformation relationship is executed until the number of updates meets the preset conditions, and then the target transformation relationship is determined based on the updated initial transformation relationship.

[0103] In some embodiments, the determination module 111 is also used to construct a point cloud constraint based on the facial point cloud and the aligned dental model, wherein the point cloud constraint includes a first point cloud constraint corresponding to the aligned maxillary model in the aligned dental model and a second point cloud constraint corresponding to the aligned mandibular model in the aligned dental model; based on the initial facial image, the model posture and the shooting internal parameters corresponding to the initial facial image, a tooth contour constraint is constructed, wherein the tooth contour constraint includes a first tooth contour constraint corresponding to the aligned maxillary model and a second tooth contour constraint corresponding to the aligned mandibular model; and according to the point cloud constraint and the tooth contour constraint, the updated initial transformation relationship is determined.

[0104] In some embodiments, the determination module 111 is also used to determine the second coordinate point in the facial point cloud corresponding to the first coordinate point in the aligned maxillary model based on the matching of the aligned maxillary model and the facial point cloud, and construct a first point cloud constraint based on the first coordinate point, the second coordinate point and the normal vector of the second coordinate point.

[0105] In some embodiments, the determination module 111 is also used to use the model pose and shooting internal parameters of the aligned maxillary model to render the aligned maxillary model to a two-dimensional pixel plane, obtain a first rendered image, extract the first tooth contour in the first rendered image, extract the second tooth contour in the initial facial image, and determine the second contour point in the second tooth contour corresponding to the first contour point in the first tooth contour by matching the first tooth contour with the second tooth contour, and construct a first tooth contour constraint based on the first coordinate point corresponding to the second contour point and the first contour point in the aligned maxillary model.

[0106] The rendering module 112 is used to render the preset tooth model in the reference tooth model to the preset tooth area in the initial facial image using the target conversion relationship to obtain a target facial image showing the design effect of the preset tooth model.

[0107] In some embodiments, the target transformation relationship includes a first target transformation relationship corresponding to the aligned maxillary model and a second target transformation relationship corresponding to the aligned mandibular model.

[0108] In some embodiments, the rendering module 112 is also used to determine the intraoral area in the initial facial image based on the second tooth contour, adjust the pixel value of the preset tooth area in the intraoral area to obtain the adjusted intraoral area, and use the first target conversion relationship and the shooting internal parameters to render the preset tooth model in the maxillary model of the reference tooth model to a two-dimensional pixel plane, and / or use the second target conversion relationship and the shooting internal parameters to render the preset tooth model in the mandibular model of the reference tooth model to a two-dimensional pixel plane to obtain a second rendered image, and generate the target facial image based on the adjusted intraoral area and the second rendered image.

[0109] In which, during the rendering process of the second rendered image, the first part of the teeth in the preset tooth model that is not blocked by the original tooth model corresponding to the non-preset tooth model and the second part of the teeth blocked by the corresponding original tooth model are determined from the reference tooth model, and according to the target conversion relationship corresponding to the preset tooth model, the first area corresponding to the first part of the teeth in the two-dimensional pixel plane is colored, and the pixel values ​​of the second area corresponding to the second part of the teeth in the two-dimensional pixel plane and the area corresponding to the non-preset tooth model in the two-dimensional pixel plane are adjusted to obtain the second rendered image.

[0110] In some embodiments of the present application, the tooth design effect preview device 11 further includes a generation module 113 ( Figure 18 Not shown), the generation module 113 is used to generate an updated target facial image based on the target facial image using a preset generation model, or to generate a video based on multiple updated target facial images.

[0111] like Figure 19FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 10 may be a mobile phone, tablet computer, laptop computer, or other electronic device. The present embodiment does not impose any restrictions on the specific type of electronic device.

[0112] like Figure 19 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102, and the input / output interface 104 via the bus 105.

[0113] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB) and controller area network (CAN). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0114] Memory 102 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 103 and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0115] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by processor 103. Non-volatile memory can include disk storage devices and flash memory. For example, flash memory can be Nand Flash.

[0116] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the method for previewing the tooth design effect can be implemented on the electronic device 10.

[0117] In other embodiments, Figure 19 The electronic device 10 shown further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0118] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0119] The processor 103 provides computing and control capabilities. For example, the processor 103 is configured to execute a computer program stored in the memory 102 to implement the above-mentioned tooth design effect preview method.

[0120] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0121] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the input / output interface 104 in the electronic device 10 .

[0122] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0123] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0124] The computer-readable storage medium may be the internal memory of the electronic device described in the above embodiments, such as the hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device.

[0125] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0127] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0128] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0129] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0130] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in this application may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for previewing tooth design effects, characterized in that: The method comprises: Acquiring user data and a reference dental model, wherein the user data includes an original dental model, a facial model, an initial facial image, and a facial point cloud corresponding to the initial facial image of the user, and the original dental model and the facial model are aligned with the reference dental model; Determining a target transformation relationship of the original tooth model relative to the initial facial image based on the alignment processing of the facial model with the facial point cloud and the original tooth model; The target conversion relationship is used to render the preset tooth model in the reference tooth model to the preset tooth area in the initial facial image, thereby obtaining a target facial image showing the design effect of the preset tooth model.

2. The method for previewing tooth design effects according to claim 1, wherein: The step of determining a target transformation relationship between the original tooth model and the initial facial image based on the alignment processing between the facial model and the facial point cloud and the original tooth model includes: Aligning the rigid body region between the facial model and the facial point cloud to obtain an initial transformation relationship between the original tooth model and the initial facial image; aligning the original tooth model to the facial point cloud using the initial transformation relationship to obtain an aligned tooth model; Based on the aligned tooth model and the facial point cloud, an update process of the initial transformation relationship is executed until the update meets a preset condition, and then the target transformation relationship is determined based on the obtained updated initial transformation relationship.

3. The method for previewing tooth design effects according to claim 2, wherein: The update process includes: constructing a point cloud constraint based on the facial point cloud and the aligned dental model, wherein the point cloud constraint includes a first point cloud constraint corresponding to the aligned maxillary model in the aligned dental model and a second point cloud constraint corresponding to the aligned mandibular model in the aligned dental model; constructing tooth contour constraints based on the initial facial image, the aligned tooth model, and shooting internal parameters corresponding to the initial facial image, the tooth contour constraints including a first tooth contour constraint corresponding to the aligned maxillary model and a second tooth contour constraint corresponding to the aligned mandibular model; The updated initial transformation relationship is determined according to the point cloud constraint and the tooth contour constraint.

4. The method for previewing tooth design effects according to claim 3, wherein: The method for constructing the first point cloud constraint includes: According to the matching between the aligned maxillary model and the facial point cloud, determining a second coordinate point in the facial point cloud corresponding to a first coordinate point in the aligned maxillary model; The first point cloud constraint is constructed according to the first coordinate point, the second coordinate point, and the normal vector of the second coordinate point.

5. The method for previewing tooth design effects according to claim 3, wherein: The method for constructing the first tooth profile constraint comprises: Rendering the aligned maxillary model onto a two-dimensional pixel plane using the model pose of the aligned maxillary model and the shooting internal reference to obtain a first rendered image; extracting a first tooth contour in the first rendered image; extracting a second tooth contour in the initial facial image; By matching the first tooth contour with the second tooth contour, determining a second contour point in the second tooth contour corresponding to a first contour point in the first tooth contour; The first tooth contour constraint is constructed according to the first coordinate point corresponding to the second contour point and the first contour point in the aligned maxillary model.

6. The method for previewing tooth design effects according to claim 5, wherein: The target transformation relationship includes a first target transformation relationship corresponding to the aligned maxillary model and a second target transformation relationship corresponding to the aligned mandibular model. The target transformation relationship is used to render a preset tooth model in the reference tooth model to a preset tooth area in the initial facial image to obtain a target facial image showing a design effect of the preset tooth model, including: determining an intraoral region in the initial facial image based on the second tooth contour; In the intraoral area, adjusting the pixel value of the preset tooth area to obtain an adjusted intraoral area; Rendering the preset tooth model in the maxillary model of the reference tooth model to the two-dimensional pixel plane using the first target conversion relationship and the shooting internal parameter, and / or rendering the preset tooth model in the mandibular model of the reference tooth model to the two-dimensional pixel plane using the second target conversion relationship and the shooting internal parameter, to obtain a second rendered image; The target facial image is generated according to the adjusted intraoral area and the second rendered image.

7. The method for previewing tooth design effects according to claim 6, wherein: During the rendering process of the second rendered image, the first part of the teeth in the preset tooth model that is not blocked by the original tooth model corresponding to the non-preset tooth model and the second part of the teeth that are blocked by the corresponding original tooth model are determined from the reference tooth model. According to the target conversion relationship corresponding to the preset tooth model, the first area corresponding to the first part of the teeth in the two-dimensional pixel plane is colored, and the pixel values ​​of the second area corresponding to the second part of the teeth in the two-dimensional pixel plane and the area corresponding to the non-preset tooth model in the two-dimensional pixel plane are adjusted to obtain the second rendered image.

8. The method for previewing tooth design effects according to claim 1 or 6, wherein: The method further comprises: Based on the target facial image, generating an updated target facial image using a preset generation model; Alternatively, a video is generated based on multiple updated target face images.

9. A tooth design effect device, characterized in that: The device comprises: an acquisition module, configured to acquire user data and a reference dental model, wherein the user data includes an original dental model, a facial model, an initial facial image, and a facial point cloud corresponding to the initial facial image of the user, and the original dental model and the facial model are aligned with the reference dental model; a determination module, configured to determine a target transformation relationship of the original tooth model relative to the initial facial image based on the alignment processing of the facial model with the facial point cloud and the original tooth model; A rendering module is used to render the preset tooth model in the reference tooth model to the preset tooth area in the initial facial image using the target conversion relationship to obtain a target facial image showing the design effect of the preset tooth model.

10. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the tooth design effect preview method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor in an electronic device, the method for previewing a tooth design effect according to any one of claims 1 to 8 is implemented.

Images