Method and system for acquiring preposed data of three-dimensional digital smile design

Abstract

The invention relates to an artificial intelligence technology, and discloses a method and system for obtaining three-dimensional digital smile design preposed data, and the method comprises the steps: calibrating a transformation matrix of each camera of a system in advance, collecting the face three-dimensional data of a to-be-designed person through a plurality of depth cameras and RGB cameras, and obtaining the face three-dimensional data of the to-be-designed person; performing point cloud conversion on the face three-dimensional data acquired by each camera by using the transformation matrix to obtain a coarse registration point cloud, and performing fine registration on the coarse registration point cloud to obtain a face point cloud; performing grid reconstruction on the face point cloud to obtain a face grid model, and performing tooth region recognition on the face grid model to obtain a tooth marking face model; obtaining an oral cavity scanning grid model of a to-be-designed person, and separating an oral cavity scanning tooth area grid model from the oral cavity scanning grid model; and performing data alignment on the mouth scanning tooth area grid model and the tooth marking face model to obtain three-dimensional digital smile design preposed data. According to the method, the efficiency and the precision of three-dimensional digital smile design preposed data can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a method for obtaining pre-data of three-dimensional digital smile design and a computer readable storage medium. BACKGROUND

[0002] As a core technology in the field of dental aesthetics, smile design aims to restore the harmonious beauty of a patient's smile by fine-tuning the arrangement, color, shape and size of teeth, as well as taking into account the lips and facial contours. Traditionally, this design process mainly relies on two-dimensional images such as photos or X-rays to analyze and plan aesthetic restoration solutions. However, the limitations of two-dimensional images lie in their inability to fully and accurately represent the actual layout and morphological characteristics of teeth in three-dimensional space, which undoubtedly weakens the accuracy of the design and the authenticity of the preview effect.

[0003] In view of this, the digital smile design technology based on three-dimensional data that has emerged in recent years aims to overcome the above limitations by collecting high-precision three-dimensional tooth models and accurately matching them with three-dimensional facial models to achieve a more intuitive and personalized aesthetic design effect. However, in the registration process of three-dimensional tooth models and three-dimensional facial models, doctors often need to manually select multiple corresponding points for accurate alignment, which is not only time-consuming and labor-intensive, but also relatively inefficient, becoming one of the key bottlenecks restricting the widespread application of three-dimensional digital smile design. In summary, the current three-dimensional digital smile design still has deficiencies in achieving efficient and automated matching of facial and tooth models. SUMMARY

[0004] The present application provides a method for obtaining pre-data of three-dimensional digital smile design and a computer readable storage medium, which mainly aims to solve the problems of low matching efficiency and low matching accuracy of traditional three-dimensional digital smile design.

[0005] To achieve the above-mentioned purpose, the present application provides a method for obtaining pre-data of three-dimensional digital smile design, comprising:

[0006] Pre-calibrate the system's camera transformation matrix, use multiple depth cameras and RGB cameras to collect the three-dimensional facial data of the person to be designed, and use the transformation matrix to convert the point cloud of the three-dimensional facial data to obtain coarse registration point cloud, and perform fine registration on the coarse registration point cloud to obtain complete facial point cloud;

[0007] Perform grid reconstruction on the complete facial point cloud to obtain a facial grid model, identify the tooth region of the facial grid model, and obtain a tooth-labeled facial model;

[0008] acquire an oral scanning grid model of a designer, separate an oral tooth area grid model from the oral scanning grid model;

[0009] align the oral tooth area grid model with the tooth marked face model to obtain three-dimensional digital smile design pre-data.

[0010] Optionally, the point cloud conversion of the face three-dimensional data by using the transformation matrix to obtain coarse registration point cloud, comprising:

[0011] using the transformation matrix to perform data transformation on the depth three-dimensional data in the face three-dimensional data to obtain coarse registration data;

[0012] using the following formula to perform data transformation:

[0013] PC ci = RT i ×PC i

[0014] wherein PC ci is the point cloud data of the face collected by the i-th depth camera after position transformation, located in the RGB camera coordinate system, that is, the coarse registration point cloud, RT i is the transformation matrix of the i-th depth camera relative to the RGB camera coordinate system, PC i is the point cloud data of the face collected by the i-th depth camera;

[0015] Optionally, the coarse registration point cloud is finely registered to obtain a complete face point cloud, comprising:

[0016] kdtree is used to find the nearest matching points between the point clouds originally belonging to different depth cameras in the coarse registration point cloud;

[0017] nearest point search is performed on each data in the coarse registration point cloud in the RGB data of the face three-dimensional data to obtain a corresponding point pair;

[0018] a fine matching transformation matrix is calculated by using the corresponding point pair;

[0019] the depth three-dimensional data in the face three-dimensional data is transformed by using the fine matching transformation matrix value, and a complete face point cloud is obtained by merging.

[0020] Optionally, the complete face point cloud is subjected to grid reconstruction to obtain a face grid model, comprising:

[0021] the complete face point cloud is subjected to filtering and smoothing processing to obtain preprocessed face point cloud data;

[0022] mapping the pre-processed face point cloud data into a preset two-dimensional face RGB image template to obtain a point cloud template;

[0023] identifying the pre-processed face point cloud data in the mouth region of the point cloud template as a mouth point cloud to obtain a mouth point cloud;

[0024] performing high-resolution sampling processing on the mouth point cloud in the pre-processed face point cloud data;

[0025] performing low-resolution sampling processing on other part point clouds in the pre-processed face point cloud data;

[0026] performing Poisson surface reconstruction on the pre-processed face point cloud data after sampling processing to obtain a face mesh model.

[0027] Optionally, the mapping the pre-processed face point cloud data into a preset two-dimensional face RGB image template to obtain a point cloud template comprises:

[0028] projecting all point clouds of the pre-processed face point cloud data into the two-dimensional face RGB image template by using the following formula:

[0029] λUV j =KC RGB ×P j

[0030] wherein P j is the jth point in the pre-processed point cloud data, λ is a preset scaling factor, UV j is the two-dimensional coordinate of point P j in the two-dimensional face RGB image template, and KC RGB is an intrinsic matrix of an RGB camera.

[0031] Optionally, the performing Poisson surface reconstruction on the pre-processed face point cloud data after sampling processing to obtain a face mesh model comprises:

[0032] fitting a normal vector corresponding to each point of the pre-processed face point cloud data after sampling processing;

[0033] constructing an octree according to the pre-processed face point cloud data after sampling processing;

[0034] constructing a Poisson equation by using the octree and the normal vector;

[0035] performing equation solving on the Poisson equation to obtain a to-be-processed node value;

[0036] performing isosurface extraction on the to-be-processed node value to generate an initial triangular mesh;

[0037] Smooth the initial triangular mesh to obtain a face mesh model.

[0038] Optionally, the tooth region identification on the face mesh model is performed to obtain a tooth-labeled face model, including:

[0039] The three-dimensional point cloud data corresponding to the tooth region is identified from the face three-dimensional data to obtain tooth point cloud data;

[0040] The mouth point cloud corresponding to the face mesh model is obtained, and a k-dimensional tree is established according to the mouth point cloud;

[0041] The k-dimensional tree is used to find the adjacent points of all face sheet vertices of the mouth point cloud in the face mesh model;

[0042] The mesh face sheet of the adjacent points in the face mesh model is marked as a tooth region to obtain a tooth-labeled face model.

[0043] Optionally, the tooth region grid model is separated from the oral cavity scanning grid model, including:

[0044] The tooth region of the oral cavity scanning grid model is identified;

[0045] The three-dimensional grid corresponding to the tooth region is extracted from the oral cavity scanning grid model;

[0046] The oral cavity scanning tooth region grid model is generated according to the three-dimensional grid.

[0047] Optionally, the data alignment of the oral cavity scanning tooth region grid model and the tooth-labeled face model is performed to obtain three-dimensional digital smile design preposition data, including:

[0048] The key feature points are located from the oral cavity scanning tooth region grid model and the tooth-labeled face model;

[0049] The oral cavity scanning tooth region grid model and the face mesh model are coarsely aligned according to the key feature points to obtain a coarse alignment result;

[0050] The position of the oral cavity scanning tooth region grid model is adjusted according to the coarse alignment result to obtain a precise alignment result, and three-dimensional digital smile design preposition data is obtained.

[0051] In order to solve the above problems, the present application further provides a system for obtaining three-dimensional digital smile design preposition data, including:

[0052] A point cloud conversion module is configured to pre-calibrate a transformation matrix of each camera of a system, collect three-dimensional data of a face of a designer to be designed by using a plurality of depth cameras and an RGB camera, and convert point clouds of the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point clouds, and perform fine registration on the coarse registration point clouds to obtain complete face point clouds.

[0053] A model establishing module is configured to perform mesh reconstruction on the complete face point clouds to obtain a face mesh model, and perform tooth region identification on the face mesh model to obtain a tooth marked face model.

[0054] A model separating module is configured to obtain an oral cavity scanning mesh model of the designer to be designed, and separate an oral tooth region mesh model from the oral cavity scanning mesh model.

[0055] A data alignment module is configured to perform data alignment on the oral tooth region mesh model and the tooth marked face model to obtain three-dimensional digital smile design pre-data.

[0056] The embodiment of the present application ensures the completeness and accuracy of the data by collecting three-dimensional data of a face from multiple angles by using depth cameras and RGB cameras, and improves the data consistency by converting point cloud data of different sources to the same coordinate system by using a transformation matrix. The recognition efficiency and accuracy are improved by automatically identifying teeth and mouth regions. The high-resolution oral cavity scanning mesh model can accurately capture the geometric features of teeth, provide a clear reference for subsequent design, automatically separate the tooth region from the complex oral cavity structure, and generate an accurate tooth region mesh model. The automatic alignment of the tooth model and the face model is realized, the tedious manual operation is avoided, and the acquisition efficiency of smile design pre-data is improved. Therefore, the method and medium for obtaining three-dimensional digital smile design pre-data provided by the present application can solve the problems of low matching efficiency and low matching accuracy of traditional three-dimensional digital smile design. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A flowchart of a method for obtaining three-dimensional digital smile design pre-data provided by an embodiment of the present application is shown.

[0058] Figure 2 A face three-dimensional data acquisition system device provided by an embodiment of the present application is shown.

[0059] Figure 3 A functional module diagram of a device for obtaining three-dimensional digital smile design pre-data provided by an embodiment of the present application is shown.

[0060] Figure 4 A structural diagram of an electronic device for implementing the method for obtaining three-dimensional digital smile design pre-data provided by an embodiment of the present application is shown.

[0061] The objectives, functional characteristics and advantages of the present application will be further explained in conjunction with the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein merely exemplify the application and do not limit the application.

[0063] Embodiments of the present application provide a method for obtaining three-dimensional digital smile design pre-data. The execution subject of the method for obtaining three-dimensional digital smile design pre-data includes but is not limited to at least one of electronic devices such as a server and a terminal which can be configured to execute the method provided by the embodiments of the present application. In other words, the method for obtaining three-dimensional digital smile design pre-data can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0064] Reference Figure 1 As shown in the figure, the flowchart of the method for obtaining three-dimensional digital smile design pre-data provided by an embodiment of the present application. In this embodiment, the method for obtaining three-dimensional digital smile design pre-data includes:

[0065] S1, calibrate the system camera transformation matrix in advance, collect the three-dimensional data of the face of the designer to be designed by using a plurality of depth cameras and RGB cameras, and convert the point cloud of the three-dimensional data of the face by using the transformation matrix to obtain a rough registration point cloud. The rough registration point cloud is precisely registered to obtain a complete face point cloud.

[0066] In the embodiment of the present application, the collection of three-dimensional data of the face of the designer to be designed means that the three-dimensional data of the face of the designer to be designed is collected by a multi-angle face smiling image data collection of the designer to be designed by a face three-dimensional data collection system device, and the three-dimensional data of the face is obtained, including depth three-dimensional data collected by a depth camera and RGB data collected by an RGB camera. Figure 2 As shown in the figure, the face three-dimensional data collection system device is composed of at least three depth cameras and RGB cameras, and four depth cameras are selected in the present application.

[0067] In detail, the depth camera is a camera capable of acquiring distance information of objects in a scene, which can measure the distance from the object to the camera in addition to recording color information, thereby generating a depth map or depth information, the depth map is a two-dimensional array, and each pixel in the array represents the depth value of the corresponding object in the scene; the RGB camera is a 2D, and the main function is to capture and record color information of a scene, which records an image through three color channels of red (R), green (G) and blue (B), and each pixel point has intensity values of the three colors.

[0068] In the embodiment of the application, the point cloud conversion refers to converting the three-dimensional face data captured by each depth camera to the coordinate system of the RGB camera by using a pre-calibrated transformation matrix to obtain a coarse registration point cloud.

[0069] In the embodiment of the application, the three-dimensional face data including depth data and RGB data is collected from multiple angles, so that the integrity and accuracy of the data are ensured; at the same time, the point cloud data from different depth cameras is unified to the RGB camera coordinate system by using the transformation matrix, so that the consistency of the data is ensured, and the registration error is reduced.

[0070] In the embodiment of the application, the point cloud conversion of the three-dimensional face data by using the transformation matrix to obtain the coarse registration point cloud comprises:

[0071] The depth three-dimensional data in the three-dimensional face data is transformed by using the transformation matrix to obtain the coarse registration point cloud.

[0072] In the embodiment of the application, the transformation matrix can be obtained according to the parameters of each acquisition camera of the system, the acquisition camera parameters refer to the camera intrinsic matrix including focal length, principal point position, etc., and the transformation matrix from the depth camera coordinate system to the RGB camera coordinate system is calculated according to the camera parameters.

[0073] In the embodiment of the application, the following formula is used for data transformation:

[0074] PC ci =RT i ×PC i

[0075] Wherein, PC ci is the point cloud data of the face collected by the i-th depth camera after position transformation and located in the RGB camera coordinate system, that is, the coarse registration point cloud, RT i is the transformation matrix of the i-th depth camera relative to the RGB camera coordinate system, and PC i is the point cloud data of the face collected by the i-th depth camera.

[0076] In the embodiment of the present application, the depth three-dimensional data in the face three-dimensional data is subjected to data transformation by using the transformation matrix and the data transformation formula, so that the position of the depth three-dimensional data in the three-dimensional data under the RGB camera coordinate system, i.e., the coarse registration data, is obtained.

[0077] In the embodiment of the present application, the registration data corresponding to all the depth three-dimensional data is put into the same coordinate system to obtain the coarse registration point cloud.

[0078] In the embodiment of the present application, the coarse registration point cloud is subjected to fine registration to obtain the complete face point cloud, which comprises:

[0079] The nearest point search is performed on each data in the coarse registration point cloud in the RGB data in the face three-dimensional data to obtain the corresponding point pair.

[0080] The fine matching transformation matrix is calculated by using the corresponding point pair.

[0081] The depth three-dimensional data in the face three-dimensional data is subjected to data transformation by using the fine matching transformation matrix value, and the complete face point cloud is obtained by merging.

[0082] In the embodiment of the present application, the nearest point search refers to the closest point between the point clouds originally belonging to different depth cameras in the coarse registration point cloud, so that the corresponding point pair is obtained; the Euclidean distance between each point in the coarse registration point cloud originally belonging to different depth cameras is calculated, and the data point pair with the minimum Euclidean distance is selected as the corresponding point pair. The kdtree can be used to find the closest point.

[0083] In the embodiment of the present application, the fine matching transformation matrix is composed of a rotation matrix and a translation vector, and the depth three-dimensional data can be accurately aligned by rigid transformation, and can be accurately mapped into the rgb image; the covariance matrix is constructed by using the corresponding point pair, the singular value decomposition is performed on the covariance matrix, the rotation matrix is extracted from the decomposition result, and the translation vector is obtained by calculating the difference between the centers of the corresponding point pairs.

[0084] In the embodiment of the present application, the depth three-dimensional data is subjected to data transformation by using the calculated rotation and translation parameters, so that the complete face point cloud is obtained.

[0085] In the embodiment of the present application, the camera calibration and the transformation matrix calculation ensure that the point cloud data from different sources can be fused in a unified coordinate system, improve the consistency and reliability of the data, and the coarse registration and fine registration of the point cloud data can reduce the error during data fusion and improve the accuracy of the data.

[0086] S2, grid reconstruction is performed on the complete human face point cloud to obtain a human face grid model, and tooth region identification is performed on the human face grid model to obtain a tooth marked human face model.

[0087] In the embodiment of the present application, by performing grid reconstruction on the complete human face point cloud, different grid distributions can be realized in different regions, the computing resources are optimized, and the details of the key regions are maintained, thereby realizing the rapid generation of the human face grid model.

[0088] The grid reconstruction on the complete human face point cloud to obtain the human face grid model comprises:

[0089] Filtering and smoothing processing is performed on the complete human face point cloud to obtain preprocessed human face point cloud data;

[0090] The preprocessed human face point cloud data is mapped into a preset two-dimensional human face RGB image template to obtain a point cloud template;

[0091] The preprocessed human face point cloud data in the mouth region in the point cloud template is marked as mouth point cloud to obtain the mouth point cloud;

[0092] High-resolution sampling processing is performed on the mouth point cloud in the preprocessed human face point cloud data;

[0093] Low-resolution sampling processing is performed on other part point cloud in the preprocessed human face point cloud data;

[0094] Poisson surface reconstruction is performed on the preprocessed human face point cloud data after sampling processing to obtain the human face grid model.

[0095] In the embodiment of the present application, a convolutional neural network model for identifying a mouth region of interest of a two-dimensional human face image is pre-trained, the mouth region of the collected human face three-dimensional data is identified by the convolutional neural network to obtain a mouth region of interest, the human face point cloud is mapped into a human face RGB image coordinate system by using a transformation matrix to obtain a mouth coordinate of each point in the mouth region, and the preprocessed data is divided into point clouds according to the mouth coordinate and the mouth region of interest to obtain a mouth point cloud.

[0096] All point clouds of the preprocessed human face point cloud data are projected into the two-dimensional human face RGB image template by using the following formula:

[0097] λUV j =KC RGB ×P j

[0098] wherein, P j is the jth point in the preprocessed point cloud data, λ is a preset scaling factor, UV j is the point Pj A two-dimensional coordinate in a two-dimensional human face RGB image template, KC RGB Is the intrinsic matrix of the RGB camera.

[0099] In the embodiment of the application, the pre-processed face point cloud data after sampling processing is subjected to Poisson surface reconstruction to obtain a face mesh model, comprising:

[0100] The normal vector corresponding to each point of the pre-processed face point cloud data after sampling processing is fitted;

[0101] An octree is constructed according to the pre-processed face point cloud data after sampling processing;

[0102] The octree and the normal vector are used to construct a Poisson equation;

[0103] The Poisson equation is solved to obtain a to-be-processed node value;

[0104] The to-be-processed node value is subjected to isosurface extraction to generate an initial triangular mesh;

[0105] The initial triangular mesh is subjected to mesh smoothing to obtain a face mesh model.

[0106] In the embodiment of the application, at the junction of two meshes, it is ensured that the vertex is shared, if the topological structure of the two meshes in the splicing area is different, a new vertex is inserted to match the structure of the other party, the patch is adjusted, the triangular patch of the splicing area is smoothly transitioned with the adjacent area, and finally a face mesh is obtained.

[0107] In the embodiment of the application, the face mesh model is subjected to tooth region identification to obtain a tooth-labeled face model, comprising:

[0108] Three-dimensional point cloud data corresponding to the tooth region is identified from the face three-dimensional data to obtain tooth point cloud;

[0109] The mouth point cloud corresponding to the face mesh model is obtained, and a k-dimensional tree is established according to the mouth point cloud;

[0110] The k-dimensional tree is used to find the adjacent points of all patch vertices of the mouth point cloud in the face mesh model;

[0111] The adjacent points in the mesh patch of the face mesh model are marked as tooth regions to obtain a tooth-labeled face model.

[0112] In the embodiment of the present application, the mouth point cloud is obtained in the step of performing grid reconstruction on the complete face point cloud to obtain the face grid model, and a k-d tree is constructed from the mouth point cloud, wherein the k-d tree is a tree-shaped data structure for multi-dimensional space data, can be used for fast spatial query, such as nearest neighbor search, range search, etc., and the k-d tree organizes data points efficiently by recursively partitioning space into hyperplanes, so that the target area can be quickly located when searching.

[0113] Further, the face patch vertex is three corner points constituting each face patch, and the vertices have specific coordinate positions in three-dimensional space, which define the geometric shape of the face model, for each face patch vertex in the face grid model, whether there is a neighboring point in the k-d tree of the mouth point cloud is searched with the vertex as a query point, if there is a neighboring point, the grid patch of the vertex in the face grid model is marked as a tooth area, and a tooth-marked face model is obtained.

[0114] In the embodiment of the present application, the k-d tree can quickly query the grid patch of the mouth point cloud in the face grid model, which can save the time and manpower required for marking the mouth area, make the whole three-dimensional digital smile design process more efficient, improve the accuracy of the mouth area recognition, and reduce the deviation caused by human factors.

[0115] S3, obtaining an oral cavity scanning grid model of a designer to be designed, and separating an oral cavity scanning tooth area grid model from the oral cavity scanning grid model.

[0116] In the embodiment of the present application, the oral cavity scanning grid model refers to capturing high-resolution images inside the oral cavity by laser or optical means using a high-precision oral cavity scanner. The oral cavity scanner emits a light beam and receives a reflected signal to measure the shape and color information inside the oral cavity.

[0117] In the embodiment of the present application, the high-precision oral cavity scanning grid model can accurately capture the details of the teeth, including position, shape and size; automatically identifying and segmenting the tooth area from the oral cavity scanning grid model also improves the efficiency and accuracy of tooth segmentation.

[0118] The separation of the oral cavity scanning tooth area grid model from the oral cavity scanning grid model comprises:

[0119] Identifying the tooth area of the oral cavity scanning grid model;

[0120] Extracting a three-dimensional grid corresponding to the tooth area from the oral cavity scanning grid model;

[0121] Generating an oral cavity scanning tooth area grid model according to the three-dimensional grid.

[0122] In the embodiment of the present application, a pre-trained convolutional neural network model is used, which is trained on a large number of labeled oral scan data, to identify and segment the tooth region. The trained model is applied to new oral scan data to predict and label the tooth region.

[0123] In the embodiment of the present application, the convolutional neural network model is used to predict a grid structure containing only the tooth region from the original oral scan grid model.

[0124] In the embodiment of the present application, the tooth grid structure is exported or created as a new grid model, namely an oral scan tooth region network model. This model will only contain tooth-related vertices and patches.

[0125] In the embodiment of the present application, through the high-resolution oral scan grid model, the position, shape, size and direction of the teeth can be accurately captured, and then the tooth region can be accurately separated from the complex oral structure to obtain the tooth region grid model, thereby improving the accuracy of data alignment between the oral scan tooth region network model and the tooth-labeled face model.

[0126] S4, aligning the oral scan tooth region grid model with the tooth-labeled face model to obtain three-dimensional digital smile design pre-data.

[0127] In the embodiment of the present application, by aligning the oral scan tooth region grid model with the face grid model, the accuracy of tooth position during smile design is ensured, and the natural coordination of facial features and teeth is realized.

[0128] In the embodiment of the present application, the alignment of the oral scan tooth region grid model with the tooth-labeled face model to obtain three-dimensional digital smile design pre-data includes:

[0129] Positioning key feature points from the oral scan tooth region grid model and the tooth-labeled face model;

[0130] Coarsely aligning the oral scan tooth region grid model and the face grid model according to the key feature points to obtain a coarse alignment result;

[0131] Adjusting the position of the oral scan tooth region grid model according to the coarse alignment result to obtain a precise alignment result and obtain three-dimensional digital smile design pre-data.

[0132] In the embodiment of the present application, according to the key feature points, a preliminary alignment operation is performed. This usually involves calculating the similarity transformation (such as rotation and translation) between the two models, so that the key points coincide as much as possible. Iterative Closest Point algorithm or other point cloud registration methods can be used to optimize the alignment process to obtain a preliminary coarse alignment result.

[0133] In the embodiment of the present application, the tooth scanning area network model is aligned with the tooth marked face model, the spatial relationship between the facial features and the teeth is correctly matched, the influence of soft tissue and other non-rigid data on the alignment of the face data and the oral scanning data is reduced, the automatic alignment of the low-precision face three-dimensional grid data and the high-precision oral scanning data is realized, the manual alignment operation steps are simplified, and the alignment efficiency is improved.

[0134] As shown in Figure 3 is a functional module diagram of the device for obtaining three-dimensional digital smile design pre-data provided by an embodiment of the present application.

[0135] The device 100 for obtaining three-dimensional digital smile design pre-data can be installed in an electronic device. According to the functions to be implemented, the device 100 for obtaining three-dimensional digital smile design pre-data can include a point cloud conversion module 101, a model establishment module 102, a model separation module 103, and a data alignment module 104. The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete a fixed function, and are stored in the memory of the electronic device.

[0136] In the embodiment, the functions of each module / unit are as follows:

[0137] The point cloud conversion module 101 is configured to calibrate the transformation matrix of each camera of the system in advance, collect the three-dimensional face data of the person to be designed by using a plurality of depth cameras and RGB cameras, perform point cloud conversion on the three-dimensional face data by using the transformation matrix, obtain coarse registration point cloud, and perform fine registration on the coarse registration point cloud to obtain complete face point cloud.

[0138] The model establishment module 102 is configured to perform grid reconstruction on the complete face point cloud to obtain a face grid model, and perform tooth area identification on the face grid model to obtain a tooth marked face model.

[0139] The model separation module 103 is configured to obtain an oral scanning grid model of the person to be designed, and separate an oral tooth scanning area grid model from the oral scanning grid model.

[0140] The data alignment module 104 is configured to perform data alignment on the oral tooth scanning area grid model and the tooth marked face model to obtain three-dimensional digital smile design pre-data.

[0141] In detail, each module in the device 100 for obtaining three-dimensional digital smile design pre-data in the embodiment of the present application adopts the above Figures 1 to 2The method for obtaining the three-dimensional digital smile design pre-data uses the same technical means as the method described in the background and can achieve the same technical effects, which will not be described here.

[0142] As shown in Figure 4 is a structural schematic diagram of an electronic device for implementing the method for obtaining the three-dimensional digital smile design pre-data according to an embodiment of the present application.

[0143] The electronic device 1 can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can further include a computer program stored in the memory 11 and executable on the processor 10, such as a program for obtaining three-dimensional digital smile design pre-data.

[0144] In some embodiments, the processor 10 can be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, which connects all components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (such as executing the program for obtaining three-dimensional digital smile design pre-data), and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0145] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both the internal storage unit and the external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, such as the code of the program for obtaining three-dimensional digital smile design pre-data, but also to temporarily store data that has been output or will be output.

[0146] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11 and the at least one processor 10, etc.

[0147] The communication interface 13 is configured to realize the communication between the electronic device and other devices, and includes a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually configured to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display can also be appropriately referred to as a display screen or a display unit, and is configured to display information processed in the electronic device and to display a visualized user interface.

[0148] Only the electronic device with components is shown in the figure, and those skilled in the art can understand that the structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include fewer or more components than those shown in the figure, or some components can be combined, or different components can be arranged.

[0149] For example, although not shown, the electronic device can further include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so as to realize functions such as charge management, discharge management and power consumption management through the power management device. The power supply can also include one or more direct current or alternating current power supplies, a recharging device, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, etc. The electronic device can also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be described here.

[0150] It should be understood that the embodiments are only for illustration, and the scope of the patent application is not limited by the structure.

[0151] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0152] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0153] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0154] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0155] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0156] Specifically, the specific implementation method of the above-mentioned instructions can refer to the description of the related steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0157] Further, the modules / units integrated in the electronic device 1 are realized in the form of software function units and sold or used as independent products, which can be stored in a computer readable storage medium. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

[0158] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following steps when executed by the processor of the electronic device:

[0159] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0160] The pre-calibration system is used to calibrate the transformation matrix of each camera, and the three-dimensional data of the face of the designer to be designed is collected by using multiple depth cameras and RGB cameras, and the point cloud conversion is performed on the three-dimensional data of the face by using the transformation matrix to obtain coarse registration point cloud.

[0161] obtaining an oral cavity scanning mesh model of a designer to be designed, separating an oral cavity scanning mesh model of a tooth area from the oral cavity scanning mesh model;

[0162] aligning the oral cavity scanning mesh model of the tooth area with the tooth marked face model to obtain three-dimensional digital smile design pre-data.

[0163] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of the modules can be different from the above. For example, the division of the modules can be performed in another manner, and the actual implementation can be different from the above.

[0164] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0165] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0166] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0167] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is not limited only by the above description, and therefore all changes within the meaning and scope of equivalent elements falling within the scope of protection are intended to be included in the present application.

[0168] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0169] Furthermore, the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural and vice-versa, unless the context clearly requires these exclusions. The conjunction "or" is used to link items in a list or a set of alternatives, and is not disjunctive, unless the context clearly requires it to be disjunctive. The conjunction "and" is used to link items in a list or a set of alternatives, and is not conjunctive, unless the context clearly requires it to be conjunctive. The prefix "first", "second", "third", etc. is used to identify similar entities of a different class or a different category, and is not used to designate a particular order of precedence, unless the context clearly requires it to be so.

[0170] Finally, it should be noted that the above-mentioned embodiments illustrate rather than limit the application, since the scope of the application is determined by the appended claims.

Claims

1. A method for acquiring preliminary data for three-dimensional digital smile design, characterized in that, The method includes: The transformation matrices of each camera in the system are pre-calibrated. The three-dimensional facial data of the subject is collected using multiple depth cameras and RGB cameras. The three-dimensional facial data is then converted into a point cloud using the transformation matrices to obtain a coarse registration point cloud. The coarse registration point cloud is then finely registered to obtain a complete facial point cloud. The complete face point cloud is reconstructed into a mesh to obtain a face mesh model. The face mesh model is then used to identify the tooth region to obtain a face model with tooth markings. Obtain the oral cavity scanning mesh model of the subject to be designed, and separate the exit scanning tooth area mesh model from the oral cavity scanning mesh model; The oral scan tooth region mesh model is aligned with the tooth-marked face model to obtain the preliminary data for three-dimensional digital smile design.

2. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 1, characterized in that, The step of performing point cloud transformation on the 3D facial data using the transformation matrix to obtain a coarsely registered point cloud includes: The transformation matrix is ​​used to transform the depth 3D data in the face 3D data to obtain coarse registration data. Use the following formula to perform data transformation: PC ci =RT i ×PC i Among them, PC ci The point cloud data of the face acquired by the i-th depth camera, after position transformation, is the point cloud data located in the RGB camera coordinate system, i.e., the coarse registration point cloud. RT i PC is the transformation matrix of the i-th depth camera relative to the RGB camera coordinate system. i The face point cloud data collected by the i-th depth camera.

3. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 1, characterized in that, The process of performing fine registration on the coarsely registered point cloud to obtain a complete face point cloud includes: For each data point in the coarse registration point cloud, a nearest point search is performed on the RGB data of the face 3D data to obtain the corresponding point pair; Calculate the fine-matching transformation matrix using the corresponding point pairs; The depth 3D data in the face 3D data is transformed using the fine matching transformation matrix value and merged to obtain a complete face point cloud.

4. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 1, characterized in that, The step of reconstructing the complete face point cloud into a face mesh model includes: The complete face point cloud is filtered and smoothed to obtain preprocessed face point cloud data; The preprocessed face point cloud data is mapped onto a preset two-dimensional face RGB image template to obtain a point cloud template; The preprocessed face point cloud data in the mouth area of ​​the point cloud template is identified as the mouth point cloud to obtain the mouth point cloud. The mouth point cloud in the preprocessed face point cloud data is subjected to high-resolution sampling processing; Low-resolution sampling processing is performed on the point clouds of other parts in the preprocessed face point cloud data; Poisson surface reconstruction is performed on the preprocessed face point cloud data after sampling to obtain a face mesh model.

5. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 4, characterized in that, The step of mapping the preprocessed face point cloud data onto a preset two-dimensional face RGB image template to obtain the point cloud template includes: The following formula is used to project all point clouds of the preprocessed face point cloud data onto the two-dimensional face RGB image template: λUV j =KC RGB ×P j Among them, P j For the j-th point in the preprocessed point cloud data, λ is a preset scaling factor, and UV... j Let P be the point j In the two-dimensional coordinates of the RGB image template of a two-dimensional face, KC RGB This is the intrinsic parameter matrix of the RGB camera.

6. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 4, characterized in that, The step of reconstructing the face mesh model from the preprocessed face point cloud data after sampling and processing using a Poisson surface includes: Fit the normal vector corresponding to each point of the preprocessed face point cloud data after sampling; An octree is constructed based on the preprocessed face point cloud data after sampling. The Poisson equation is constructed using the octree and the normal vector. Solve the Poisson equation to obtain the values ​​of the nodes to be processed; Isosurfaces are extracted from the node values ​​to be processed to generate an initial triangular mesh; The initial triangular mesh is smoothed to obtain a face mesh model.

7. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 1, characterized in that, The step of performing tooth region recognition on the face mesh model to obtain a tooth-marked face model includes: Obtain the mouth point cloud corresponding to the face mesh model, and build a k-dimensional tree based on the mouth point cloud; The k-dimensional tree is used to find the nearest points of all facet vertices in the face mesh model of the mouth point cloud; The neighboring points are marked as tooth regions in the mesh area of ​​the face mesh model to obtain a tooth-marked face model.

8. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 1, characterized in that, The step of separating the exit scan dental region mesh model from the oral cavity scan mesh model includes: Identify the tooth regions in the oral cavity scanning mesh model; Extract the three-dimensional mesh corresponding to the tooth region from the oral cavity scanning mesh model; A mesh model of the dental region is generated based on the three-dimensional mesh.

9. The method for obtaining preliminary data for three-dimensional digital smile design as described in claim 1, characterized in that, The step of aligning the oral scan tooth region mesh model with the tooth-marked face model to obtain 3D digital smile design pre-data includes: Key feature points are located from the oral scan tooth region mesh model and the tooth-marked face model; Based on the key feature points, the oral scan tooth region mesh model and the face mesh model are coarsely aligned to obtain the coarse alignment result. Based on the coarse alignment result, the position of the oral scan tooth region mesh model is adjusted to obtain the precise alignment result, thus obtaining the preliminary data for three-dimensional digital smile design.

10. A system for acquiring preliminary data for three-dimensional digital smile design, comprising: The point cloud conversion module is used to pre-calibrate the transformation matrices of each camera in the system, collect the 3D facial data of the subject using multiple depth cameras and RGB cameras, and perform point cloud conversion on the 3D facial data using the transformation matrices to obtain a coarse registration point cloud. The coarse registration point cloud is then finely registered to obtain a complete facial point cloud. The model building module is used to perform mesh reconstruction on the complete face point cloud to obtain a face mesh model, and to perform tooth region recognition on the face mesh model to obtain a tooth-marked face model. The model separation module is used to obtain the oral cavity scanning mesh model of the subject and separate the exit scanning tooth area mesh model from the oral cavity scanning mesh model; The data alignment module is used to align the oral scan tooth region mesh model with the tooth-marked face model to obtain the three-dimensional digital smile design pre-data.

Images