A method and device for simulating and visualizing mandibular movements through motion capture.

Abstract

This application proposes a method and apparatus for mandibular motion simulation visualization through motion capture, comprising the following steps: generating a three-dimensional tooth model using patient oral scan data, fixing markers on the patient's teeth, calibrating an RGBD camera to acquire mandibular motion video; constructing a three-dimensional coordinate system using the RGBD camera, calculating the center point coordinates of the markers in each frame image, and obtaining the rotation matrix and translation vector of adjacent frame images; selecting tooth feature points in the first frame image and registering the three-dimensional tooth model with them; and driving the tooth model to move using the translation vector and rotation matrix of adjacent frame images. This scheme uses singular value decomposition to obtain the rotation matrix and translation vector between two adjacent frame images, which reduces the computational load while improving the accuracy of the transformation, and calculates the distance between point pairs using an error function for registration to obtain better visualization results.

Description

Technical Field

[0001] This application relates to the field of dental medicine, and in particular to a method and apparatus for simulating and visualizing mandibular movements through motion capture. Background Technology

[0002] There are two main technical means for three-dimensional digital reconstruction of patients' teeth: oral scanning and CBCT imaging (Cone beam CT). Oral scanning technology can quickly and non-invasively obtain a three-dimensional digital surface model of the internal structure of the oral cavity, including high-precision crowns of all teeth and a part of the gingival tissue. CBCT scans the object with a rotating X-ray beam and obtains three-dimensional structural information through three-dimensional reconstruction algorithms. However, CBCT images generally have cases where the maxillary teeth are obscured or connected to part of the mandibular teeth, and the complete surface of the mandibular teeth cannot be obtained. This makes it difficult to manually select the feature points of the mandibular teeth. In addition, the accuracy of the tooth surface reconstructed by CBCT images is far inferior to that of the oral scanning model.

[0003] In dental diagnosis, not only is a three-dimensional digitally reconstructed tooth model required, but motion capture is also usually needed to obtain the patient's mandibular movement trajectory in order to determine the patient's occlusion and obtain a more accurate diagnosis.

[0004] Motion capture technology refers to the technique of recording and processing the movements of people or other objects. It is widely used in many fields such as film, medicine, and games, and is an important research method in computer vision, ergonomics, and biomechanics. It typically uses 3D motion capture equipment to comprehensively record the movement of objects in three-dimensional space. In the dental field, mechanical, optical, and inertial motion capture equipment are mainly used to capture the patient's mandibular movements. Common optical motion capture equipment consists of monocular or binocular cameras. Binocular cameras have two cameras placed on either side of the object being photographed, calculating the object's position in a three-dimensional coordinate system. However, camera placement is cumbersome, and object coordinate calculations are extensive. Correspondingly, current techniques commonly used in dental diagnosis typically combine monocular or binocular cameras with CBCT data to obtain the mandibular movement trajectory. However, because the accuracy of CBCT data is far inferior to that of oral scan models, and oral scan models cannot effectively capture motion, it is currently impossible to accurately establish a high-fidelity mandibular movement trajectory through motion capture, thus affecting the accuracy of dental diagnosis. Summary of the Invention

[0005] This invention provides a method and apparatus for simulating and visualizing mandibular movements through motion capture. It uses singular value decomposition to obtain the rotation matrix and translation vector between two adjacent frames of images during the mandibular movement of the test subject. This reduces the amount of computation and improves the accuracy of mandibular movement visualization. Furthermore, it uses an error function to calculate the distance between point pairs for registration, thereby obtaining a better mandibular movement visualization effect.

[0006] In a first aspect, embodiments of this application provide a method for simulating and visualizing mandibular movements through motion capture, the method comprising:

[0007] Acquire the patient's oral cavity scan data and obtain a three-dimensional tooth surface model based on the patient's oral cavity scan data;

[0008] Set at least two markers with different patterns for each marker, fix each marker to the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement.

[0009] Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the center point of the same marker in adjacent frame images and the rotation matrix.

[0010] At least two tooth feature points are obtained in the first frame of the mandibular motion video. The model corresponding point of each tooth feature point is obtained in the three-dimensional tooth surface model. The tooth feature point and its model corresponding point form a corresponding point pair. The registration rotation matrix and registration translation vector of the three-dimensional tooth surface model are calculated according to the corresponding point pair, and the three-dimensional tooth surface model is registered to the first frame image.

[0011] The three-dimensional tooth surface model is driven to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

[0012] Secondly, embodiments of this application provide a device for simulating and visualizing mandibular movements through motion capture, comprising:

[0013] Acquisition module: Acquires the patient's oral cavity scan data and generates a three-dimensional tooth surface model based on the patient's oral cavity scan data;

[0014] Calibration module: Set at least two markers with different patterns for each marker, fix each marker on the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement;

[0015] Construction module: Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the same marker in adjacent frame images and the rotation matrix.

[0016] Registration module: Obtain at least two tooth feature points in the first frame of the mandibular motion video, obtain the model corresponding point of each tooth feature point in the three-dimensional tooth surface model, wherein the tooth feature point and its model corresponding point form a corresponding point pair, calculate the registration rotation matrix and registration translation vector for registering the three-dimensional tooth surface model to the first frame image based on the corresponding point pair, and register the three-dimensional tooth surface model to the first frame image.

[0017] Motion module: Drives the three-dimensional tooth surface model to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

[0018] Thirdly, embodiments of this application provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform a method for simulating and visualizing mandibular movements through motion capture.

[0019] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including a method for simulating and visualizing mandibular movements through motion capture.

[0020] The main contributions and innovations of this invention are as follows:

[0021] This application embodiment uses markers fixed to the patient's teeth and an RGBD camera to acquire mandibular motion videos. Acquiring mandibular motion videos using an RGBD camera offers higher scene accuracy and lower computational cost, achieving highly accurate reconstruction of real mandibular motion with only one RGBD camera, three fixed markers, and oral scan data. Furthermore, this solution uses singular value decomposition to calculate the rotation matrix and translation vector between adjacent RGBD image frames, which not only reduces computational cost but also better simulates the mandibular transformation between these two frames through rigid body transformation. This solution constructs an error function and uses a nonlinear optimization algorithm to iteratively optimize the initial rotation matrix and initial translation vector to obtain better registration results, resulting in improved mandibular motion visualization through rendering.

[0022] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a flowchart of a method for simulating and visualizing mandibular movements through motion capture, according to an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the markers used according to the embodiments of this application;

[0026] Figure 3 This is a schematic diagram illustrating the fixing of the marker to the patient's teeth in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram showing the selection of corresponding points according to the embodiments of this application;

[0028] Figure 5 This is a structural block diagram of a mandibular motion simulation visualization device based on an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0031] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0032] Example 1

[0033] Before introducing this plan, let's first explain the terms used in it:

[0034] RGBD camera: A camera that combines an RGB camera and a depth sensor. It can simultaneously capture color RGB images and depth images of a scene, obtain distance information for each pixel in the scene, and store it in the depth image. Compared with traditional stereo cameras, RGBD cameras have the advantages of relatively simple camera position configuration, less computation, and more accurate scene depth information.

[0035] This application provides a method for simulating and visualizing mandibular movements through motion capture. Specifically, refer to... Figure 1 The method includes:

[0036] Acquire the patient's oral cavity scan data and obtain a three-dimensional tooth surface model based on the patient's oral cavity scan data;

[0037] Set at least two markers with different patterns for each marker, fix each marker to the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement.

[0038] Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the center point of the same marker in adjacent frame images and the rotation matrix.

[0039] At least two tooth feature points are obtained in the first frame of the mandibular motion video. The model corresponding point of each tooth feature point is obtained in the three-dimensional tooth surface model. The tooth feature point and its model corresponding point form a corresponding point pair. The registration rotation matrix and registration translation vector of the three-dimensional tooth surface model are calculated according to the corresponding point pair, and the three-dimensional tooth surface model is registered to the first frame image.

[0040] The three-dimensional tooth surface model is driven to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

[0041] In some specific embodiments, an oral scanner is used to collect oral scanning data of the patient's upper and lower teeth, and a three-dimensional tooth surface model is obtained based on the oral scanning data. The oral scanning data is stored in STL format.

[0042] In some embodiments, in the step of “setting at least two markers with different patterns for each marker and fixing each marker to the patient’s teeth”, the markers are black and white marking blocks that are fitted onto the corresponding teeth of the patient using a transparent material.

[0043] Specifically, the standard markers are used for camera calibration.

[0044] Furthermore, the standard markers can provide known geometric features and dimensions in camera calibration to calculate various camera parameters and correct image distortion, ensuring that image measurements and calculations are geometrically accurate and reliable.

[0045] For example, the markers used in this solution are as follows: Figure 2 As shown, each marker is a combination of 5x5 black and white planar blocks, and the standard marker in this scheme is a black and white checkerboard.

[0046] An exemplary diagram illustrating how this method fixes the marker to the patient's teeth is shown below. Figure 3 As shown, this scheme uses three markers to calibrate the RGBD camera.

[0047] In some embodiments, in the step of "fixing each marker to the patient's teeth and calibrating the parameters of the RGBD camera using the markers", the markers include at least one standard marker. The RGBD camera is used to capture an RGB image and a corresponding depth image of the patient. Corner points of the standard markers are extracted from the RGB image. Intrinsic parameters of the RGBD camera are obtained based on the corner points. Extrinsic parameters of the RGBD camera are obtained based on the intrinsic parameters. Camera distortion parameters of the RGBD camera are obtained based on the intrinsic and extrinsic parameters. Depth distortion parameters of the depth image are calculated. The RGBD camera is calibrated and corrected using the camera distortion parameters and the depth distortion parameters.

[0048] In some specific embodiments, this solution uses the Zhang Zhengyou calibration method to calibrate the RGBD camera, and then completes the parameter correction of the entire RGBD camera by performing distortion correction on the depth image.

[0049] In some specific embodiments, in the step of "using the corrected RGBD camera to face the patient's face to obtain the mandibular movement video corresponding to the patient's mandibular movement", the test subject is guided to perform the opening and closing of the upper and lower jaws and the forward and backward and left and right movements while exposing as many teeth as possible to obtain the mandibular movement video.

[0050] Specifically, in order to obtain better video of jaw movement, the RGBD camera is used to capture images of the patient's face.

[0051] In some specific embodiments, the three-dimensional coordinates of the center point of each marker in the three-dimensional coordinate system are obtained based on the intrinsic and extrinsic parameters of the RGBD camera and the depth distortion parameters.

[0052] In some embodiments, in the step of "calculating the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image", the coordinates of each center point in the current frame image are first normalized, and the average value of all center point coordinates is calculated to obtain the centroid coordinates. The difference between each center point coordinate and the centroid coordinates is obtained to obtain the difference coordinates. The difference coordinates are saved in the form of a matrix to obtain the coordinate matrix of the current frame image.

[0053] Specifically, the coordinates of each center point in the current frame image are normalized by dividing by the modulus, where the modulus is the length vector from the origin of the three-dimensional coordinate system to the coordinates of the center point. Normalizing the coordinates of each center point ensures that the coordinate values ​​have a consistent scale, which facilitates subsequent calculations.

[0054] In this scheme, the difference coordinates are stored as a 3*3 coordinate matrix. For example, if there are n frames of images, n matrices A can be obtained. i , i∈[1,n].

[0055] Specifically, the coordinate matrix is ​​stored column-wise.

[0056] In some embodiments, in the step of "calculating the rotation matrix after performing singular value decomposition on the coordinate matrix of adjacent frame images, and calculating the translation vector by taking the center point coordinates of the center point of the same landmark in adjacent frame images and the rotation matrix", the singular matrix is ​​obtained by multiplying the transpose of the coordinate matrix of the next frame image in adjacent frame images by the coordinate matrix of the previous frame image. The singular matrix is ​​decomposed to obtain a left singular matrix and a right singular matrix. The rotation matrix is ​​obtained by multiplying the left singular matrix by the transpose of the right singular matrix. The translation result is obtained by multiplying the center point coordinates of any landmark in the previous frame image in two adjacent frames by the rotation matrix. The translation vector is obtained by subtracting the translation result from the center point coordinates of the landmark in the next frame image that is the same as the previous frame image.

[0057] For example, for the coordinate matrix A of two adjacent frame images k and A k+1 Calculate S k =A k A k+1 T The singular value decomposition of S k S is obtained by multiplying three matrices. k =U k ∑ k V k T Where Uk is a 3×3 square matrix, and the vectors in Uk are left singular vectors, all of which are orthogonal, ∑ k It is a 3×3 square matrix. The elements on the diagonal are singular values, and all other elements are 0. V k T It is a 3×3 square matrix, V k T The vectors in the matrix are right singular vectors, all orthogonal, and T is the transpose matrix. Therefore, the rotation matrix is ​​R. k =U k V k T .

[0058] For example, if p k Let p be the center coordinates of any landmark in the k-th frame image. k+1 If the center coordinates of the same landmark are in the (k+1)th frame of the image, then the translation vector t k =P k+1 -R k p k .

[0059] Specifically, if there are a total of n images, then n-1 rotation matrices and translation vectors can be obtained.

[0060] Specifically, this scheme uses singular value decomposition to obtain the rotation matrix and translation vector between every two frames, which can greatly reduce the amount of computation. By performing a rigid body transformation on the coordinates of the three marker center points of the previous frame, the coordinates of the three marker center points of the next adjacent image can be obtained.

[0061] In some embodiments, in the step of “obtaining at least two tooth feature points in the first frame of the mandibular motion video”, the tooth feature points are evenly distributed across the patient’s maxillary and mandibular teeth.

[0062] In this scheme, the number of tooth feature points is 6. Three maxillary tooth feature points and three mandibular tooth feature points are manually selected in the first frame image. Then, corresponding points are selected in the 3D tooth surface model as shown below. Figure 4 As shown.

[0063] Specifically, distributing the tooth feature points evenly across the patient's maxillary and mandibular teeth allows for better subsequent registration and other operations.

[0064] In some embodiments, in the step of "calculating the registration rotation matrix and registration translation vector for registering the three-dimensional tooth surface model to the first frame image based on the corresponding point pairs", the coordinates of the tooth feature points and the coordinates of the corresponding points of the model are obtained in each corresponding point pair. A tooth feature point matrix is ​​formed using all the tooth feature point coordinates, and a model corresponding point matrix is ​​formed using all the model corresponding point coordinates. The sum of distances between all corresponding point pairs is calculated based on the tooth feature point matrix and the model corresponding point matrix. An error function is constructed based on the sum of distances between all corresponding point pairs. An initial rotation matrix and an initial translation vector are obtained. The tooth feature point matrix is ​​iteratively optimized using the initial rotation matrix and the initial translation vector until the error function is less than a first set threshold to obtain the registration rotation matrix and the registration translation vector.

[0065] Specifically, the coordinates of each tooth feature point are obtained based on the camera's intrinsic and extrinsic parameters and depth distortion parameters. Since the three-dimensional tooth surface model is directly placed into the three-dimensional coordinate system, the coordinates of the corresponding points on the model can be obtained directly.

[0066] Specifically, the points in each corresponding point pair are normalized by dividing the coordinates by the modulus.

[0067] For example, both the tooth feature point matrix and the model corresponding point matrix are 3×3 matrices and are stored in columns.

[0068] Specifically, in order to improve the accuracy of registration, the maxillary teeth are registered first, and then the mandibular teeth are registered.

[0069] For example, the initial translation vector and the initial rotation matrix can be updated based on the error function and the Hessian matrix.

[0070] Specifically, the initial rotation matrix and the initial translation vector are obtained by initializing the parameters in the rotation matrix and the translation matrix.

[0071] In some embodiments, in the step of "driving the three-dimensional tooth surface model to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames", the three-dimensional tooth surface model is driven to undergo rigid body transformation according to the rotation matrix and the translation vector, and the transformation result is visualized and rendered.

[0072] Specifically, the mandibular part of the oral cavity scanning model is driven to undergo n-1 rigid body transformations based on n-1 rotation matrices and translation vectors. The results of each transformation are then visualized and rendered to achieve motion simulation visualization.

[0073] Example 2

[0074] Based on the same concept, referencing Figure 5 This application also proposes a device for simulating and visualizing mandibular movements through motion capture, comprising:

[0075] Acquisition module: Acquires the patient's oral cavity scan data and generates a three-dimensional tooth surface model based on the patient's oral cavity scan data;

[0076] Calibration module: Set at least two markers with different patterns for each marker, fix each marker on the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement;

[0077] Construction module: Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the same marker in adjacent frame images and the rotation matrix.

[0078] Registration module: Obtain at least two tooth feature points in the first frame of the mandibular motion video, obtain the model corresponding point of each tooth feature point in the three-dimensional tooth surface model, wherein the tooth feature point and its model corresponding point form a corresponding point pair, calculate the registration rotation matrix and registration translation vector for registering the three-dimensional tooth surface model to the first frame image based on the corresponding point pair, and register the three-dimensional tooth surface model to the first frame image.

[0079] Motion module: Drives the three-dimensional tooth surface model to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

[0080] Example 3

[0081] This embodiment also provides an electronic device, see reference. Figure 6 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0082] Specifically, the processor 402 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0083] The memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 404 may include removable or non-removable (or fixed) media. Where appropriate, the memory 404 may be internal or external to a data processing device. In a particular embodiment, the memory 404 is non-volatile memory. In a particular embodiment, the memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0084] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0085] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the methods for simulating and visualizing mandibular movements through motion capture in the above embodiments.

[0086] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0087] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0088] The input / output device 408 is used to input or output information. In this embodiment, the input information may be oral cavity scan data, camera parameters, etc., and the output information may be mandibular movement visualization results, etc.

[0089] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program:

[0090] S101. Obtain the patient's oral cavity scan data and obtain a three-dimensional tooth surface model based on the patient's oral cavity scan data;

[0091] S102. Set at least two markers with different patterns for each marker, fix each marker on the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement.

[0092] S103. Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all the center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the center point of the same marker in adjacent frame images and the rotation matrix.

[0093] S104. Obtain at least two tooth feature points in the first frame of the mandibular motion video, obtain the model corresponding point of each tooth feature point in the three-dimensional tooth surface model, wherein the tooth feature point and its model corresponding point form a corresponding point pair, calculate the registration rotation matrix and registration translation vector of the three-dimensional tooth surface model to the first frame image according to the corresponding point pair, and register the three-dimensional tooth surface model to the first frame image.

[0094] S105. Drive the three-dimensional tooth surface model to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

[0095] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0096] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0097] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 6Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0098] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for simulating and visualizing mandibular movements through motion capture, characterized in that, Includes the following steps: Acquire the patient's oral cavity scan data and obtain a three-dimensional tooth surface model based on the patient's oral cavity scan data; Set at least two markers with different patterns for each marker, fix each marker to the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement. Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the center point of the same marker in adjacent frame images and the rotation matrix. At least two tooth feature points are obtained in the first frame of the mandibular motion video. The model corresponding point of each tooth feature point is obtained in the three-dimensional tooth surface model. The tooth feature point and its model corresponding point form a corresponding point pair. The registration rotation matrix and registration translation vector of the three-dimensional tooth surface model are calculated according to the corresponding point pair, and the three-dimensional tooth surface model is registered to the first frame image. The three-dimensional tooth surface model is driven to move based on the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

2. The method for simulating and visualizing mandibular movement through motion capture according to claim 1, characterized in that, In the step of "setting at least two markers with different patterns for each marker and fixing each marker to the patient's teeth", the markers are black and white marking blocks that are placed on the corresponding teeth of the patient using a transparent material.

3. The method for simulating and visualizing mandibular movement through motion capture according to claim 1, characterized in that, In the step of "calculating the rotation matrix after performing singular value decomposition on the coordinate matrix of adjacent frame images, and calculating the translation vector by taking the center point coordinates of the same landmark in adjacent frame images and the rotation matrix", the singular matrix is ​​obtained by multiplying the transpose of the coordinate matrix of the next frame image in the adjacent frame images by the coordinate matrix of the previous frame image. The singular matrix is ​​decomposed to obtain a left singular matrix and a right singular matrix. The rotation matrix is ​​obtained by multiplying the left singular matrix by the transpose of the right singular matrix. The translation result is obtained by multiplying the center point coordinates of any landmark in the previous frame image in the two adjacent frames by the rotation matrix. The translation vector is obtained by subtracting the translation result from the center point coordinates of the same landmark in the next frame image as the previous frame image.

4. The method for simulating and visualizing mandibular movement through motion capture according to claim 1, characterized in that, In the step of "calculating the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image", the coordinates of each center point in the current frame image are first normalized and the average value of all center point coordinates is calculated to obtain the centroid coordinates. The difference between each center point coordinate and the centroid coordinates is obtained to obtain the difference coordinates. The difference coordinates are saved in the form of a matrix to obtain the coordinate matrix of the current frame image.

5. The method for simulating and visualizing mandibular movement through motion capture according to claim 1, characterized in that, In the step of "obtaining at least two tooth feature points in the first frame of the mandibular motion video", the tooth feature points are evenly distributed on the patient's maxillary and mandibular teeth.

6. The method for simulating and visualizing mandibular movement through motion capture according to claim 1, characterized in that, In the step of "calculating the registration rotation matrix and registration translation vector for registering the three-dimensional tooth surface model to the first frame image based on the corresponding point pairs", the coordinates of the tooth feature points and the coordinates of the corresponding points of the model in each corresponding point pair are obtained. The tooth feature point matrix is ​​formed using all the tooth feature point coordinates, and the model corresponding point matrix is ​​formed using all the model corresponding point coordinates. The sum of the distances between all corresponding point pairs is calculated based on the tooth feature point matrix and the model corresponding point matrix. An error function is constructed based on the sum of the distances between all corresponding point pairs. An initial rotation matrix and an initial translation vector are obtained. The tooth feature point matrix is ​​iteratively optimized using the initial rotation matrix and the initial translation vector until the error function is less than a first set threshold to obtain the registration rotation matrix and the registration translation vector.

7. The method for simulating and visualizing mandibular movement through motion capture according to claim 1, characterized in that, In the step of "driving the three-dimensional tooth surface model to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames", the three-dimensional tooth surface model is driven to perform rigid body transformation according to the rotation matrix and the translation vector, and the transformation result is visualized and rendered.

8. A device for simulating and visualizing mandibular movements through motion capture, characterized in that, Includes the following steps: Acquisition module: Acquires the patient's oral cavity scan data and generates a three-dimensional tooth surface model based on the patient's oral cavity scan data; Calibration module: Set at least two markers with different patterns for each marker, fix each marker on the patient's teeth, use the markers to calibrate and correct the parameters of the RGBD camera, and use the corrected RGBD camera to acquire the mandibular movement video corresponding to the patient's mandibular movement; Construction module: Construct a three-dimensional coordinate system corresponding to the RGBD camera, obtain the center point coordinates of the center point of each marker in each frame of the mandibular motion video in the three-dimensional coordinate system, calculate the coordinate matrix of the current frame image based on the coordinates of all center points in the current frame image, calculate the rotation matrix after performing singular value decomposition on the coordinate matrices of adjacent frame images, and calculate the translation vector by taking the center point coordinates of the same marker in adjacent frame images and the rotation matrix. Registration module: Obtain at least two tooth feature points in the first frame of the mandibular motion video, obtain the model corresponding point of each tooth feature point in the three-dimensional tooth surface model, wherein the tooth feature point and its model corresponding point form a corresponding point pair, calculate the registration rotation matrix and registration translation vector for registering the three-dimensional tooth surface model to the first frame image based on the corresponding point pair, and register the three-dimensional tooth surface model to the first frame image. Motion module: Drives the three-dimensional tooth surface model to move according to the translation vector between two adjacent frames and the rotation matrix between two adjacent frames.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a method for simulating and visualizing mandibular movements through motion capture as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling the process to execute the process, the process including a method for simulating and visualizing mandibular movements through motion capture according to any one of claims 1-7.

Images