A systematic restoration method of temporomandibular joint motion based on multimodal fusion

Abstract

The present invention discloses a method for systematic restoration of temporomandibular joint motion based on multimodal fusion, the steps of which are as follows: static data acquisition; dynamic data acquisition; data combination registration and binding to obtain a facial registration model, a skull maxillary binding model, and a mandibular mandibular binding model; video solution to obtain dynamic lower marker plate position data and bind it to the mandibular mandibular binding model to obtain a mandibular mandibular dynamic binding model, the facial registration model, the skull maxillary binding model, and the mandibular mandibular dynamic binding model are combined to obtain a patient's skull facial dynamic model with mandibular mandibular dynamic information, thereby accurately restoring the patient's true oral state. The restoration method of the present invention directly collects jaw motion data from video images, and after optimization processing, performs motion reproduction and visualization on a human skull model, which can achieve fast and accurate trajectory reproduction, help doctors perform treatment and testing, improve medical efficiency and quality, and take into account economy.

Description

Technical Field

[0001] The present invention relates to the technical field of temporomandibular joint motion restoration, and in particular to a method for systematic restoration of temporomandibular joint motion based on multimodal fusion. Background Art

[0002] The stomatognathic system comprises multiple components, forming a mutually constrained yet coordinated whole. A comprehensive assessment of patients with stomatognathic system dysfunction and a thorough understanding of the static and dynamic contact states of the dentition, the structure of the temporomandibular joint, and motor coordination are crucial for understanding the causes of the disease, the extent of damage to the masticatory system, particularly the temporomandibular joint, and the prognosis of the disease. This is also the foundation for subsequent treatment. In recent years, with the advancement of digital technology, multimodal data fusion has become possible, combining CT, intraoral and extraoral scan data, and electronic facebow mandibular trajectory data, enabling a more comprehensive assessment of the temporomandibular joint and occlusal function.

[0003] The electronic face bow uses digital technology to simulate the movement of the human jaw and joints. It is highly effective in diagnosing jaw disorders, as well as in restoring and treating teeth. Traditional face bow technology requires sophisticated mechanical devices, is complex to operate, difficult to carry, and provides limited information. Existing electronic face bow technology, all developed by foreign medical device companies, mostly utilizes sensor solutions, such as optical and motion sensors, to collect jaw movement data. These devices are extremely expensive. Therefore, a new technical solution is needed that balances cost-effectiveness and practicality.

[0004] With the advancement of modern medicine, a growing number of clinical auxiliary examination methods are emerging, and multimodal medical data is available to assist in the diagnosis and treatment of diseases. Anatomical images primarily describe human morphology, such as cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI), X-rays, and ultrasound. Functional images, such as positron emission tomography (PET) and single-photon emission computed tomography (SPECT), primarily describe human metabolism and function. Furthermore, the motion trajectory of specific body parts can be obtained through various methods. This information, often used in isolation, plays a crucial role in the diagnosis and treatment of diseases. However, in some cases, these single-modality data evaluation methods often have limitations and lack connectivity. Integrating these diverse medical data into a single approach—multimodal data fusion—can leverage the strengths of multiple auxiliary methods.

[0005] Applying the concept of multimodal data fusion to the evaluation of the stomatognathic system can greatly assist dentists in the more comprehensive diagnosis and treatment of stomatognathic system diseases. Because the components within the stomatognathic system are not isolated when evaluating the system, static occlusion should be associated with dynamic articulation, the temporomandibular joint, and the muscles of the stomatognathic system, and observed from the perspective of overall functional relationships. A global assessment of the temporomandibular joint, static and dynamic articulation, and muscles can, to a certain extent, avoid treatment risks, reduce the number of patient visits, assist in the development of long-term treatment plans, and facilitate multidisciplinary collaboration to achieve more satisfactory treatment results.

[0006] Multimodal data fusion involves computer digitization and medical image registration, which aligns internal and external features in spatial coordinates to produce a new, more comprehensive fused image. Appropriate multimodal data fusion can provide clues for disease diagnosis and treatment, maximizing the extraction of useful information from each image. In clinical dentistry, various data types can serve as the basis for multimodal fusion, such as DICOM, STL, OBJ, and XML. DICOM files are acquired by CBCT, STL is the basic storage format for dental scans, and mandibular motion trajectories are typically stored as XML files. The acquisition equipment includes mechanical or three-dimensional recorders based on magnetoelectric conversion, ultrasound positioning, or optical positioning. The purpose of multimodal data fusion is to register at least two different types of data in three dimensions, so that they can reflect comprehensive diagnostic and treatment information simultaneously in time and space. Moreover, existing multimodal technologies do not have the function of exporting mandibular trajectory information, nor can they arbitrarily modify the designated position of the condyle according to the doctor's needs. The facial model cannot move with the movement of the mandible or move independently according to set actions such as smiling and pronunciation of special words, so that doctors can easily check the patient's tooth exposure when performing these movements. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for systematic restoration of temporomandibular joint motion based on multimodal fusion to address the problems existing in the prior art.

[0008] The purpose of the present invention is to be solved by the following technical solutions:

[0009] A method for systematic restoration of temporomandibular joint motion based on multimodal fusion, characterized by the following steps:

[0010] A. Use oral CBCT equipment to obtain static CT data of the patient's oral cavity, use a mobile phone or tablet to obtain static facial data of the patient, and use oral scanning equipment to obtain static scans of the patient's teeth, including the maxillary model and mandibular model.

[0011] B. Bond the marker plate to the patient, use a scanning device to obtain the patient's scanned occlusal model, and use a fixed-position handheld camera to capture the patient's oral movement video and calibration video;

[0012] C. Based on the patient's oral static CT data in step A, obtain a CT skull maxillary model, a CT mandibular model, and a CT facial model;

[0013] D. Convert the patient's facial static data in step A into a textured 3D facial model, and then register it with the CT facial model in step C to obtain a facial registration model;

[0014] E. Using the patient scanned occlusal model in step B as a reference, the static scanned maxillary model and the static scanned mandibular model in step A are moved and registered to the patient scanned occlusal model and fused into an oral scan registered fusion model. Using the CT skull maxillary model in step C as a reference, the oral scan registered fusion model is moved and registered to the CT skull maxillary model. Using the oral scan registered fusion model as a reference, the CT mandibular model in step C and the marker plate model corresponding to the marker plate in step B are moved and registered to the oral scan registered fusion model.

[0015] F. Based on the registration in step E, the static scanned maxillary model in the oral scan registration fusion model, the upper marker plate model in the marker plate model, and the CT skull maxillary model are bound to obtain the skull maxillary bound model; the static scanned mandibular model in the oral scan registration fusion model, the lower marker plate model in the marker plate model, and the CT mandibular mandibular model are bound to obtain the mandibular mandibular bound model;

[0016] G. Obtaining the spatial position coordinate data of the marker in each frame of the patient's oral movement video based on the calibration video in step B and the patient's oral movement video;

[0017] H. Binding the dynamic lower marker plate position data in the marker plate spatial position coordinate data in step G to the mandibular mandibular binding model in step F to obtain the mandibular mandibular dynamic binding model;

[0018] I. Combine the facial registration model in step D, the maxillary rigging model in step F, and the mandibular dynamic rigging model in step H to obtain a patient's skull facial dynamic model with mandibular dynamic information, thereby accurately restoring the patient's true oral state.

[0019] The oral CBCT device in step A can capture the position of the condyle, thereby obtaining static CT data of the patient's oral cavity including the position of the condyle.

[0020] The method for obtaining the static facial data of the patient in step A is: taking photos of the patient's face using a mobile phone or tablet, obtaining at least three rows of photos with more than 5 photos in each row, when taking photos, the left and right rotation angle of the patient's face is ±60°, and the up and down pitch angle of the face is ±50°, and the shooting content includes movements in the smiling open state and the normal closed state.

[0021] The marking plate in step B includes an upper marking plate and a lower marking plate, which are respectively bonded to the labial and buccal sides of the upper and lower dentitions; the patient scanned occlusal model in step B includes a portion bonded to the marking plate and the upper or lower jaw.

[0022] The patient's oral movement video in step B includes but is not limited to the left lateral movement, right lateral movement, forward movement, maximum forward movement, backward movement, and maximum opening movement of the oral cavity; the shooting angle of the handheld camera in step B is perpendicular to the center position of the upper marking plate and the lower marking plate in the marking plate. The handheld camera is generally a mobile phone, tablet computer or other camera that can transmit data to the control system (such as a computer) in real time; the calibration video in step B is consistent with the focal length of the handheld camera of the patient's oral movement video.

[0023] The patient's facial static data in step D is converted into a textured facial 3D model using a point cloud 3D reconstruction algorithm, and then the facial 3D model is imported into the CT facial model in step C for registration.

[0024] The registration in step D and step E refers to: first determining the mobile model and the reference model; then selecting at least three non-collinear feature points on the mobile model and the reference model respectively, and the three non-collinear feature points on the mobile model and the reference model are consistent and need to correspond one to one; then using the ICP algorithm to move and register the mobile model to the reference model.

[0025] The method for obtaining the spatial position coordinate data of the marker plate in step G is as follows: introducing the calibration video in step B, calculating the internal parameter matrix A and the distortion coefficient B of the handheld shooting device through the calibration algorithm, and storing them as a calibration file; introducing the patient's oral movement video in step B, using the posture estimation algorithm in combination with the calibration file, calculating the translation vector t and the rotation vector r of the spatial position of the marker plate in the patient's oral movement video in the virtual three-dimensional coordinate system, and then converting the rotation vector into a 3X3 rotation matrix through the Rodriguez formula, adding the translation vector to the 3X3 rotation matrix, and finally combining them into a 4X4 transformation matrix for representing the obtained spatial position coordinate data of the marker plate in each frame of the patient's oral movement video.

[0026] The marking plate spatial position coordinate data in step G and step H include upper marking plate spatial position coordinate data and lower marking plate spatial position coordinate data. The translation vector t and rotation vector r of the upper marking plate spatial position coordinate data need to be transferred to the lower marking plate spatial position coordinate data to obtain the dynamic lower marking plate position data for binding to the mandibular binding model in step F.

[0027] The points of the spatial coordinates of the lower marker plate in two adjacent frames of video in the mandibular dynamic binding model in step H are connected in sequence, and based on the consistency of the mandibular incision point and the central tooth position bonded to the lower marker plate, and the positional relationship between the left condyle and the right condyle and the mandibular incision point, the mandibular incision point motion trajectory, the left condyle motion trajectory and the right condyle motion trajectory are obtained; the mandibular incision point motion trajectory, the left condyle motion trajectory and the right condyle motion trajectory can calculate the personalized motion characteristic parameters of the patient's oral cavity.

[0028] The present invention has the following advantages over the prior art:

[0029] The present invention provides a method for identifying jaw movements using rigid connection markers and computer vision algorithms. The method directly collects jaw movement data from video images, and after optimization processing, reproduces and visualizes the movement on a human skull model. This method can quickly and accurately reproduce the trajectory, help doctors perform occlusion detection, simplify the previously tedious debugging work, improve medical efficiency and quality, and take into account economic efficiency.

[0030] The digital facebow method based on computer vision of the present invention can ensure accuracy, and has rapid calculation and short waiting time, and can help dentists obtain jaw movement data.

[0031] Compared with other existing sensor solutions, the restoration method of the present invention is low-cost; the technical route of the present invention only requires a mobile phone or tablet to shoot a video to obtain the dynamic information of the mandibular bone, effectively reducing the solution cost.

[0032] Compared with other existing technical solutions, the restoration method of the present invention is simple to operate. In the past, both electronic and mechanical face bows required manual operation by doctors, but the present invention only requires the use of a fixed handheld camera to complete video acquisition, which is simple and quick. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart of the method for systematic restoration of temporomandibular joint motion based on multimodal fusion of the present invention;

[0034] Figure 2 A schematic structural diagram of a marking plate in an embodiment of the present invention;

[0035] Figure 3This is the effect diagram after static data registration and fusion of the embodiment provided by the present invention;

[0036] Figure 4 This is an example of the sagittal projection trajectory and calculation diagram of the mandibular tangent point movement according to an embodiment of the present invention. The unit of measurement is millimeter (mm).

[0037] Among them: 1—CT skull maxillary model; 2—CT mandibular model; 3—facial 3D model; 4—static scanning maxillary model; 5—static scanning mandibular model; 6—marker plate; 61—upper marker plate; 62—lower marker plate. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] like Figure 1-3As shown: A method for systematic restoration of temporomandibular joint movement based on multimodal fusion, comprising the following steps: using an oral CBCT device to obtain static CT data of the patient's oral cavity, using a mobile phone or tablet to obtain static facial data of the patient, using an oral scanning device to obtain a static scanning maxillary model and a static scanning mandibular model of the patient's teeth; bonding a marking plate to the patient, using a scanning device to obtain a scanning occlusal model of the patient, using a fixed-setting handheld shooting device to obtain a patient's oral movement video and a calibration video, wherein the calibration video has the same focal length as the handheld shooting device of the patient's oral movement video; decomposing the patient's oral static CT data to obtain a CT head The patient's facial static data is converted into a textured facial 3D model using a point cloud 3D reconstruction algorithm using a cranial maxillary model, a CT mandibular model, and a CT facial model. The facial 3D model is registered with the CT facial model to obtain a facial registration model. Taking the patient's scanned occlusal model as a reference, the static scanned maxillary model and the static scanned mandibular model are moved and registered to the patient's scanned occlusal model and fused into an oral scan registration fusion model. Taking the CT skull maxillary model as a reference, the oral scan registration fusion model is moved and registered to the CT skull maxillary model. Taking the oral scan registration fusion model as a reference, the CT mandibular model and the marker plate model corresponding to the marker plate are moved. The CT head maxillary model is aligned with the oral scan registration fusion model; on the basis of the above alignment, the static scanned maxillary model in the oral scan registration fusion model, the upper marker plate model in the marker plate model, and the CT skull maxillary model are bound to obtain the skull maxillary bound model; the static scanned mandibular model in the oral scan registration fusion model, the lower marker plate model in the marker plate model, and the CT mandibular mandibular model are bound to obtain the mandibular mandibular bound model; based on the calibration video and the patient's oral movement video, the marker plate spatial position coordinate data in each frame of the patient's oral movement video is obtained, and the marker plate spatial position coordinate data includes the upper marker plate spatial position coordinate data and the lower marker plate spatial position coordinate data. To set the coordinate data, it is necessary to transfer the translation vector t and the rotation vector r of the upper marker plate spatial position coordinate data to the lower marker plate spatial position coordinate data to obtain the dynamic lower marker plate position data for binding to the mandibular mandibular binding model in step F; bind the dynamic lower marker plate position data in the marker plate spatial position coordinate data to the mandibular mandibular binding model to obtain the mandibular mandibular dynamic binding model; combine the facial alignment model, the skull maxillary binding model and the mandibular mandibular dynamic binding model to obtain the patient's skull facial dynamic model with mandibular mandibular dynamic information, thereby accurately restoring the patient's true oral state.

[0040] In the above-mentioned restoration method, the oral CBCT equipment is required to be able to capture the position of the condyle, and then obtain the patient's oral static CT data including the position of the condyle; the method for obtaining the patient's facial static data is: use a mobile phone or tablet to take pictures of the patient's face, and obtain at least three rows of pictures, with more than 5 pictures in each row. When taking pictures, the patient's face is rotated left and right at an angle of ±60° and the face is pitched up and down at an angle of ±50°. The shooting content includes movements in the open-mouth smiling state and the normal closed-mouth state.

[0041] like Figure 3 As shown, the marker plate includes an upper marker plate and a lower marker plate, which are respectively bonded to the labial and buccal sides of the upper and lower dentitions. The patient's scanned occlusal model includes the portion bonded to the marker plate and the upper or lower jaw. The patient's oral movement video includes, but is not limited to, left lateral movement, right lateral movement, protrusive movement, maximum protrusive movement, retrusive movement, and maximum opening movement. The handheld camera's viewing angle is perpendicular to the center of the upper and lower marker plates on the marker plate. The handheld camera is generally a mobile phone, tablet computer, or other device capable of transmitting data in real time to a control system (such as a computer).

[0042] The registration mentioned in the technical solution of the present invention refers to: first determining the mobile model and the reference model; then selecting at least three non-collinear feature points on the mobile model and the reference model respectively, and the three non-collinear feature points on the mobile model and the reference model are consistent and need to correspond one to one; then the mobile model is moved and registered to the reference model through the ICP algorithm.

[0043] In the above method, the method for obtaining the spatial position coordinate data of the marker plate is as follows: introducing the calibration video in step B, calculating the intrinsic parameter matrix A and the distortion coefficient B of the handheld shooting device through the calibration algorithm, and storing them as a calibration file; introducing the patient's oral movement video in step B, using the posture estimation algorithm in combination with the calibration file, calculating the translation vector t and rotation vector r of the spatial position of the marker plate in the patient's oral movement video in the virtual three-dimensional coordinate system, and then converting the rotation vector into a 3X3 rotation matrix through the Rodriguez formula, adding the translation vector to the 3X3 rotation matrix, and finally combining them into a 4X4 transformation matrix for representing the obtained spatial position coordinate data of the marker plate in each frame of the patient's oral movement video.

[0044] In the above method, the points of the spatial coordinates of the lower marker plate in two adjacent frames of video in the mandibular dynamic binding model are connected in sequence, and based on the consistency of the mandibular incision point and the central tooth position to which the lower marker plate is bonded, and the positional relationship between the left condyle and the right condyle and the mandibular incision point, the mandibular incision point motion trajectory, the left condyle motion trajectory and the right condyle motion trajectory are obtained. The mandibular incision point motion trajectory, the left condyle motion trajectory and the right condyle motion trajectory can calculate the personalized motion characteristic parameters of the patient's oral cavity.

[0045] Example

[0046] A systematic restoration method of temporomandibular joint motion based on multimodal fusion, such as Figure 1 , including the following steps:

[0047] Step 1: Static data collection:

[0048] Step 11: Acquire static oral CT data. Use an oral CBCT device capable of capturing the condyle to obtain a DICOM sequence file. During the CBCT scan, the patient should bite some filler (such as cotton) to separate the upper and lower teeth slightly to facilitate model segmentation after 3D reconstruction. During the scan, the patient should remain standing as much as possible to avoid motion artifacts that could affect the accuracy of the 3D reconstruction.

[0049] Step 12: Collect static facial data of the patient. Use a mobile phone or tablet to take photos of the patient's face, obtaining at least three rows of photos, with at least five photos in each row. When taking photos, the patient's face should be rotated left and right at an angle of ±60° and pitched up and down at an angle of ±50°. The photos should include: smiling with open mouth and normal closed mouth. During the photo shooting process, the patient should remain as still as possible with eyes open, and avoid closing their eyes at the moment of shooting.

[0050] Step 13: Acquire an oral scan static model. Use an oral scanner to scan the patient's upper and lower jaws to obtain a static scan upper jaw model and a static scan lower jaw model of the patient's teeth.

[0051] Step 2: Dynamic data collection:

[0052] Step 21, bonding the marking plate. The marking plate 6 has a marking code for positioning identification on the top and an adjustable structure at the tail. The arc and angle for bonding can be adjusted according to the patient's oral situation. Figure 2 The marking plate 6 shown, Figure 3 The upper marking plate 61 and the lower marking plate 62 shown;

[0053] Step 22: Scanning the occlusal data. Due to individual differences in bonding between operators, the acquisition of occlusal data is adjusted after bonding the marking plate to effectively reduce operational errors. When scanning the occlusal data, the characteristic data of the bonding area needs to be scanned together to directly obtain the patient's scanned occlusal model.

[0054] Step 23: Video data collection is divided into two parts:

[0055] Step 231: Calibration video shooting. Place the phone or tablet about 20cm-30cm away from the calibration plate, shoot the calibration plate from multiple angles, and manually adjust the focus during shooting to make the marker plate as clear as possible in the viewfinder of the phone or tablet. Record the specific focus value.

[0056] Step 232: Shoot a video of the patient's oral movements. The shooting distance should be kept at about 20cm-30cm as much as possible. The focal length must be kept the same as that of the calibration video. The shooting angle of view should be as perpendicular as possible to the initial positions of the upper marking plate 61 and the lower marking plate 62. The video of the patient's oral movements includes but is not limited to the following movements: left lateral movement, right lateral movement, and forward movement.

[0057] Step 3: Static data registration and fusion:

[0058] Step 31: Static data preprocessing, CT data preprocessing: Based on the 3D reconstruction algorithm in VTK, adjust the HU value to obtain the CT facial model and CT skull model respectively. The CT skull model is further divided into the CT skull maxillary model 1 and the CT mandibular model 2 using the segmentation algorithm; Facial static data preprocessing: Based on the AliceVision photogrammetry computer vision framework, several photos are taken and the point cloud reconstruction algorithm is used to generate a textured facial 3D model 3;

[0059] Step 32: Static data registration and fusion. The result after registration in this step is as follows: Figure 3 As shown in the figure, the relative positions of the CT facial model and the CT maxillary model 1 cannot be changed during the registration process. The facial three-dimensional model 3 is registered with the CT facial model to obtain a facial registration model; with the patient's scanned occlusal model as a reference, the static scanned maxillary model 4 and the static scanned mandibular model 5 are registered to the patient's scanned occlusal model through the ICP registration algorithm and fused into an oral scan registration fusion model; with the CT maxillary model 1 as a reference, the oral scan registration fusion model is moved and registered to the CT maxillary model 1, and the oral scan registration fusion model is used as the reference. The CT mandibular model 2 in step C and the marker plate model corresponding to the marker plate in step B are moved and registered to the oral scan registration fusion model. After the registration is completed, the static scanned maxillary model 4 in the oral scan registration fusion model, the upper marker plate model in the marker plate model, and the CT skull maxillary model 1 are bound to obtain the skull maxillary bound model. The static scanned mandibular model 5 in the oral scan registration fusion model, the lower marker plate model in the marker plate model, and the CT mandibular model 2 are bound to obtain the mandibular bound model.

[0060] Step 4: Solve the video to obtain the dynamic lower marker position data:

[0061] Step 41: Calculate the calibration video, convert the calibration video into an image, extract part of the image as needed, and then detect the ChArUco calibration plate in the image. According to the detection algorithm, the intrinsic parameter matrix M1 and distortion matrix M2 of the shooting phone or tablet are obtained.

[0062] Step 42, the patient's oral movement video is solved, each frame of the patient's oral movement video is grayed and threshold adjusted for preprocessing, the identification code on the marker plate 6 is better recognized in the posture estimation, and each frame of the preprocessed picture, the internal parameter matrix M1 and the distortion matrix M2 of the mobile phone or tablet are imported into the posture estimation algorithm to obtain the identification code. The position information of the identification code in each frame of the picture corresponds to the three-dimensional space, which is represented by the translation vector t and the rotation vector r. Then, the rotation vector is converted into a 3X3 rotation matrix through the Rodriguez formula, and the translation vector is added to the 3X3 rotation matrix. The matrix is ​​then combined into a 4X4 transformation matrix, which is used to represent the spatial position coordinate data of the marker plate in each frame of the patient's oral movement video. Since the upper marker plate is bound to the oral scan maxillary model and the CT skull maxillary model, and the actual maxillary body is fixed relative to the skull maxillary body during oral movement, it is impossible to completely avoid the position of the upper marker plate in the video image from being completely stationary during video shooting. Therefore, the translation vector t and rotation vector r of the upper marker plate need to be transferred to the lower marker plate, so that the dynamic lower marker plate position data finally obtained will be more accurate.

[0063] Step 5, dynamic data binding:

[0064] Step 51: Bind dynamic mandibular data. Bind the dynamic lower marker plate position data to the mandibular binding model to obtain a dynamic mandibular binding model. The position data of the mandibular binding model is consistent with the dynamic lower marker plate position data. As the time frame moves, the mandibular binding model can change its spatial coordinates accordingly. In medicine, the three reference points for mandibular movement are the mandibular incision point (IP), the left condyle (LP), and the right condyle (RP). In the patient's oral movement video, the mandibular incision point is basically consistent with the central tooth position where the lower marker plate is bonded. The positions of the left and right condyles can be manually specified based on the CT mandibular model. By sequentially connecting the points of the lower marker plate's spatial coordinates in the two video frames (the upper marker plate is set to be fixed, and the jitter error caused by shooting is superimposed on the spatial coordinates of the lower marker plate), the motion trajectory of the mandibular incision point (IP) can be generated. Similarly, the motion trajectories of the left condyle (LP) and the right condyle (RP) can be obtained. In order to facilitate observation and calculation, the motion trajectories of the mandibular incision point (IP), left condyle (LP), and right condyle (RP) are projected, and the parameters of the patient's oral personalized motion characteristics are calculated based on the trajectory, including but not limited to the following characteristic parameters: several characteristic parameters of mandibular incision point movement such as left lateral movement angle, right lateral movement angle, protrusive movement angle, retrograde movement distance, maximum opening distance, opening type deviation angle, etc., characteristic parameters of condylar movement such as left protrusive condyle inclination, right protrusive condyle inclination, left Bennett angle, right Bennett angle, left instantaneous lateral displacement distance, right instantaneous lateral displacement distance, etc. The projection trajectory and calculation diagram example (sagittal plane projection of mandibular incision point movement) are as follows: Figure 4 shown.

[0065] Step 52, facial dynamic data binding. In the process of combining the facial registration model, the maxillary skull binding model and the mandibular dynamic binding model, it is necessary to set different weight values ​​for the points on the facial registration model according to the direction and density of the facial muscle tissue, and then specify the mandibular dynamic binding model as the object to drive facial movement. After dynamic binding, the points on the three-dimensional facial model will follow the mandibular dynamic binding model to perform soft-body movements, so that a patient's skull facial dynamic model with mandibular dynamic information can be obtained.

[0066] The present invention provides a method for identifying jaw movements using rigid connection markers and computer vision algorithms. The method directly collects jaw movement data from video images, and after optimization processing, reproduces and visualizes the movement on a human skull model. This method can quickly and accurately reproduce the trajectory, help doctors perform occlusion detection, simplify the previously tedious debugging work, improve medical efficiency and quality, and take into account economic efficiency.

[0067] The above embodiments are only for illustrating the technical ideas of the present invention and cannot be used to limit the scope of protection of the present invention. Any changes made on the basis of the technical solutions in accordance with the technical ideas proposed by the present invention fall within the scope of protection of the present invention; any technologies not involved in the present invention can be implemented by existing technologies.

Claims

1. A method for systematic restoration of temporomandibular joint motion based on multimodal fusion, characterized by: A. Use oral CBCT equipment to obtain static CT data of the patient's oral cavity, use a mobile phone or tablet to obtain static facial data of the patient, and use oral scanning equipment to obtain static scans of the patient's teeth, including the maxillary model and mandibular model. B. Bond a marker plate to the patient, use an oral scanning device to obtain a scanned occlusal model of the patient, and use a fixed-position handheld camera to capture a video of the patient's oral movements and a calibration video; C. Based on the patient's oral static CT data in step A, obtain a CT skull maxillary model, a CT mandibular model, and a CT facial model; D. Convert the patient's facial static data in step A into a textured 3D facial model, and then register it with the CT facial model in step C to obtain a facial registration model; E. Using the patient scanned occlusal model in step B as a reference, the static scanned maxillary model and the static scanned mandibular model in step A are moved and registered to the patient scanned occlusal model and fused into an oral scan registered fusion model. Using the CT skull maxillary model in step C as a reference, the oral scan registered fusion model is moved and registered to the CT skull maxillary model. Using the oral scan registered fusion model as a reference, the CT mandibular model in step C and the marker plate model corresponding to the marker plate in step B are moved and registered to the oral scan registered fusion model. F. Based on the registration in step E, the static scanned maxillary model in the oral scan registration fusion model, the upper marker plate model in the marker plate model, and the CT skull maxillary model are bound to obtain the skull maxillary bound model; the static scanned mandibular model in the oral scan registration fusion model, the lower marker plate model in the marker plate model, and the CT mandibular mandibular model are bound to obtain the mandibular mandibular bound model; G. Obtaining the spatial position coordinate data of the marker in each frame of the patient's oral movement video based on the calibration video in step B and the patient's oral movement video; H. Binding the dynamic lower marker plate position data in the marker plate spatial position coordinate data in step G to the mandibular mandibular binding model in step F to obtain the mandibular mandibular dynamic binding model; I. Combine the facial registration model in step D, the maxillary rigging model in step F, and the mandibular dynamic rigging model in step H to obtain a patient's skull facial dynamic model with mandibular dynamic information, thereby accurately restoring the patient's true oral state.

2. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to claim 1, characterized in that: The oral CBCT device in step A can capture the position of the condyle, thereby obtaining static CT data of the patient's oral cavity including the position of the condyle.

3. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to claim 1, characterized in that: The method for obtaining the static facial data of the patient in step A is: taking photos of the patient's face using a mobile phone or tablet, obtaining at least three rows of photos with more than 5 photos in each row, when taking photos, the left and right rotation angle of the patient's face is ±60°, and the up and down pitch angle of the face is ±50°, and the shooting content includes movements in the smiling open state and the normal closed state.

4. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to claim 1, characterized in that: The marking plate in step B includes an upper marking plate and a lower marking plate, which are respectively bonded to the labial and buccal sides of the upper and lower dentitions; the patient scanned occlusal model in step B includes a portion bonded to the marking plate and the upper or lower jaw.

5. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The patient's oral movement video in step B includes but is not limited to the left lateral movement, right lateral movement, forward movement, maximum forward movement, backward movement, and maximum opening movement of the oral cavity; the shooting angle of the handheld shooting device in step B is perpendicular to the center position of the upper marking plate and the lower marking plate in the marking plate; the calibration video in step B is consistent with the focal length of the handheld shooting device of the patient's oral movement video.

6. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The patient's facial static data in step D is converted into a textured facial 3D model using a point cloud 3D reconstruction algorithm, and then the facial 3D model is imported into the CT facial model in step C for registration.

7. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The registration in step D and step E refers to: first determining the mobile model and the reference model; then selecting at least three non-collinear feature points on the mobile model and the reference model respectively, and the three non-collinear feature points on the mobile model and the reference model are consistent and need to correspond one to one; then using the ICP algorithm to move and register the mobile model to the reference model.

8. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The method for obtaining the spatial position coordinate data of the marker plate in step G is as follows: introducing the calibration video in step B, calculating the internal parameter matrix A and the distortion coefficient B of the handheld shooting device through the calibration algorithm, and storing them as a calibration file; introducing the patient's oral movement video in step B, using the posture estimation algorithm in combination with the calibration file, calculating the translation vector t and the rotation vector r of the spatial position of the marker plate in the patient's oral movement video in the virtual three-dimensional coordinate system, and then converting the rotation vector into a 3X3 rotation matrix through the Rodriguez formula, adding the translation vector to the 3X3 rotation matrix, and finally combining them into a 4X4 transformation matrix for representing the obtained spatial position coordinate data of the marker plate in each frame of the patient's oral movement video.

9. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to claim 8, characterized in that: The marking plate spatial position coordinate data in step G and step H include upper marking plate spatial position coordinate data and lower marking plate spatial position coordinate data. The translation vector t and rotation vector r of the upper marking plate spatial position coordinate data need to be transferred to the lower marking plate spatial position coordinate data to obtain the dynamic lower marking plate position data for binding to the mandibular binding model in step F.

10. The method for systematic restoration of temporomandibular joint motion based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The points of the spatial coordinates of the lower marker plate in two adjacent frames of video in the mandibular dynamic binding model in step H are connected in sequence, and based on the consistency of the mandibular incision point and the central tooth position bonded to the lower marker plate, and the positional relationship between the left condyle and the right condyle and the mandibular incision point, the mandibular incision point motion trajectory, the left condyle motion trajectory and the right condyle motion trajectory are obtained; the mandibular incision point motion trajectory, the left condyle motion trajectory and the right condyle motion trajectory can calculate the personalized motion characteristic parameters of the patient's oral cavity.

Images