Orthodontic treatment monitoring method and device based on multi-view jaw posture estimation

Abstract

This invention discloses a method and device for monitoring orthodontic treatment based on multi-view jaw pose estimation. Given a pre-orthodontic 3D jaw model and a post-orthodontic RGB oral scan video, a tooth video instance segmentation model is first trained to obtain an instance segmentation mask for each tooth. An instance propagation module is designed, which can construct instance relationships in the temporal domain using only bounding box positions, class scores, and semantic segmentation masks. This invention employs a temporal consistency loss to learn feature encodings that maintain high similarity between frames for the same instance. Then, a data-driven method combined with multi-view geometric relationships is used to optimize and estimate the jaw pose of the fixed tooth frame. Bidirectional constrained pose tracking is used to infer the jaw pose in frames not containing fixed teeth. Finally, an iterative method is used to estimate the relative pose change of each orthodontic tooth. This method can accurately estimate the pose changes of orthodontic teeth even without obtaining a post-orthodontic 3D jaw model, achieving effective monitoring of orthodontic treatment.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and device for monitoring orthodontic treatment based on multi-view jaw posture estimation. Background Technology

[0002] Orthodontic treatment is an important dental treatment method because malocclusion increases the incidence of tooth decay, causes psychological discomfort, harms health, and reduces quality of life. While orthodontic treatment is helpful, patients need to visit the dental clinic regularly to assess whether the results of each stage of orthodontic treatment are meeting expectations, making the process cumbersome and complex. With the development of artificial intelligence, enabling patients to scan and capture their teeth using simple and portable RGB / RGB-D devices for remote monitoring of orthodontic patients, this has gained attention in the medical and academic communities because it not only reduces the time costs for orthodontists but also reduces social costs, providing convenience for orthodontic patients.

[0003] However, orthodontic treatment monitoring differs from image-based 6D object pose estimation because, in orthodontic treatment monitoring, the current image is compared only with the 3D model reconstructed in the previous stage of orthodontic treatment, while object 6D pose estimation compares both the 3D model and the image at the same time point. Furthermore, orthodontic treatment monitoring focuses on the relative pose changes of individual teeth before and after orthodontic treatment, while object 6D pose estimation is used to estimate the pose of the object relative to the camera. Additionally, in orthodontic treatment monitoring, tooth texture is weak and their position changes across different frames, making segmentation of each tooth challenging. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and device for orthodontic treatment monitoring based on multi-view jaw posture estimation.

[0005] The objective of this invention is achieved through the following technical solution: a method for monitoring orthodontic treatment based on multi-view jaw posture estimation, the method comprising the following steps:

[0006] S1, acquire the RGB oral cavity scan video of the patient at the current stage;

[0007] S2, Train a tooth video instance segmentation model based on the oral cavity scan video to obtain the instance segmentation mask for each tooth in the video;

[0008] S3, the jaw pose of each frame is inferred using a multi-view jaw pose estimation method, including the following sub-steps:

[0009] S3.1, Multi-view jaw 6D pose estimation based on fixed teeth, specifically: For frames with fixed teeth, the initial 6D pose of the fixed teeth is predicted according to the ZebraPose algorithm, the relative pose of the camera is obtained according to the pose of the same fixed teeth in different frames, the rationality of the relative pose of the camera is judged by the adjacent teeth of the fixed teeth, and the fixed tooth pose is updated based on the bundle adjustment loss using the reasonable relative pose of the camera.

[0010] S3.2, Bidirectional 6D jaw pose tracking, specifically: a frame that does not contain fixed teeth is taken as the target frame to be tracked, and the jaw pose in the target frame is estimated by using the visual features of the nearest neighboring frames containing fixed teeth through a bidirectional recurrent neural network.

[0011] S4 uses an iterative method to estimate the relative pose change of each orthodontic tooth during orthodontic treatment, minimizing the pixel matching loss between the projection of the orthodontic tooth 3D model before orthodontics and the foreground image of the orthodontic tooth after orthodontics in each iteration.

[0012] Furthermore, the input to the tooth video instance segmentation model is a set of frames {I} of length T extracted from the video. t}, t=1,2,...,T, these frames are processed through a backbone network to extract features, resulting in a feature map {x} for each frame. t Then, these feature maps are encoded using a transformer encoder with added positional encoding (PE). Next, the frame query is input into the transformer decoder to obtain the instance query. The instance query is then passed through the instance propagation module to obtain the predicted instance bounding box and instance category. Combined with the pixel-level semantic information obtained by the transformer encoder through deconvolution and upsampling layers, the tooth instance segmentation mask is obtained.

[0013] Furthermore, the instance propagation module includes two branches: an instance bounding box prediction branch and an instance class prediction branch; the input of the instance propagation module assumes that the i-th instance query in frame t is... Through initialization The query weight is propagated to the (t+1)th frame;

[0014] (1) Instance bounding box prediction branch: The instance bounding box of each instance is propagated between frames by learning the offsets of position and scale between frames. The i-th instance bounding box r in frame t is... t i It can be obtained through the following formula:

[0015]

[0016] Where σ(·) is the sigmoid function, Wr It is the weight matrix that needs to be learned; in the first frame of the video, the instance box r1 i Query by instance The mapping function is obtained, that is

[0017] (2) Instance Class Prediction Branch: Since the appearance of an instance is inconsistent in different frames, the class probability distribution of the instance will change. The class vector of the i-th instance in frame t is... for:

[0018]

[0019] Where σ(·) is the sigmoid function, W c and W t is the weight matrix to be learned, f is the index of the storage frame. For the t-th frame, its storage frames are the td-th frame to the (t-1)-th frame, d is the preset number of storage frames, and Concate(·) is the concatenation function.

[0020] Furthermore, the temporal consistency loss L is designed in the tooth video instance segmentation model. tc Instances appearing in the same frame in different frames form a positive set, while instances appearing in different frames form a negative set, resulting in a temporal consistency loss L. tc The calculation formula is as follows:

[0021]

[0022] Where d is the preset number of storage frames, f is the index of the storage frame, k is the number of instances whose highest probability value of the class vector corresponding to the instance is higher than the set threshold, and τ is the decay coefficient.

[0023] Furthermore, the total loss function is the instance classification loss L. cls Instance frame loss L box Semantic segmentation loss L mask and time consistency loss L tc The linear combination of is given by the following formula:

[0024] L total =λ c L cls +λ b L box +λ m L mask +L tc

[0025] Where, λ c , λ b , λ m These are coefficients used to balance different constraints.

[0026] Furthermore, the multi-view jaw 6D pose estimation based on fixed teeth specifically includes:

[0027] The tooth foreground region obtained by segmenting each frame instance in the video. As input, where t is the frame index, the initial 6D pose T of the fixed tooth lm in frame t is predicted according to the ZebraPose algorithm. t,lm ;

[0028] Assuming that both frame s and frame t contain a fixed tooth lm, As for the relative camera posture, the reasonableness of the relative camera posture is evaluated in the following ways:

[0029] Let n be the index of the teeth adjacent to the fixed tooth lm, and let these teeth appear simultaneously in the same frame, T s,n and T s,t T t,n The distance between them is measured as follows:

[0030]

[0031] Where, γ n It is the set of surface point clouds of tooth n;

[0032] The calculated distance D is compared with a preset threshold C. If D < C, the camera's relative pose is determined to be reasonable and can be used for subsequent loss function calculation; otherwise, it is deemed unreasonable and discarded.

[0033] The posture T of the fixed tooth lm t,lm Recovery is achieved using the bundle adjustment loss function, as shown in the following formula:

[0034]

[0035] Where, χ lm It is a set of surface point clouds of a fixed tooth lm, S s,lm It is the foreground region of a fixed tooth lm in the s-th frame extracted by instance segmentation, and π is the camera intrinsic parameter matrix.

[0036] Furthermore, the bidirectional jaw 6D pose tracking specifically includes:

[0037] For frames that do not contain fixed teeth, the jaw pose is estimated by pose tracking, and the starting frame of tracking must contain fixed teeth to obtain the initial jaw pose during the tracking process;

[0038] Assume the input video contains {I} s ,...,I t ,...,I u}, where s, t, and u are frame indices; feature extraction is performed on these frames using a backbone network to obtain visual features {x}. s ,...x t ,...x u Assume the jaw pose T in frame s and frame u. s and T u It was obtained by using fixed teeth; the visual features x of the t-th frame. t and the latent vector h of frame t-1 t-1 Input the forward ConvLSTM layer to estimate the forward latent vector h of frame t. t and forward output vector The forward output vector and the backward latent vector g of frame t+1 t+1 Input the backward ConvLSTM layer to estimate the backward latent vector g of frame t. t and backward output vector The backward output vector The jaw pose of frame t is estimated after a linear mapping layer. The forward latent vector h in the first frame s-1 And the backward latent vector g in the last frame u+1 Obtained through linear mapping layers respectively;

[0039] During the training phase, the L1 norm loss function L in the jaw 6D pose space is optimized. error Assume the true 6D jaw pose in frame t is T. t ,have:

[0040]

[0041] To enhance the smoothness of pose prediction across frames, a regularization loss term L is added. reg :

[0042]

[0043] The total loss function L total It is a linear combination of the above two items:

[0044] L total =L error +λ reg L reg

[0045] Where, λ reg These are combined weights used to control the degree of smoothness.

[0046] Furthermore, step S4 specifically includes:

[0047] The jaw pose obtained in frame t from S3 is used as the initial pose of the orthodontic teeth ln in frame t. The input is fed into the posture update module, which continuously compares the projection of the orthodontic tooth 3D model before orthodontics with the foreground image of the orthodontic tooth after orthodontics to update the posture changes of the orthodontic teeth during the orthodontic treatment process.

[0048] For the 3D jaw model Z before orthodontic treatment, the 3D model of the orthodontic teeth is first obtained through a 3D instance segmentation algorithm; in the it-th iteration, the projection image of the orthodontic teeth ln before orthodontic treatment is obtained. It is achieved by using a 3D model of orthodontic teeth. ln And the 6D pose estimated in the (it-1)th iteration The obtained; observation image I o,ln It was obtained by extracting the ln region of orthodontic teeth through instance segmentation;

[0049] Projected image and observed image I o,ln The input is fed into the pose update module to predict the relative pose change of the orthodontic tooth ln in the it-th iteration.

[0050] The loss function used is pixel matching loss L. pm This is used to penalize the difference between the projected image and the observed image.

[0051]

[0052] Where π is the camera intrinsic parameter matrix, It is a 3D model of orthodontic teeth Z ln The color value of the 3D point i on the surface. It is the observed image I o,ln The pixel color value corresponding to 3D point i; pixel matching loss is used to find the projected image in the it-th iteration. and observed image I o,ln Relative attitude changes between

[0053] The present invention also provides an orthodontic treatment monitoring device based on multi-view jaw posture estimation, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the above-mentioned orthodontic treatment monitoring method based on multi-view jaw posture estimation.

[0054] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-described orthodontic treatment monitoring method based on multi-view jaw posture estimation.

[0055] Compared with the prior art, the beneficial effects of this invention are as follows:

[0056] 1. This invention proposes a two-dimensional-three-dimensional registration method, which uses a 3D tooth model before orthodontics and an RGB oral scan video after orthodontics to estimate the relative pose changes of orthodontic teeth during orthodontic treatment.

[0057] 2. This invention introduces an instance propagation module, in which the temporal association of instances is modeled by three factors: box position, class score, and semantic segmentation mask, which greatly reduces computational complexity and memory consumption.

[0058] 3. This invention designs a time consistency loss that makes the relationship between positive samples greater than the relationship between negative samples.

[0059] 4. This invention explores and studies orthodontic knowledge as an intermediate factor in local rigid body deformation registration.

[0060] 5. This invention combines deep learning methods with geometric optimization to improve the effectiveness of multi-view jaw pose estimation. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall framework provided in an embodiment of the present invention;

[0062] Figure 2 This is a flowchart of tooth video instance segmentation provided in an embodiment of the present invention;

[0063] Figure 3 This is a flowchart of 6D jaw pose estimation based on fixed teeth provided in an embodiment of the present invention;

[0064] Figure 4 This is a flowchart of bidirectional jaw 6D posture tracking provided in an embodiment of the present invention;

[0065] Figure 5 This is a flowchart of an iterative tooth pair provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] This invention provides a method for monitoring orthodontic treatment based on multi-view jaw posture estimation, comprising the following steps:

[0068] S1, acquire the RGB oral cavity scan video of the patient at the current stage;

[0069] S2, Train a tooth video instance segmentation model based on the oral cavity scan video to obtain the instance segmentation mask for each tooth in the video;

[0070] S3, the jaw pose of each frame is inferred using a multi-view jaw pose estimation method, including the following sub-steps:

[0071] S3.1, Multi-view jaw 6D pose estimation based on fixed teeth, specifically: For frames with fixed teeth, the initial 6D pose of the fixed teeth is predicted according to the ZebraPose algorithm, the relative pose of the camera is obtained according to the pose of the same fixed teeth in different frames, the rationality of the relative pose of the camera is judged by the adjacent teeth of the fixed teeth, and the fixed tooth pose is updated based on the bundle adjustment loss using the reasonable relative pose of the camera.

[0072] S3.2, Bidirectional 6D jaw pose tracking, specifically: a frame that does not contain fixed teeth is taken as the target frame to be tracked, and the jaw pose in the target frame is estimated by using the visual features of the nearest neighboring frames containing fixed teeth through a bidirectional recurrent neural network.

[0073] S4 uses an iterative method to estimate the relative pose change of each orthodontic tooth during orthodontic treatment, minimizing the pixel matching loss between the projection of the orthodontic tooth 3D model before orthodontics and the foreground image of the orthodontic tooth after orthodontics in each iteration.

[0074] Furthermore, the input to the tooth video instance segmentation model is a set of frames {I} of length T extracted from the video. t}, t=1,2,...,T, these frames are processed through a backbone network (ResNet-50 is used in this embodiment) to extract features, resulting in a feature map {x} for each frame. t Then, these feature maps are encoded using a transformer encoder with added position encoding (PE) (in this embodiment, the transformer encoder has 5 stacked unit blocks); next, the frame query (including position query and content query) is input into the transformer decoder to obtain the instance query, and then the instance query is passed through the instance propagation module to obtain the predicted instance box B and instance category C, and then combined with pixel-level semantic information M (in this embodiment, deconvolution and upsampling layers are used to obtain the category vector of each pixel in the image, which corresponds to the semantic segmentation loss during training) to obtain the tooth instance segmentation mask.

[0075] Furthermore, the instance propagation module includes two branches: an instance bounding box prediction branch and an instance class prediction branch; the input of the instance propagation module assumes that the i-th instance query in frame t is... Through initialization The query weight is propagated to the (t+1)th frame;

[0076] (1) Instance bounding box prediction branch: The instance bounding box of each instance is propagated between frames by learning the offsets of position and scale between frames. The i-th instance bounding box r in frame t is... t i It can be obtained through the following formula:

[0077]

[0078] Where σ(·) is the sigmoid function, W r It is the weight matrix that needs to be learned; in the first frame of the video, the instance box r1 i Query by instance The mapping function is obtained, that is

[0079] (2) Instance Class Prediction Branch: Since the appearance of an instance is inconsistent in different frames, the class probability distribution of the instance will change. The class vector of the i-th instance in frame t is... for:

[0080]

[0081] Where σ(·) is the sigmoid function, W c and W t is the weight matrix to be learned, f is the index of the storage frame. For the t-th frame, its storage frames are the td-th frame to the (t-1)-th frame. d is the preset number of storage frames. Concate(·) is the concatenation function used to concatenate the class vector of the i-th instance across all storage frames.

[0082] Furthermore, in the tooth video instance segmentation model, the present invention designs a temporal consistency loss L... tc Instances appearing in the same frame in different frames form a positive set, while instances appearing in different frames form a negative set, resulting in a temporal consistency loss L. tc The calculation formula is as follows:

[0083]

[0084] Where d is the preset number of storage frames, f is the index of the storage frame, k is the number of instances whose highest probability value of the category vector corresponding to the instance is higher than the set threshold, and τ is the attenuation coefficient, which is 2.0 in this embodiment.

[0085] The total loss function is the instance classification loss L. cls Instance frame loss L box Semantic segmentation loss L mask and time consistency loss L tc The linear combination of is given by the following formula:

[0086] Ltotal =λ c L cls +λ b L box +λ m L mask +L tc

[0087] Where, λ c , λ b , λ m It is a coefficient used to balance different constraints; in this embodiment, λ is set. c =2.0, λ b =5.0, λ m =2.0.

[0088] Furthermore, the multi-view jaw 6D pose estimation based on fixed teeth specifically includes:

[0089] The tooth foreground region obtained by segmenting each frame instance in the video. As input, where t is the frame index, the initial 6D pose T of the fixed tooth lm in frame t is predicted according to the ZebraPose algorithm. t,lm T t,lm It is a homogeneous matrix, including a 3×3 3D rotation matrix R. t,lm and a 3D translation vector t t,lm ;

[0090] Assuming that both frame s and frame t contain a fixed tooth lm, As for the relative camera posture, the reasonableness of the relative camera posture is evaluated in the following ways:

[0091] Let n be the index of the teeth adjacent to the fixed tooth lm, and let these teeth appear simultaneously in the same frame, T s,n and T s,t T t,n The distance between them is measured as follows:

[0092]

[0093] Where, γ n It is the set of surface point clouds of tooth n;

[0094] The calculated distance D is compared with a preset threshold C. If D < C, the camera's relative pose is determined to be reasonable and can be used for subsequent loss function calculation; otherwise, it is deemed unreasonable and discarded.

[0095] The posture T of the fixed tooth lm t,lm Recovery is achieved using the bundle adjustment (BA) loss function, as shown in the following formula:

[0096]

[0097] Where, χ lm It is a set of surface point clouds of a fixed tooth lm, S s,lm It is the foreground region of a fixed tooth lm in the s-th frame extracted by instance segmentation, and π is the camera intrinsic parameter matrix.

[0098] Furthermore, the bidirectional jaw 6D pose tracking specifically includes:

[0099] For frames that do not contain fixed teeth, the jaw pose is estimated by pose tracking, and the starting frame of tracking must contain fixed teeth to obtain the initial jaw pose during the tracking process;

[0100] Assume the input video contains {I} s ,...,I t ,...,I u}, where s, t, and u are frame indices; a backbone network (ResNet50 in this embodiment) is used to extract features from these frames to obtain visual features {x}. s ,...x t ,...x u Without loss of generality, assume that the jaw pose T in frame s and frame u is... s and T u It was obtained by using fixed teeth; the visual features x of the t-th frame. t and the latent vector h of frame t-1 t-1 Input the forward ConvLSTM layer to estimate the forward latent vector h of frame t. t and forward output vector The forward output vector and the backward latent vector g of frame t+1 t+1 Input the backward ConvLSTM layer to estimate the backward latent vector g of frame t. t and backward output vector The backward output vector The jaw pose of frame t is estimated using a linear mapping layer FC3(·). The formula is as follows:

[0101]

[0102]

[0103]

[0104] Among them, the forward latent vector h in the first frame (i.e., the s-th frame) s-1 And the backward latent vector g in the last frame (i.e., the u-th frame).u+1 These are obtained through linear mapping layers FC1(·) and FC2(·), respectively.

[0105] h s-1 =FC1(T s )

[0106] g u+1 =FC2(T u )

[0107] During the training phase, the L1 norm loss function L in the jaw 6D pose space is optimized. error Assume the true 6D jaw pose in frame t is T. t ,have:

[0108]

[0109] To enhance the smoothness of pose prediction across frames, a regularization loss term L is added. reg :

[0110]

[0111] The total loss function L total It is a linear combination of the above two items:

[0112] L total =L error +λ reg L reg

[0113] Where, λ reg This is a combination weight used to control the smoothness; in this embodiment, λ is set. reg =2.0.

[0114] Furthermore, step S4 specifically includes:

[0115] The jaw pose obtained in frame t from S3 is used as the initial pose of the orthodontic teeth ln in frame t. The input is fed into the posture update module (DeepIM network is used in this embodiment), which updates the posture changes of the orthodontic teeth during the orthodontic treatment process by continuously comparing the projection of the 3D model of the orthodontic teeth before orthodontics with the foreground image of the orthodontic teeth after orthodontics.

[0116] For the 3D jaw model Z before orthodontic treatment, the 3D model of the orthodontic teeth is first obtained through a 3D instance segmentation algorithm; in the it-th iteration, the projection image of the orthodontic teeth ln before orthodontic treatment is obtained. It is achieved by using a 3D model of orthodontic teeth. ln And the 6D pose estimated in the (it-1)th iteration The obtained; observation image I o,lnIt was obtained by extracting the ln region of orthodontic teeth through instance segmentation;

[0117] Projected image and observed image I o,ln The input is fed into the pose update module to predict the relative pose change of the orthodontic tooth ln in the it-th iteration.

[0118] The loss function used is pixel matching loss L. pm This is used to penalize the difference between the projected image and the observed image.

[0119]

[0120] Where π is the camera intrinsic parameter matrix, It is a 3D model of orthodontic teeth Z ln The color value of the 3D point i on the surface. It is the observed image I o,ln The pixel color value corresponding to 3D point i; pixel matching loss is used to find the projected image in the it-th iteration. and observed image I o,ln Relative attitude changes between

[0121] The orthodontic treatment monitoring method based on multi-view jaw posture estimation provided in this embodiment can accurately monitor the degree of orthodontic treatment. The process of monitoring teeth in intraoral RGB video data using this embodiment includes two parts: training and testing. The orthodontic treatment monitoring method used in this embodiment is described below with reference to the accompanying drawings.

[0122] Figure 1 This is a schematic diagram of the overall framework provided in an embodiment of the present invention; it mainly includes three parts: video instance segmentation, multi-view jaw pose estimation module, and tooth alignment. The model is trained using PyTorch on four RTX 3090 graphics cards.

[0123] The datasets used for video instance segmentation were the Shining3D and Aoralscan3 tooth instance segmentation datasets. The training, validation, and test sets of the Shining3D dataset had sample sizes of 1866, 272, and 272, respectively, while the Aoralscan3 dataset had sample sizes of 1573, 244, and 244, respectively. All images were 640x480 pixels and labeled using LabelMe. Average precision (AP) was used as the evaluation metric at 50% and 75% as IoU thresholds. Data augmentation was employed to expand the dataset to approximately 160,000 images, including vertical / horizontal flipping, translation variance adjustment, and scaling. ResNet-50 was used as the backbone network. The Transformer encoder and decoder consisted of five modules. AdamW was used as the optimizer with a momentum of 0.9 and a weight decay of 2.0 × 10⁻⁶. -3 Each mini-batch contains 4 samples. The initial learning rate is 4.0 × 10⁻⁶. -3 Then, after 70,000 iterations, it decreased to 2.0 × 10. -3 .

[0124] Tooth and jaw pose estimation was performed using the Shining3D 3D tooth pose dataset and the Shining3D tooth registration dataset. Jaw models were derived from hospital patient scans. The true relative pose of each tooth was generated by adding random perturbations to the tooth models. The Shining3D 3D tooth pose dataset contained 1689, 150, and 150 samples for training, validation, and testing, respectively. The constructed Aoralscan3 tooth registration dataset included 1667 training samples, 156 validation samples, and 176 test samples. ADD-S and ADD(-S) and their AUC were used as evaluation metrics. Data augmentation methods, such as rotation and translation, were employed to expand the dataset. The network was trained for 100 epochs using AdamW with a weight decay of 1.0 × 10⁻⁶. -4 The batch size is 4. The learning rate is 2.0 × 10⁻⁶. -2 And within each epoch, at 4.0 × 10 -3 The step size decays exponentially. A threshold of 0.1 mm is set for the matching radius.

[0125] During testing, given an RGB oral cavity scan video, the foreground region of each tooth is first segmented into video instances. Then, multi-view pose evaluation and bidirectional pose tracking are used to infer the jaw pose. Finally, tooth alignment is performed iteratively, outputting the rotational and translational offsets of the teeth. Performance evaluations on the Shining3D tooth pose dataset and the Aoralscan tooth registration dataset validate the effectiveness of the proposed method.

[0126] Figure 2 This is a flowchart of tooth video instance segmentation provided in an embodiment of the present invention, where PE is a position code;

[0127] Figure 3 This is a flowchart of 6D jaw pose estimation based on fixed teeth provided in an embodiment of the present invention;

[0128] Figure 4 This is a flowchart of bidirectional jaw 6D pose tracking provided in an embodiment of the present invention, wherein FC is a fully connected network;

[0129] Figure 5 This is a flowchart of an iterative tooth pair provided in an embodiment of the present invention.

[0130] This invention also provides an orthodontic treatment monitoring device based on multi-view jaw posture estimation, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the above-mentioned orthodontic treatment monitoring method based on multi-view jaw posture estimation.

[0131] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described orthodontic treatment monitoring method based on multi-view jaw posture estimation.

[0132] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of orthodontic treatment monitoring based on multi-view jaw pose estimation, characterized in that, The method comprises the following steps: S1, acquiring an RGB oral scanning video of a patient in a current stage; S2, training a tooth video instance segmentation model based on the oral scanning video to obtain an instance segmentation mask of each tooth in the video; S3, inferring a jaw pose of each frame by using a multi-view jaw pose estimation method, comprising the following sub-steps: S3.1, multi-view jaw 6D pose estimation based on fixed teeth, specifically: for a frame with fixed teeth, an initial 6D pose of the fixed teeth is predicted according to the ZebraPose algorithm, a camera relative pose is obtained according to the poses of the same fixed teeth in different frames, the reasonableness of the camera relative pose is judged by using the adjacent teeth of the fixed teeth, and the fixed tooth pose is updated based on the bundle adjustment loss by using the reasonable camera relative pose; S3.2, bidirectional jaw 6D pose tracking, specifically: taking a frame not containing fixed teeth as a target frame to be tracked, using visual features of the frames containing fixed teeth closest to the front and back of the target frame, and estimating the jaw pose in the target frame by using a bidirectional recurrent neural network; S4, estimating the relative pose change of each orthodontic tooth in the orthodontic treatment process by using an iterative method, and minimizing the pixel matching loss between the projection of the 3D model of the orthodontic tooth before orthodontic treatment and the foreground image of the orthodontic tooth after orthodontic treatment in each iteration process; specifically: The first obtained from S3 Jaw pose in frame as the first In-frame orthodontic teeth initial posture The image is input into the posture update module, which updates the posture changes of the orthodontic teeth during orthodontic treatment by continuously comparing the projection of the orthodontic tooth 3D model before orthodontics with the foreground image of the orthodontic teeth after orthodontics. 3D jaw model before orthodontics First, a 3D model of orthodontic teeth is obtained using a 3D instance segmentation algorithm; in the second step... In this iteration, the pre-orthodontic teeth Projected image It is achieved by using 3D models of orthodontic teeth. and the The 6D pose estimated in the next iteration Obtained; Observational images Extracting orthodontic teeth through instance segmentation Obtained from the region; projected images and observed images are input into a pose update module to predict an orthodontic tooth relative pose changes in the first iteration .

2. The method of claim 1, wherein, The input of the tooth video instance segmentation model is a frame set with a length of extracted from the video , , the frames are subjected to feature extraction through a backbone network to obtain a feature map of each frame ; then the feature maps are encoded using a transformer encoder with position encoding PE; next, the frame query is input into a transformer decoder to obtain an instance query, and then the instance query is subjected to instance propagation to obtain a predicted instance frame and an instance category, and combined with the pixel-level semantic information obtained through the deconvolution and up-sampling layer of the transformer encoder encoder to obtain a tooth instance segmentation mask.

3. The method of claim 2, wherein, The instance propagation module includes two branches: an instance box prediction branch and an instance class prediction branch; the input of the instance propagation module assumes that the first... The first frame The query for each instance is as follows Through initialization The query weight is propagated to the first frame; (1) Instance bounding box prediction branch: The instance bounding box of each instance is propagated between frames by learning the offsets of position and scale between frames. The first frame Instance box It can be obtained through the following formula: wherein, is a sigmoid function, is a weight matrix to be learned; in the 1st frame of the video, instance box by instance query a mapping function, i.e. ; (2) Instance class prediction branch: As the appearance of an instance is inconsistent in different frames, the class probability distribution of the instance will change, and the class vector of the i-th instance in the j-th frame is: ​​​ wherein, is a sigmoid function, and is a weight matrix to be learned, is an index of a stored frame, for the frame, the stored frame is the frame to the frame, is a preset number of stored frames, is a stitching function.

4. The method of claim 1, wherein, The tooth video instance segmentation model is designed with a temporal consistency loss The same instance appearing in different frames constitutes a positive set, while different instances appearing in different frames constitute a negative set, and the temporal consistency loss The calculation formula is as follows: wherein, is a preset storage frame number, is an index of a storage frame, is the number of instances whose highest probability value of the category vector corresponding to the instance is higher than a set threshold value, is a decay coefficient.

5. The method of claim 4, wherein, In the tooth video instance segmentation model, a total loss function is a linear combination of an instance classification loss , an instance frame loss , a semantic segmentation loss , and a temporal consistency loss , and the formula is as follows: wherein , , are coefficients for balancing different constraints.

6. The method of claim 1, wherein, The multi-view jaw 6D pose estimation based on fixed teeth is specifically: The tooth foreground region segmented from each frame instance in the video As input, wherein is the index of the frame, the initial 6D pose of the fixed tooth in the frame is predicted according to the ZebraPose algorithm The initial 6D pose ; Assume that the first frame and the second frame both contain a fixed tooth , the camera relative pose is evaluated for reasonableness by: Let be the index of the tooth adjacent to the fixed tooth and these teeth appear simultaneously in the same frame, and the distance metric between them is as follows: wherein, is a set of surface point clouds of teeth ​ The calculated distance is compared with a preset threshold value If the camera relative pose is determined to be reasonable and can be used for subsequent loss function calculation, otherwise it is determined to be unreasonable and is discarded. Fixing the pose of the teeth of the teeth Recovery through bundle adjustment loss function, formula as follows: in, It is a fixed tooth surface point cloud collection, The first is extracted through instance segmentation. Fixed teeth in frame Foreground area, It is the camera intrinsic parameter matrix.

7. The method of claim 1, wherein, The bidirectional jaw 6D pose tracking is specifically: For frames not containing fixed teeth, the jaw bone pose is estimated by pose tracking, and the starting frame of tracking needs to contain fixed teeth to obtain the initial pose of the jaw bone in the tracking process; Assume the input video contains the following frames: ,in , and It is the index of the frame; the backbone network is used to extract features from these frames to obtain visual features. Assume the first Frame and the Jaw pose in the frame and It was achieved by using fixed teeth; the first Visual features of frames and the Latent vectors of frames Input to the forward ConvLSTM layer to estimate the th Forward latent vector of a frame and forward output vector ; the forward output vector and the Frame backward latent vector The input is fed back to a ConvLSTM layer to estimate the th... Frame backward latent vector and backward output vector ; the backward output vector The first value is obtained after estimation through a linear mapping layer. Jaw pose of the frame The forward latent vector in the first frame And the backward latent vector in the last frame Obtained through linear mapping layers respectively; In the training phase, the L1 norm loss function in the jaw 6D pose space is optimized , assuming the true jaw 6D pose in the -th frame is , we have: To enhance the smoothness of the inter-frame predicted poses, a regularization loss term is added : Total loss function is a linear combination of the two above. wherein, is a combination weight for controlling the degree of smoothing.

8. The method of claim 1, wherein, In step S4, the loss function used is the pixel matching loss for penalizing the difference between the projected image and the observed image: in, It is the camera intrinsic parameter matrix. It is a 3D model of orthodontic teeth. 3D points on color value, It is an observed image Top and 3D points The corresponding pixel color value; pixel matching loss is used to find the first... Projected image in the next iteration and observation images Relative attitude changes between .

9. An orthodontic treatment monitoring device based on multi-view jaw pose estimation, comprising a memory and one or more processors, the memory having stored therein executable code, the device characterized in that, The processor executes the executable code, and is configured to implement the orthodontic treatment monitoring method based on multi-view jaw pose estimation according to any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor, and the orthodontic treatment monitoring method based on multi-view jaw pose estimation according to any one of claims 1-8 is implemented.

Images