Virtual projection 3D oral and maxillofacial key point detection method based on deep learning

Abstract

The invention discloses a virtual projection 3D oral and maxillofacial key point detection method based on deep learning, and aims to solve the problem of insufficient 3D image key point detection precision and efficiency in the prior art. The method comprises the following steps: projecting 3D image data into a plurality of 2D projection images by using a virtual projection technology; then key point detection and marking are carried out on the 2D image by using a deep learning network model, and 2D mark point coordinates are obtained; and finally, performing 3D reconstruction by adopting a virtual back projection method, and back-projecting the 2D mark point information to a 3D space to obtain 3D mark point coordinates. According to the method, 2D key point detection and 3D projection and reconstruction technologies are combined, rapid, high-precision and automatic detection of 3D image mark key points is realized, and the method can be widely applied to the fields of oral medical image analysis and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision regression technology, and in particular to a method for automatically detecting key feature points in oral and maxillofacial CBCT images. Background Art

[0002] Maxillofacial deformities refer to structural abnormalities or functional impairments of the skull and facial soft and hard tissues during development, and are complex conditions in oral and maxillofacial surgery. Maxillofacial deformities can significantly affect a patient's facial appearance, occlusal function, respiratory system, and speech abilities, and in severe cases, may lead to psychological and social interaction problems. Accurate preoperative assessment of the deformity is crucial to ensure the success of surgical treatment. Analyzing key features in mandibular CBCT can provide a comprehensive assessment of jawbone position, facial soft tissue condition, and related functions, enabling early intervention and providing precise guidance for surgical planning.

[0003] Currently, the assessment of jawbone position, facial soft tissue condition, and related functions still primarily relies on manual annotation of key points. This diagnostic method is time-consuming, labor-intensive, and highly susceptible to subjective human factors. Therefore, there is an urgent need in the dental clinic to develop an automated detection technology for key points in mandibular CBCT images to assist clinicians in assessing mandibular deformities and improve clinical diagnostic efficiency.

[0004] With the impressive performance of deep learning in tasks such as image target detection and recognition, some scholars have applied deep learning models to the field of oral medical image processing and analysis. For example, Katherine WLVig et al., in "Treatment planning principles for orthodontic-surgical correction of dento-facial anomalies," proposed a 3D classification standard for maxillary and mandibular developmental abnormalities and combined it with the analysis of facial soft tissues and occlusal relationships to provide guidance for orthognathic surgery. Koji Hashimoto et al., in "Diagnostic efficacy of three-dimensional images by helical CT for lesions in the maxillofacial region," applied CBCT technology to assess complex craniofacial skeletal deformities and proposed a 3D surgical simulation method based on CBCT images, providing theoretical support for personalized surgical planning. Yue et al., in "Downregulation of serum microRNA-205 as a potential diagnostic and prognostic biomarker for human glioma," first used a corrected active shape model to detect 12 anatomical landmarks and proposed a method for automatically identifying abnormalities in skull anatomy. While the aforementioned studies have achieved some success in processing oral medical images, the recognition accuracy of 3D oral medical images is not high and requires enormous computing power. False detections may lead to misjudgments in clinical decision-making, and insufficient computing power may delay surgical scheduling. Existing 3D point detection methods are still not ideal for large-scale datasets and real-time clinical applications, and the long training and inference times limit their application in real-world clinical environments. Summary of the Invention

[0005] Based on the aforementioned problems, this invention is proposed.

[0006] To address the aforementioned technical problems, this invention proposes a method for automatically detecting key feature points in oral and maxillofacial CBCT images. The method involves projecting a 3D maxillofacial CBCT image onto a 2D plane using virtual projection, then automatically detecting and annotating the mapping of key points in the projection image using a deep learning-based 2D point detection model. Finally, the mapping between the projection image and the key points is reconstructed into 3D space using virtual back projection, thus completing the automatic detection and annotation of key points in the oral and maxillofacial CBCT image.

[0007] The method for automatically detecting key feature points in CBCT images of the mandibular face includes the following steps:

[0008] Step 1: Data preprocessing: Preprocess the 3D oral and maxillofacial medical images, integrate the DICOM format CBCT images into NIFTI format 3D images for convenient subsequent processing, manually annotate the feature key points in the NIFTI images and obtain the voxel coordinates of the key points in 3D space.

[0009] Step 2: Virtual Projection: Based on the virtual projection algorithm, the preprocessed 3D volume data is virtually projected. A ray passes through the volume data from a fixed projection center, and the cumulative intensity value of all points on the projection path is recorded. A 2D virtual projection map is generated according to the defined projection angles. These perspectives can completely cover the geometric features of the mandibular face. Due to the overlap of different projection perspectives, the key points projected onto the 2D plane will have multiple projection points distributed in a certain way. To obtain accurate 2D projection points, this invention uses the weighted centroid method to aggregate and calculate these points, obtaining the mapping coordinates of the key points in the 3D image at each angle of the virtual projection map. This projection method decomposes the complex 3D problem into several independent 2D problems, reducing the computational complexity while ensuring the accuracy of key point detection.

[0010] Step 3: Model Training: Input the generated 2D virtual projection map into the point detection deep learning model, extract the coordinates of 2D key points on each projection map, and generate an automatic point detection model for virtual projection images after training.

[0011] Step 4: Perform the virtual projection operation in Step 2 on the 3D mandibular CBCT image to be detected, input the virtual projection image into the automatic point detection model in Step 3, automatically detect and label the 2D projection points, and obtain the coordinates of the projection points.

[0012] Step 5: Virtual backprojection. The 2D projection point coordinates obtained in Step 4 are virtually backprojected. Based on the geometric parameters recorded in the virtual projection (including the distance from the X-ray source to the rotation center, the distance from the detector to the rotation center, the projection angle, etc.), the backprojection mapping of each 2D keypoint in 3D space is calculated. The intersection points of the rays in 3D space are determined by constructing a multi-view ray equation system, obtaining the reconstructed 3D keypoint coordinates. The 3D keypoints are then labeled onto the 3D mandibular CBCT image, completing the automatic labeling.

[0013] The technical solution further defined in this invention is:

[0014] In a preferred embodiment of the 3D mandibular facial key point detection method described in this invention, the method for constructing the virtual projection model involves first establishing a geometric cone-beam model of the virtual projection in 3D space. The vertices (i.e., the emission points of the rays) correspond to the viewpoint of the virtual projection, and the base of the cone-beam is a 2D projection plane. The distance between the projection plane and the vertices of the cone-beam is set as the focal length, and the size and resolution of the projection plane are adjusted according to the size of the input 3D image and the target accuracy.

[0015] Ray tracing generates projection points: For the projection viewpoint, a series of rays are emitted towards the projection plane. These rays originate from the cone beam apex and pass through each keypoint in the 3D image space. For each ray, its intersection with a voxel in the 3D image is calculated, and the position of the projection point in the 2D projection plane is determined. To avoid distortion of geometric information or uncertainty in the position of keypoints during projection, ray tracing technology is used to directly simulate the propagation path of the rays in the cone beam and is mapped strictly according to the spatial coordinates of the keypoints in the 3D image. During projection, the relative positional relationship between the ray direction and the keypoints in 3D space is preserved to ensure that the 2D projection points can accurately reflect the geometric characteristics of the keypoints.

[0016] Then, based on the intersection points of the rays and the projection plane, corresponding projection images are generated on the 2D projection plane. The projection images represent the contribution of voxels in the 3D image to the rays in the form of grayscale values ​​or pixel intensities. By aggregating the projection points of each ray, a final 2D virtual projection map is generated, where each keypoint appears as a local region with a high-density distribution in the projection map. During the projection process, the step size and accuracy of the ray tracing algorithm are adjusted to adapt to 3D images of different resolutions and complexities. Through multi-view projection, keypoints in the 3D image are projected onto the 2D plane from multiple angles, further enhancing the projection results' ability to express geometric properties.

[0017] As a preferred embodiment of the 3D oral and mandibular facial key point detection method described in this invention, the specific process of calculating the mapping coordinates of key points in the projection map is as follows: based on the principle of virtual projection, all projection rays related to the target key points form a point set on the 2D plane, and this point set exhibits certain local distribution characteristics. The weighted centroid method is used to calculate the aggregation point, and each projection point in the point set is assigned a weight. The weight is usually based on the spatial density of the point (the density of the projection point distribution) and the light intensity value or gray value of the point (reflecting the importance of the corresponding voxel). The centroid position is calculated using the following formula:

[0018]

[0019] Where, x i and y i Let w be the coordinates of the i-th projection point. iThe corresponding weights are used. The result obtained through the weighted centroid method is used as the final position of the 2D projection point. This method can effectively fuse projection point information within a local area and eliminate the bias caused by the distribution of projection points;

[0020] As a preferred embodiment of the 3D oral and maxillofacial key point detection method of the present invention, the virtual back projection accurately calculates the position of key points in 3D space based on the projection matrix and key point depth information. The specific implementation process includes the following steps: First, based on the geometric parameters of the forward projection, for each projection angle θ... i Calculate the virtual ray source S i Coordinates:

[0021] S x =source_origin×sinθ i

[0022] S y = -source_origin×cosθ i

[0023] Where source_origin is the preset distance from the ray source to the center of rotation.

[0024] Then, the corresponding point D on the detector plane is calculated based on the detector center coordinates and the pixel offset of the projection point. i Therefore, the ray equation is constructed as follows:

[0025] X = S i +t×(D i -S i )

[0026] For each key point, collect its linear equations at N angles, and construct a set of virtual orthogonal projection constraint equations.

[0027] [Id i ×d i T X = [Id] i ×d i T ]S i (i = 1, 2, ..., N)

[0028] Where d i Let I be the unit vector of the ray direction of the i-th viewpoint, and let I be the identity matrix. The system of equations is transformed into a least squares optimization problem, and the matrix factorization technique SVD is used to solve it to obtain the reconstructed coordinates of the key points in 3D space.

[0029] As a preferred embodiment of the 3D oral and maxillofacial key point detection method described in this invention, compared with the prior art, the beneficial effects of this invention are as follows: This invention decomposes the 3D problem into multiple 2D problems through virtual projection, reducing the complexity of directly processing key points in 3D space, saving model training and inference time, and enabling the method to still operate efficiently under limited computing resources. Simultaneously, the weighted centroid method is used to accurately calculate the coordinates of the projection points, significantly improving the key point positioning accuracy on the 2D virtual projection map, thereby indirectly improving the detection accuracy of 3D key points from multiple perspectives. By covering the oral and maxillofacial geometry through multi-view virtual projection, the morphological features of the target area can be comprehensively captured, making the key point detection process more robust and effectively reducing detection errors caused by occlusion, data noise, or changes in key point shape. The entire process from data preprocessing and key point extraction to 3D back-projection annotation is automated, greatly reducing the workload of manual annotation and improving the efficiency of key point annotation. This method can be flexibly applied to fields such as oral and maxillofacial anatomy analysis, disease diagnosis, and surgical planning. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of an implementation method for the training phase of the present invention.

[0031] Figure 2 This is a schematic diagram of an implementation method during the testing phase of the present invention.

[0032] Figure 3 This is a schematic diagram of a virtual back projection embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of the CBCT image and key points reconstructed by virtual back projection in this invention. DETAILED DESCRIPTION

[0034] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0035] Example 1, referring to Figures 1-4 As an embodiment of the present invention, a method for detecting key points in 3D oral and maxillofacial structures based on deep learning is provided, comprising the following steps:

[0036] Step 1: Data Preprocessing

[0037] Forty sets of oral and maxillofacial CBCT medical images were converted from DICOM format to NIFTI format to facilitate subsequent data processing and use in deep learning models. The NIFTI format better represents the multidimensional characteristics of medical image data. Key maxillofacial features, such as important anatomical features, were manually annotated in the NIFTI format 3D images. The annotation results are saved in voxel coordinates (x, y, z) in 3D space, providing the basic data for subsequent virtual projection.

[0038] Step 2: Virtual Projection

[0039] Reference Figure 1 A virtual projection model is constructed using the virtual geometric cone-beam projection principle. Rays are emitted from a virtual viewpoint, passing through 3D volume data, and the cumulative intensity value of each ray passing through a voxel is recorded. The apex of the cone-beam model corresponds to the projection viewpoint, and the base of the cone-beam is the projection plane. The resolution of the base is set to 1200×1200 based on the size of the input 3D image. The distance from the virtual viewpoint to the 3D image is 3000 pixels, and the distance from the detector to the object is also set to 3000 pixels, equal to the distance from the ray source to the object, ensuring the stability of the projection model. For each projection viewpoint, the path of the ray passing through the volume data is simulated, and a corresponding projection point is generated on the projection plane. The relative positional relationship between the ray path and the geometric structure of the 3D image is ensured to be consistent. The 2D coordinate calculation formula for the projection point is as follows:

[0040]

[0041] Where P is the projection matrix, x, y, z are the coordinates of the keypoints in 3D space, and u, v are the projection points on the 2D plane. Since keypoints projected onto the 2D plane may form multiple projection point distribution areas, a weighted centroid method is used for aggregation calculation to improve the accuracy of the projection points: based on the spatial density and grayscale distribution of the projection points, each point is assigned a weight; the centroid coordinates are calculated using the following formula:

[0042]

[0043] Where, x i and y i Let ω be the coordinates of the i-th projection point. i The corresponding weights are used. The final result serves as the mapping coordinates of the keypoints on the 2D projection map. 360 uniformly distributed projection angles from 0 to 2π are selected to virtually project the 3D image, generating multiple 2D projection maps covering the complete mandibular facial geometry and the mapped projection points of the keypoints. These projection maps provide rich geometric information for subsequent model training.

[0044] Step 3: Model Training

[0045] First, prepare the training data. Use the 2D virtual projection map generated in step 2 as the input data, and map the corresponding keypoints to the projection point coordinates as the label data to construct the training dataset. A deep learning-based keypoint detection model (PFLD in this example) is used to extract the coordinates of keypoints in each projection map. The model structure includes a multi-layer convolutional neural network for extracting key feature points in the image; the loss function used is WingLoss, calculated as follows:

[0046]

[0047] Where x is the error between the predicted and the true values, ω is the threshold of the loss function, and ∈ controls the nonlinearity of the error, or other loss functions suitable for keypoint regression can be used to optimize model performance. The normalized mean squared error (NME) is calculated as follows:

[0048]

[0049] Where N is the total number of key points, P i Let g be the coordinates of the i-th predicted keypoint. i Let the coordinates of the i-th true keypoint be used, or other metrics suitable for keypoint regression problems be used as evaluation metrics for model accuracy. Through iterative optimization using a large-scale training dataset, an automatic point detection model on a 2D virtual projection map is obtained. After training, a deep learning model capable of automatically detecting keypoints in a 2D projection map is generated for keypoint detection tasks.

[0050] Step 4: Key Point Detection

[0051] Reference Figure 2 Repeat the virtual projection process in step 2 on the 3D oral and maxillofacial CBCT image to be detected to generate multiple 2D projection maps. Input the generated 2D projection maps into the point detection model trained in step 3 to automatically detect and label key points on the projection maps and obtain the 2D coordinates of each key point.

[0052] Step 5: Virtual Back Projection

[0053] Based on the geometric parameters of the forward projection, for each projection angle θ i Calculate the virtual ray source S i Coordinates and detector center point D0 i Coordinates:

[0054]

[0055] Where source_origin is the preset distance from the ray source to the center of rotation.

[0056]

[0057] Where origin_det is the distance from the detector plane to the center of rotation. To determine the detector direction vector, two orthogonal basis vectors of the detector plane are defined as follows:

[0058]

[0059] Let the detected projection point pixel coordinates be (u i ,v i The center pixel coordinates of the detector are (u c ,v c If the physical size of a pixel is δ, then the three-dimensional detector point D... i for:

[0060] D i =D0 i +δ(u i -u c )u (i) +δ(u i -u c )v (i)

[0061] Then, the corresponding point D on the detector plane is calculated based on the detector center coordinates and the pixel offset of the projection point. i Therefore, the ray equation is constructed as follows:

[0062] X = S i +t×(D i -S i )

[0063] For each key point, collect its linear equations at N angles, and construct a set of virtual orthogonal projection constraint equations.

[0064] [Id i ×d i T X = [Id] i ×d i T ]S i (i = 1, 2, ..., N)

[0065] Where d i Let I be the unit vector of the ray direction at the i-th viewpoint, and let I be the identity matrix. The system of equations is transformed into a least-squares optimization problem, solved using SVD (Single-Square Decomposition). The constraint equations for the N viewpoints are stacked to form a 3N×3 coefficient matrix A and a 3N×1 constant term b.

[0066]

[0067] Solve the objective function using the least squares method:

[0068]

[0069] Solve using Singular Value Decomposition (SVD):

[0070]

[0071] Finally, the coordinates of the reconstructed 3D key points are obtained, and the 3D key points are labeled on the 3D mandibular CBCT image to complete the automatic labeling.

[0072] Step 6: Scenario Application

[0073] The labeled images can be used for downstream applications such as anatomical analysis, disease diagnosis, or surgical planning. Specific application scenarios include the following:

[0074] Oral and maxillofacial anatomy analysis: This method, through precise detection of key points in 3D oral and maxillofacial images, can reconstruct the 3D maxillofacial structure of patients, providing comprehensive anatomical information. It can be used to analyze the spatial relationships of teeth, mandible, maxilla, and surrounding soft tissues, and accurately locate important anatomical landmarks such as condyles, mandibular angles, and alveolar bone margins. It provides reliable data support for pathological structural studies of maxillofacial deformities, such as symmetry analysis of dentofacial deformities and bone structure assessment of joint disorders.

[0075] Disease Diagnosis: Based on key point detection and 3D structural reconstruction, this technology can assist doctors in the accurate diagnosis of various maxillofacial diseases. For common diseases such as mandibular fractures, jaw tumors, and temporomandibular joint disorder (TMJ), it provides intuitive imaging assessments and diagnostic evidence through key point displacement and anatomical structural changes. In diseases such as periodontal disease and osteoporosis, the progression of the disease and the degree of bone tissue damage can be assessed by detecting spatial changes in bone density and key points.

[0076] Surgical Planning: In maxillofacial surgeries (such as orthognathic surgery, implant surgery, and jawbone reconstruction surgery), this method can be used for precise preoperative planning and real-time intraoperative guidance. By detecting the location of key maxillofacial points before surgery and combining this with a 3D reconstruction model, it helps surgeons design the optimal surgical path and cutting plan, ensuring surgical precision and safety. For dental implant surgery, the optimal implant location in the alveolar bone can be determined based on key point detection, avoiding damage to critical nerves and blood vessels. For jawbone reconstruction surgery, it provides intraoperative reference points for the positioning and morphological matching of bone graft materials.

[0077] Postoperative assessment and efficacy monitoring: This method can also be used for postoperative efficacy assessment. By comparing the changes in the location and structure of key points before and after surgery, the surgical results can be quantitatively analyzed. During the postoperative recovery process, continuous monitoring of key point displacement can promptly identify potential problems, such as postoperative complications or abnormal recovery of structures.

Claims

1. A method for detecting key points in 3D oral and maxillofacial structures using virtual projection based on deep learning, characterized in that... The following steps are involved: Step 1: Preprocess the 3D oral and maxillofacial CBCT medical images by converting the Dicom format data into NIFTI format data, manually annotating the feature key points in the converted images and obtaining their voxel coordinates in 3D space. Step 2: Based on the virtual projection algorithm, the preprocessed 3D volume data is virtually projected to generate a 2D virtual projection map. Multiple projection points are projected and distributed in a certain way in the 2D plane. The weighted centroid method is used to aggregate and calculate these points to obtain the mapping coordinates of the key points in the 3D image at each angle of the virtual projection map. Step 3: Use the generated 2D virtual projection map as data and the key point mapping coordinates as labels, input them into the deep learning-based point detection model, extract the 2D key point coordinates on each projection map, and generate an automatic point detection model for the virtual projection image after training. Step 4: Perform the virtual projection operation of Step 2 on the 3D oral and maxillofacial CBCT image to be detected, input the virtual projection image into the automatic point detection model of Step 3, automatically detect and label the 2D projection points, and obtain the coordinates of the projection points. Step 5: Perform virtual backprojection on the 2D projection point coordinates obtained in Step 4. Based on the geometric parameters recorded in the virtual projection, calculate the backprojection mapping of each 2D key point in 3D space. Determine the intersection points of rays in 3D space by constructing a multi-view ray equation system to obtain the reconstructed 3D key point coordinates. Label the 3D key points on the 3D mandibular CBCT image to complete the automatic labeling.

2. The method according to claim 1, characterized in that, The virtual projection algorithm in step 2 uses ray tracing technology to generate a 2D virtual projection map to preserve the geometric properties of key points. The specific implementation process includes the following steps: First, a geometric model of cone-beam projection is established in 3D space, where the vertex of the cone beam corresponds to the viewpoint of the virtual projection, and the base of the cone beam is a 2D projection plane; the distance between the projection plane and the vertex of the cone beam is set as the focal length, and the size and resolution of the projection plane are adjusted according to the size of the input 3D image and the target precision; Ray tracing generates projection points by: For the projection viewpoint, a series of rays are emitted towards the projection plane. These rays originate from the cone beam apex and pass through each keypoint in the 3D image space; for each ray, its intersection with a voxel of the 3D image is calculated, and the position of the projection point in the 2D projection plane is determined; to avoid distortion of geometric information or uncertainty in the position of keypoints during projection, ray tracing technology is used to directly simulate the propagation path of rays in the cone beam and is strictly mapped according to the spatial coordinates of keypoints in the 3D image; during projection, the relative positional relationship between the ray direction and the keypoints in 3D space is preserved to ensure that the 2D projection points can accurately reflect the geometric characteristics of the keypoints; Then, based on the intersection point of the ray and the projection plane, a corresponding projection image is generated on the 2D projection plane; the projection image represents the contribution of voxels in the 3D image to the ray in the form of grayscale value or pixel intensity; by aggregating the projection points of each ray, a final 2D virtual projection map is generated, in which each key point is presented as a local region with high density distribution in the projection map.

3. The method according to claim 2, characterized in that, During the projection process, the step size and accuracy of the ray tracing algorithm are adjusted to adapt to 3D images of different resolutions and complexities.

4. The method according to claim 2, characterized in that, By using multi-view projection, key points in a 3D image are projected onto a 2D plane from multiple angles, further enhancing the ability of the projection results to express geometric properties.

5. The method according to claim 1 or 2, characterized in that, The coordinates of the 2D projection points obtained in step 2 are accurately calculated based on the projection matrix and keypoint depth information. The specific process is as follows: Based on the principle of virtual projection, all projection rays related to the target key points form a point set on the 2D plane, and this point set exhibits certain local distribution characteristics. The aggregation point is calculated using the weighted centroid method, and the centroid position (x) is calculated using the following formula. center ,y center ): Among them, x i and y i Let w be the coordinates of the i-th projection point. i The corresponding weights are determined based on the spatial density of the point and the light intensity or gray value of the point.

6. The method according to claim 1 or 2, characterized in that, Step 5, virtual backprojection, accurately calculates the positions of key points in 3D space based on the projection matrix and key point depth information; the specific implementation process includes the following steps: First, based on the geometric parameters of the forward projection, for each projection angle θ i Calculate the virtual ray source S i coordinates (S) x ,S y ): S x =source_origin×sinθ i S y =-source_origin×cosθ i Where source_origin is the preset distance from the ray source to the center of rotation; Then, the corresponding point D on the detector plane is calculated based on the detector center coordinates and the pixel offset of the projection point. i Therefore, the ray equation is constructed as follows: X=S i +t×(D i -S i ) Where t is the position of the scaling parameter control point X on this ray, and X represents any point on the ray; for each key point, its linear equations at N angles are collected to construct a set of virtual orthogonal projection constraint equations: [I-d i ×d i T ]X=[I-d i ×d i T ]S i (i=1,2,...,N) Where d i Let I be the unit vector of the ray direction of the i-th viewpoint, and let I be the identity matrix. The system of equations is transformed into a least squares optimization problem, and the matrix factorization technique SVD is used to solve it to obtain the reconstructed coordinates of the key points in 3D space.

Images