A Deep Learning-Based Bidirectional Prediction Method for Soft and Hard Tissue Morphology
By employing a deep learning-based bidirectional prediction method for soft and hard tissue morphology, the problem of inaccurate prediction of postoperative soft tissue changes in existing technologies has been solved. This method enables high-precision postoperative facial prediction and iterative optimization of surgical planning, thereby improving the planning quality of orthognathic surgery.
Patent Information
- Application Number
- CN202611192101.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-07
AI Technical Summary
Existing technologies cannot accurately predict postoperative soft tissue changes in orthognathic surgery, and their high computational complexity and long processing time make it impossible to support iterative optimization, resulting in insufficient accuracy in surgical planning.
A deep learning-based bidirectional prediction method for soft and hard tissue morphology is adopted. Preprocessing of preoperative cone-beam CT image data generates a triangular mesh surface model, bone point cloud data is extracted, and bidirectional prediction and iterative optimization are performed using the bidirectional prediction model for soft and hard tissue morphology to generate predicted soft tissue point cloud data.
It enables high-precision prediction of postoperative facial changes, supports iterative optimization of surgical planning, provides reliable preoperative planning decision support, and improves the quality of surgical planning.
Smart Images

Figure CN122695182B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image data processing technology, and in particular to a method for bidirectional prediction of soft and hard tissue morphology based on deep learning. Background Technology
[0002] In orthognathic surgery, accurately predicting postoperative soft tissue morphological changes is a core challenge in surgical planning. Traditional computer-aided surgical planning (CASS) systems primarily focus on three-dimensional reconstruction of the craniofacial skeletal structure and virtual osteotomy, but cannot accurately predict postoperative soft tissue changes (especially facial appearance). Existing technologies typically rely on biomechanical simulation methods, such as finite element models (FEMs) and mass-spring models (MSMs). These methods are computationally complex, time-consuming, and fail to adequately consider individual differences, making them difficult to apply in clinical practice. Furthermore, existing deep learning methods often neglect the extraction of local spatial features when dealing with complex craniofacial bone-soft tissue morphological mapping relationships, resulting in insufficient prediction accuracy. In addition, unidirectional prediction methods cannot support iterative optimization during the surgical planning process, limiting surgeons' precise control over the final surgical outcome.
[0003] Therefore, in order to address the shortcomings of existing technologies in terms of prediction accuracy, computational efficiency and clinical applicability, a deep learning-based bidirectional prediction method for soft and hard tissue morphology in orthognathic surgery planning is proposed. This method has important clinical value and practical significance for improving the quality of surgical planning and improving patient prognosis. Summary of the Invention
[0004] The technical problem to be solved by this invention is to provide a bidirectional prediction method for soft and hard tissue morphology based on deep learning, which can effectively improve prediction accuracy.
[0005] To address the aforementioned technical problems, this invention provides a deep learning-based bidirectional prediction method for soft and hard tissue morphology, comprising: acquiring preoperative cone-beam computed tomography (CBCT) image data; preprocessing the preoperative CBCT image data and inputting the preprocessed preoperative CBCT image data into a pre-trained deep learning segmentation model for segmentation to generate anatomical structures, and generating a triangular mesh surface model based on the anatomical structures; modifying the triangular mesh surface model to generate a surgical plan craniofacial skeleton model; extracting surgical plan craniofacial skeleton point cloud data based on the surgical plan craniofacial skeleton model; inputting the surgical plan craniofacial skeleton point cloud data into a pre-trained bidirectional prediction model for soft and hard tissue morphology to generate predicted soft tissue point cloud data; comparing the predicted soft tissue point cloud data with the preoperative soft tissue morphology; optimizing the predicted soft tissue point cloud data based on the comparison results, and inputting the optimized predicted soft tissue point cloud data into the bidirectional prediction model for soft and hard tissue morphology to generate adjusted predicted craniofacial skeleton point cloud data.
[0006] As an improvement to the above scheme, the step of preprocessing the preoperative cone-beam CT image data includes: converting the format of the preoperative cone-beam CT image data to generate target CT image data; cropping the background data outside the orthognathic surgery area in the target CT image data to generate baseline CT image data; and performing intensity normalization processing on the baseline CT image data according to a preset cohort-specific quantile normalization strategy.
[0007] As an improvement to the above scheme, the step of generating a triangular mesh surface model based on the anatomical structure includes: removing noise regions from the anatomical structure; and converting the anatomical structure after removing the noise regions into a triangular mesh surface model.
[0008] As an improvement to the above scheme, the step of extracting surgical plan craniofacial bone point cloud data based on the surgical plan craniofacial bone model includes: extracting the outer surface structure of the surgical plan craniofacial bone model; removing noise regions from the outer surface structure to generate an optimized mesh structure; adaptively sampling the optimized mesh structure to construct multiple initial point clouds; spatially downsampling the multiple initial point clouds to construct multiple reference point clouds; sorting the multiple reference point clouds according to a preset sorting rule; and normalizing the sorted reference point clouds to generate surgical plan craniofacial bone point cloud data.
[0009] As an improvement to the above scheme, the step of extracting the outer surface structure of the surgical plan craniofacial skeleton model includes: constructing a sampling scene in the world coordinate system based on the surgical plan craniofacial skeleton model; calculating the reference intersection points of the multi-view sampling rays in the sampling scene with the surgical plan craniofacial skeleton model, and recording the intersection coordinates and triangle face IDs of the reference intersection points; extracting valid intersection points from the reference intersection points; extracting triangle face IDs based on the valid intersection points, and constructing an outer surface sub-mesh from the surgical plan craniofacial skeleton model based on the triangle face IDs; and performing mesh topology optimization on the outer surface sub-mesh to generate the outer surface structure.
[0010] As an improvement to the above scheme, the step of constructing the sampling scene in the world coordinate system based on the surgical plan craniofacial skeleton model includes: constructing the axis-aligned bounding box of the surgical plan craniofacial skeleton model and moving the centroid of the surgical plan craniofacial skeleton model to the origin of the world coordinate system; setting a light source outside the surgical plan craniofacial skeleton model, wherein the position coordinates of the light source are (2d a , 0, 0), where d aThe diagonal length of the axis-aligned bounding box is determined. A reference plane is constructed by extracting the four vertices with the smallest X-coordinates from the eight vertices of the axis-aligned bounding box. This reference plane is then expanded to form a sampling plane, and a regular grid is constructed on the sampling plane. Starting from the position coordinates of the light source and ending at the vertices of the regular grid, a diverging conical light beam is constructed along the negative X-axis. Rotating around the origin and the Y-axis and Z-axis respectively generates a rotation transformation matrix. The conical light beam undergoes coordinate transformation according to the rotation transformation matrix to generate multi-view sampling rays.
[0011] As an improvement to the above scheme, the training steps of the bidirectional prediction model for soft and hard tissue morphology include: collecting a basic dataset, which includes several pairs of facial 3D scan data and cone-beam CT image data; performing initial coarse registration on each pair of facial 3D scan data and cone-beam CT image data to form a registration dataset; performing secondary registration on all registration datasets to unify the coordinate space of the registration datasets; constructing a bidirectional prediction model for soft and hard tissue morphology; and inputting the registration dataset with unified coordinate space into the bidirectional prediction model for soft and hard tissue morphology to train the bidirectional prediction model for soft and hard tissue morphology.
[0012] As an improvement to the above scheme, the initial coarse registration steps for paired facial 3D scan data and cone-beam CT image data include: generating craniofacial bone point cloud data and soft tissue point cloud data based on the cone-beam CT image data; extracting the source marker point set of the facial 3D scan data and the target marker point set of the soft tissue point cloud data; calculating the source center coordinates of the source marker point set and the target center coordinates of the target marker point set; transforming the source center coordinates to a local coordinate system to generate source local coordinates, and transforming the target center coordinates to a local coordinate system to generate target local coordinates; constructing a covariance matrix based on the source local coordinates and the target local coordinates; performing singular value decomposition on the covariance matrix; constructing an initial rotation matrix based on the singular value decomposition results; performing reflection correction on the initial rotation matrix to generate a reference rotation matrix; calculating the translation vector between the source center coordinates and the target center coordinates based on the reference rotation matrix; constructing a rigid transformation matrix based on the reference rotation matrix and the translation vector; and transforming the facial 3D scan data based on the rigid transformation matrix.
[0013] As an improvement to the above scheme, the secondary registration steps include: searching for the nearest target marker in the target marker set for the source marker in the transformed source marker set to construct a registration point pair set; calculating the weighted distance of each registration point pair in the registration point pair set based on geometric distance and normal direction; solving for the optimal registration point pair using the least squares algorithm based on the weighted distance of each registration point pair; optimizing the rigid transformation matrix based on the optimal registration point pair; and performing a secondary transformation on the registration dataset based on the optimized rigid transformation matrix.
[0014] As an improvement to the above scheme, the bidirectional prediction model for soft and hard tissue morphology includes a craniofacial bone-to-soft tissue branch and a soft tissue-to-craniofacial bone branch. Both the craniofacial bone-to-soft tissue branch and the soft tissue-to-craniofacial bone branch include an encoder module and a decoder module. The encoder module includes multiple stacked downsampling units, each of which includes a sampling layer, a grouping layer with an adaptive sphere query radius, a self-attention layer, and a convolutional layer. The decoder module includes upsampling units that correspond one-to-one with the downsampling units and multiple fully connected layers, with residual connections between the downsampling units and their corresponding upsampling units.
[0015] Implementing this invention has the following beneficial effects: This invention provides a deep learning-based bidirectional prediction method for soft and hard tissue morphology that can accurately predict postoperative facial changes and overcome the shortcomings of bidirectional interactive optimization capabilities. It has significant advantages such as high prediction accuracy, excellent visualization, and support for iterative optimization of surgical planning, and can provide clinicians with reliable preoperative planning decision support.
[0016] Furthermore, the deep learning-based bidirectional prediction method for soft and hard tissue morphology of this invention employs an outer surface extraction algorithm based on multi-view light projection. By simulating the interaction between light projected from multiple angles and the surgical plan craniofacial skeleton model, it accurately identifies and extracts the outermost visible surface of the surgical plan craniofacial skeleton model of anatomical structures such as the maxilla, mandible, neural canal, upper teeth, and lower teeth.
[0017] More preferably, the deep learning-based bidirectional prediction method for soft and hard tissue morphology of the present invention employs an improved normal weighted ICP algorithm to simultaneously optimize the consistency between geometric distance and normal direction. Attached Figure Description
[0018] Figure 1 This is a flowchart of an embodiment of the deep learning-based bidirectional prediction method for soft and hard tissue morphology of the present invention; Figure 2 This is a flowchart of the training process for the bidirectional prediction model of soft and hard tissue morphology in this invention. Figure 3 This is a schematic diagram of the network structure for unidirectional prediction in the bidirectional prediction model of soft and hard tissue morphology in this invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It is hereby declared that the directional terms such as up, down, left, right, front, back, inside, and outside used in this text are based solely on the accompanying drawings and are not intended to specifically limit the invention.
[0020] See Figure 1 , Figure 1 The flowchart illustrating an embodiment of the deep learning-based bidirectional prediction method for soft and hard tissue morphology of the present invention is shown, which includes: S101, acquire cone-beam computed tomography (CBCT) image data; Acquire preoperative cone-beam CT images of the patient in DICOM format, ensuring that the images cover the entire craniofacial region, with a slice thickness not exceeding 0.3 mm and a spatial resolution that meets the requirements for three-dimensional reconstruction.
[0021] S102, preprocess the preoperative cone-beam CT image data, and input the preprocessed preoperative cone-beam CT image data into a pre-trained deep learning segmentation model for segmentation to generate anatomical structures, and generate a triangular mesh surface model based on the anatomical structures. In other words, after preprocessing the preoperative cone-beam CT image data, semantic segmentation is performed end-to-end (segmented anatomical structures) using a pre-trained deep learning segmentation model, and then the anatomical structures are generated into a triangular mesh surface model with sub-millimeter precision.
[0022] Among them, the deep learning segmentation model can adopt the nnUNet model, and the anatomical structures include the maxilla, mandible, neural tube, upper teeth, lower teeth, etc.
[0023] Accordingly, the steps for preprocessing preoperative cone-beam CT image data include: (1) Convert the format of the preoperative cone-beam CT image data to generate the target CT image data; Convert the preoperative cone-beam CT image data from DICOM format to NIFTI format.
[0024] (2) Remove background data outside the orthognathic surgery area from the target CT image data to generate baseline CT image data; Based on prior knowledge of maxillofacial anatomy, redundant background data outside the orthognathic surgery area is removed.
[0025] (3) The baseline CT image data is normalized in intensity according to the preset queue-specific quantile standardization strategy.
[0026] To eliminate intensity distribution shifts in preoperative cone-beam CT imaging data caused by differences in scanning parameters and metal artifacts, this invention employs a robust quantile normalization strategy based on a proprietary cohort dataset:
[0027] in: The normalized intensity value; This is the original strength value; ; ; The training set is the global set of voxel intensity distributions.
[0028] In addition, the steps for generating a triangular mesh surface model based on anatomical structure include: (1) Remove noisy areas from the anatomical structure; In practical applications, 3D connected component analysis can be used to remove isolated noise regions smaller than 50 voxels.
[0029] (2) Convert the anatomical structure after removing the noise region into a triangular mesh surface model.
[0030] The Marching Cube algorithm is used to convert a binary mask (i.e., the anatomical structure after removing noisy regions) into a high-quality triangular mesh surface model.
[0031] S103, Modify the triangular mesh surface model to generate a craniofacial skeleton model for surgical planning; Based on a triangular mesh surface model, osteotomy lines and bone segment movements are designed in a virtual surgical planning environment to generate a craniofacial skeletal model that meets both aesthetic and functional requirements for surgical planning.
[0032] S104, Extract the corresponding craniofacial bone point cloud data of the surgical plan based on the craniofacial bone model of the surgical plan; This invention employs an adaptive sampling algorithm to extract uniformly distributed point cloud data of the craniofacial skeleton from a surgical plan model, thereby preserving key anatomical feature points. The specific steps include: (1) Extract the outer surface structure of the craniofacial skeleton model for surgical planning; Furthermore, the steps for extracting the outer surface structure of the craniofacial skeletal model for the surgical plan include: (1.1) Construct a sampling scene in the world coordinate system based on the craniofacial skeleton model of the surgical plan; More preferably, the steps for constructing a sampling scene in a world coordinate system based on a craniofacial skeleton model of the surgical plan include: (1.1.1) Construct the axis-aligned bounding box of the craniofacial skeleton model of the surgical plan to obtain the spatial extent of the craniofacial skeleton model of the surgical plan; and move the centroid of the craniofacial skeleton model of the surgical plan to the origin of the world coordinate system; (1.1.2) Set up a light source outside the craniofacial skeleton model of the surgical plan; The light source is positioned on the positive X-axis with a position coordinate of (2d). a , 0, 0), where d a This is the diagonal length of the axis-aligned bounding box.
[0033] (1.1.3) Extract the four vertices with the smallest X coordinates from the eight vertices of the axis-aligned bounding box to construct a reference plane, expand the reference plane to form a sampling plane, and construct a regular mesh on the sampling plane; (1.1.4) Starting from the position coordinates of the light source and ending at the vertices of the regular grid, construct a diverging cone-shaped ray beam along the negative X-axis: In this embodiment, the reference plane can be uniformly extended outward by 50% along its two-dimensional plane direction to form a sampling plane, and a regular grid with a spacing of 1mm can be constructed on the sampling plane, with each vertex of the regular grid being the endpoint of the light.
[0034] (1.1.5) Rotate around the origin as the center of rotation, and rotate around the Y-axis and Z-axis respectively to generate the rotation transformation matrix; In this embodiment, with the origin as the center of rotation, two sets of discrete rotations are performed sequentially: First, a 360° rotation is completed around the Y-axis with a step size of 30°, generating 12 poses; then, a 360° rotation is completed around the Z-axis with a step size of 30°, generating 12 poses; finally, based on the 24 poses, 24 rotation transformation matrices are generated. .
[0035] (1.1.6) Perform coordinate transformation on the conical light beam according to the rotation transformation matrix to generate multi-view sampling light beam.
[0036] Rotation transformation matrix Used to perform coordinate transformation on the initial conical beam to generate multi-view sampled beams.
[0037] Therefore, a complete light projection scene can be constructed through the above steps (1.1).
[0038] (1.2) Calculate the reference intersection point between the multi-view sampling ray and the craniofacial skeleton model of the surgical plan in the sampling scene, and record the intersection point coordinates and triangle face ID; Using GPU acceleration based on matrices, the system batch calculates the reference intersection points between multi-view sampled rays and the craniofacial skeleton model of the surgical plan, and accurately records the intersection coordinates and corresponding triangle face IDs of the reference intersection points.
[0039] (1.3) Extract valid intersection points from the reference intersection points; By limiting the hit distance and filtering out invalid intersections (missed or background) based on the preset light hit model, only the data of valid intersections are retained; (1.4) Extract the triangular patch index based on the triangular patch ID of the valid intersection point, and construct the outer surface sub-mesh from the craniofacial skeleton model of the surgical plan based on the triangular patch index; (1.5) Perform mesh topology optimization on the outer surface submesh to generate a lightweight outer surface structure.
[0040] Mesh topology optimization includes merging duplicate vertices, removing isolated vertices, eliminating duplicate faces, and culling degenerate triangles (area threshold <). ).
[0041] Therefore, the extraction and optimization of the outer surface structure can be achieved to a limited extent through the above steps (1.4) and (1.5).
[0042] In summary, this invention employs an outer surface extraction algorithm based on multi-view light projection. By simulating the interaction between light projected from multiple angles and the surgical plan craniofacial skeleton model, it accurately identifies and extracts the outermost visible surface of the surgical plan craniofacial skeleton model, including anatomical structures such as the maxilla, mandible, neural canal, upper teeth, and lower teeth.
[0043] (2) Remove noisy regions from the outer surface structure to generate an optimized mesh structure; Connectivity component analysis: Analyze mesh connectivity based on the adjacency relationship of triangular facets, identify and count the number of triangular facets in each connected component.
[0044] Noise suppression: Set a minimum number of facets threshold (default 50) to remove connected components below the threshold, effectively filtering out segmentation noise and preserving the main structure.
[0045] (3) Adaptive sampling of the optimized mesh structure to construct multiple initial point clouds; In this embodiment, curvature-based adaptive uniform sampling is performed on the optimized mesh structure, sampling three non-overlapping high-density initial point cloud subsets of 16384 points, while maintaining the integrity of details in key feature regions.
[0046] (4) Spatial downsampling is performed on multiple initial point clouds to construct multiple reference point clouds; In this embodiment, the farthest point sampling algorithm is applied to each initial point cloud for spatial downsampling, and the farthest point is selected iteratively to generate a uniformly distributed representative point cloud of 8192 points.
[0047] Therefore, a uniform surface point cloud can be generated through the two-step point cloud sampling in steps (3) and (4) above.
[0048] (5) Sort multiple reference point clouds according to preset sorting rules; In this embodiment, the reference point cloud can be rearranged according to the lexicographical order (ascending order) of "ZYX".
[0049] (6) Normalize the sorted baseline point cloud to generate craniofacial bone point cloud data for surgical planning.
[0050] The normalization formula is as follows:
[0051] in: The coordinates of the reference point cloud after normalization; Use the reference point cloud coordinates; The global centroid; The scaling factor is the distance between a point in the baseline point cloud and the global centroid. The maximum distance.
[0052] Therefore, this invention can achieve global consistency normalization through dataset-level statistical parameters, eliminate individual scale differences, and ensure that all samples are distributed within a unit cube in a unified coordinate system.
[0053] S105, Input the craniofacial bone point cloud data of the surgical plan into the pre-trained bidirectional prediction model of soft and hard tissue morphology to generate predicted soft tissue point cloud data. The craniofacial bone point cloud data of the surgical plan is input into a pre-trained bidirectional prediction model of soft and hard tissue morphology. By predicting the displacement of the craniofacial bone point cloud data of the surgical plan, the postoperative soft tissue can be predicted, and the predicted soft tissue point cloud data is obtained.
[0054] S106, compare the predicted soft tissue point cloud data with the preoperative soft tissue morphology; The predicted soft tissue point cloud data is visualized and then compared with the patient's preoperative soft tissue morphology using 3D rendering technology, which makes it easier for clinicians to intuitively evaluate the surgical outcome.
[0055] S107. Optimize the predicted soft tissue point cloud data based on the comparison results, and input the optimized predicted soft tissue point cloud data into the bidirectional prediction model of soft and hard tissue morphology to generate the adjusted predicted craniofacial bone point cloud data.
[0056] Based on the visualization comparison results, clinicians can adjust the soft tissue morphology parameters and re-input the optimized predicted soft tissue point cloud data into the bidirectional prediction model of soft and hard tissue morphology. By predicting the movement of the predicted soft tissue point cloud data, corresponding adjusted predicted craniofacial bone point cloud data is generated, thereby achieving closed-loop iterative optimization of surgical planning.
[0057] Therefore, the deep learning-based bidirectional prediction method for soft and hard tissue morphology of this invention can accurately predict postoperative facial changes, overcome the shortcomings of bidirectional interactive optimization capabilities, and has significant advantages such as high prediction accuracy, good visualization effect, and support for iterative optimization of surgical planning. It can provide reliable preoperative planning decision support for clinicians.
[0058] See Figure 2 , Figure 2 The flowchart of the training process for the bidirectional prediction model of soft and hard tissue morphology in this invention is shown, which includes: S201, Collect basic dataset; The basic dataset includes several pairs of facial 3D scan data and cone-beam CT image data; in this embodiment, 50 pairs of facial 3D scan data and cone-beam CT image data were collected.
[0059] It should be noted that soft tissue point cloud data extracted from cone-beam CT imaging data is subject to distortion and deformation. In practical applications, it is necessary to use "facial 3D scan data" to represent "soft tissue point cloud data" in order to construct accurate training data.
[0060] S202, Initial coarse registration is performed on the paired facial 3D scan data and cone-beam CT image data for each case to form a registration dataset; Because the spatial coordinate systems of the "facial 3D scan data" and the "cone-beam CT image data" are different, the "facial 3D scan data" needs to be registered to the "soft tissue point cloud data" extracted from the "cone-beam CT image data" to achieve spatial alignment between the "facial 3D scan data" and the "cone-beam CT image data"; and each pair of "facial 3D scan data" and "cone-beam CT image data" is registered separately.
[0061] Accordingly, the steps for initial coarse registration of paired facial 3D scan data and cone-beam CT image data to form a registered dataset include: (1) Generate craniofacial bone point cloud data and soft tissue point cloud data based on cone-beam CT image data; Can be adopted Figure 1 The method in steps S102 to S104 of the illustrated embodiment extracts craniofacial bone point cloud data and soft tissue point cloud data from cone-beam CT image data. The specific steps include: preprocessing the cone-beam CT image data in step S102, and inputting the preprocessed cone-beam CT image data into a pre-trained deep learning segmentation model for segmentation to generate anatomical structures; generating a triangular mesh surface model based on the anatomical structures; then generating a craniofacial bone model and a soft tissue model based on the triangular mesh surface model in step S103; finally, extracting craniofacial bone point cloud data from the craniofacial bone model and soft tissue point cloud data from the soft tissue model using the adaptive sampling algorithm in step S104.
[0062] During the initial coarse registration, the paired "facial 3D scan data" and "cone-beam CT image data" were registered using different transformation matrices.
[0063] (2) Extract the source landmark set of facial 3D scan data and the target landmark set of soft tissue point cloud data extracted from cone-beam CT image data; In this embodiment, both the source and target marker sets include six anatomical markers: Glabella (G), Nasion (N), Endocanthion left (En-L), Endocanthion right (En-R), Exocanthion left (Ex-L), and Exocanthion right (Ex-R).
[0064] (3) Calculate the source center coordinates of the source marker set and the target center coordinates of the target marker set; Correspondingly, the source marker set The source center coordinates can be represented as:
[0065] in: The number of source markers in the source marker set; For the first The three-dimensional coordinates of the source marker points.
[0066] Similarly, the target marker set The target center coordinates can be represented as:
[0067] in: The number of target markers in the target marker set; For the first The three-dimensional coordinates of the target marker points.
[0068] (4) Transform the source center coordinates to the local coordinate system to generate the source local coordinates, and transform the target center coordinates to the local coordinate system to generate the target local coordinates; By transforming the set of marker points to local coordinates, the translation component can be effectively eliminated, achieving centering. The specific transformation formula is as follows:
[0069]
[0070] (5) Construct the covariance matrix based on the source local coordinates and the target local coordinates; Construct the 3×3 covariance matrix of the centralized point set:
[0071] (6) Perform singular value decomposition (SVD) on the covariance matrix:
[0072] in, It is a 3×3 orthogonal matrix. It is a diagonal singular value matrix.
[0073] (7) Construct the initial rotation matrix based on the singular value decomposition results:
[0074] (8) Perform reflection correction on the initial rotation matrix to generate the reference rotation matrix; like ,make Recalculate:
[0075] make sure Satisfying rigid body rotation constraints and .
[0076] Therefore, the optimal reference rotation matrix can be solved through the above steps (7) and (8).
[0077] (9) Calculate the translation vector between the source center coordinates and the target center coordinates based on the reference rotation matrix:
[0078] By calculating the translation vector, a precise mapping from the center of the source marker set to the center of the target marker set can be achieved.
[0079] (10) Construct a rigid transformation matrix based on the reference rotation matrix and translation vector.
[0080] By combining the reference rotation matrix and translation vector, a 4×4 rigid transformation matrix is constructed:
[0081] (11) Transform the facial three-dimensional scan data according to the rigid transformation matrix.
[0082] Therefore, the initial coarse registration of facial 3D scan data can be achieved by constructing a 4×4 rigid transformation matrix in step S202 to form a registration dataset, providing an initial solution for subsequent iterative optimization.
[0083] S203, perform secondary registration on all registration datasets to unify the coordinate space of the registration datasets; Because the standards for "cone-beam CT image data" acquired by different devices are different, it is necessary to randomly select one "cone-beam CT image data" from all the "cone-beam CT image data" as the "standard cone-beam CT image data", and register the other "cone-beam CT image data" into the coordinate space of this "standard cone-beam CT image data".
[0084] During secondary registration, different "cone-beam CT image data" are registered using different transformation matrices. That is, each transformation matrix is obtained by calculating the six target markers on each "cone-beam CT image data" and registering them to six target markers randomly selected from the "standard cone-beam CT image data".
[0085] The transformation matrix for the second registration can be obtained by iterative optimization of the rigid transformation matrix. Before iterative optimization, the registration parameters need to be initialized: if the rigid transformation matrix has been constructed in step S202, then the rigid transformation matrix of step S202 is used. If a rigid transformation matrix is not constructed in step S202, then a 4×4 identity matrix is constructed as the initial rigid transformation matrix. This establishes the initial registration state; simultaneously, it sets the maximum number of iterations. Distance threshold Normal weight coefficient Convergence threshold .
[0086] This invention employs an improved Iterative Closest Point (ICP) algorithm to simultaneously optimize the consistency between geometric distance and normal direction. Specifically, the iterative optimization steps for the rigid transformation matrix include: (1) For the source markers in the transformed source marker set, search for the nearest target marker in the target marker set to pair them in order to construct a registration point pair set; In step S202, based on the current rigid transformation matrix... From the formula ( (For the source marker set) ( (This refers to the transformed set of source marker points).
[0087] Search calculation Source markers in China In the target marker set Search for the nearest target marker. Construct the corresponding registration point pair set .
[0088] (2) Calculate the weighted distance of each registration point pair in the registration point pair set based on the geometric distance and normal direction; The corresponding weighted distance function is:
[0089] in: Source marker The unit normal vector; For target markers The unit normal vector; It is the normal weight coefficient.
[0090] (3) Based on the weighted distance of each registration point pair, the optimal registration point pair is solved using the least squares algorithm:
[0091]
[0092] in: The weights for the corresponding registration point pairs; For Gaussian kernel bandwidth, under normal circumstances, .
[0093] Accordingly, when or The iteration terminates when the time is right.
[0094] In addition, the present invention also calculates the fitting index. and root mean square error of interior points To quantify registration accuracy, where:
[0095]
[0096] in, This represents the total number of point clouds in the transformed source marker point set. is the total number of samples of interior points.
[0097] (4) Optimize the rigid transformation matrix based on the best registration point pair.
[0098] The final output is the optimized 4×4 rigid transformation matrix:
[0099] in, For rotational components, This represents the translation component.
[0100] Therefore, the rigid transformation matrix can be optimized through the above iterations to achieve secondary registration.
[0101] Accordingly, the registration dataset is subjected to a secondary transformation based on the optimized rigid transformation matrix.
[0102] Combining steps S202 and S203, the total transformation matrix can be constructed, that is, the total transformation matrix is calculated based on the rigid transformation matrix before optimization and the rigid transformation matrix after optimization:
[0103] Therefore, the present invention can employ a two-step registration algorithm in steps S202 and S203, first registering the facial 3D scan data to its corresponding craniofacial bone point cloud data, and then registering all cases in the registration dataset to a randomly selected case coordinate space.
[0104] S204, Construct a two-way prediction model for soft and hard tissue morphology; The bidirectional prediction model for soft and hard tissue morphology includes branches from craniofacial bones to soft tissue and from soft tissue to craniofacial bones, wherein: The craniofacial bone-to-soft tissue (S2F) branch takes the surgical plan craniofacial bone point cloud data as input, including three-dimensional spatial coordinates and normal unit vectors, and outputs the predicted craniofacial bone point cloud displacement field and the final predicted soft tissue point cloud data. That is: predicted soft tissue point cloud data = surgical plan craniofacial bone point cloud data + craniofacial bone point cloud displacement field.
[0105] Soft tissue to craniofacial skeleton (F2S) branch: Taking the optimized predicted soft tissue point cloud data as input, including 3D spatial coordinates and normal unit vectors, the output is the predicted soft tissue point cloud displacement field, and finally the adjusted predicted craniofacial skeleton point cloud data, that is: adjusted predicted craniofacial skeleton point cloud data = optimized predicted soft tissue point cloud data + soft tissue point cloud displacement field.
[0106] The two branches mentioned above share the same network architecture, but their parameters are independent, thus enabling bidirectional shape transformation.
[0107] Accordingly, both the craniofacial bone-to-soft tissue branch and the soft tissue-to-craniofacial bone branch adopt a U-shaped neural network model, which includes an encoder module and a decoder module.
[0108] like Figure 3 As shown, the encoder module includes multiple stacked downsampling units, each of which includes a sampling layer, a grouping layer with an adaptive ball query radius, a self-attention layer, and a convolutional layer. The decoder module includes upsampling units that correspond one-to-one with the downsampling units and multiple fully connected layers, with residual connections between the downsampling units and their corresponding upsampling units.
[0109] The encoder module and decoder module are described in detail below: I. Encoder Module like Figure 3 As shown, in this embodiment, the encoder module comprises four stacked downsampling units for progressively extracting multi-scale geometric features from the input point cloud; correspondingly, each downsampling unit contains four core sub-layers: Sampling layer: The iterative farthest point sampling algorithm is adopted to select a representative subset of points from the input point set, so as to achieve point cloud downsampling while maintaining the overall geometric distribution. The algorithm ensures that key geometric structures (such as craniofacial contours and skeletal turning points) are not lost by maximizing the minimum distance between sampling points, and provides a spatially uniform set of support points for subsequent feature extraction.
[0110] Grouping layer: For each sampled center point, a ball query method is used to search for neighboring points within a specified radius to construct a local point set, providing local geometric context for subsequent convolution; the ball query radius is dynamically adjusted with network depth (shallow layers r=0.1 capture details, deep layers r=0.4 perceive the global situation), ensuring that geometric features at different scales can be effectively modeled.
[0111] Self-attention layer: On the local point set constructed by the grouping layer, a multi-head self-attention mechanism is introduced. This mechanism dynamically generates an attention weight matrix by calculating the geometric distance and feature similarity between points. This layer projects the features of each point into query, key, and value vectors. By scaling the dot product, it calculates the interdependencies within the local point set, thereby enhancing key feature points and suppressing noise points. The self-attention mechanism effectively captures long-range dependencies in local structures (such as the morphological relationship between the mandible and zygomatic bone), overcoming the limited receptive field of traditional convolutional operations and significantly improving the discriminative power of feature representations.
[0112] Convolutional layer: Based on the features enhanced by the self-attention layer, point convolution operation is performed to adaptively learn the local geometric features of the point cloud; this convolutional layer uses a multilayer perceptron to dynamically generate convolution kernel weights, which can adaptively adjust the feature extraction strategy according to the local geometric shape, complementing the self-attention layer: the former focuses on the accurate modeling of local geometric patterns, while the latter strengthens the global context relationship.
[0113] In practical applications, the first downsampling unit processes the original point cloud coordinates and normal vectors at the input layer, and then passes through a sampling layer (iterative sampling of the farthest point, with 2048 sampling points), a grouping layer (ball query radius r1=0.1, number of neighborhood points k=16), a self-attention layer, and a convolutional layer (64 output channels). The second, third, and fourth downsampling units downsample step by step, with 1024, 512, and 256 sampling points respectively, 0.1, 0.2, and 0.4 ball query radii respectively, 16 neighborhood sampling points for each layer, and 128, 256, and 512 output channels respectively.
[0114] Therefore, the encoder module extracts local geometric features and global structural representations of the point cloud layer by layer through a four-level downsampling structure. The number of point clouds is gradually reduced from 2048 points to 1024 points, 512 points, and 256 points, while the number of feature channels is gradually increased from the initial 6 dimensions (coordinates + normals) to 64, 128, and 256, finally outputting a 512-dimensional global feature vector. This hierarchical feature extraction strategy realizes the mapping from low-dimensional geometric information to high-dimensional semantic features, which not only preserves the detailed features of key anatomical structures of the craniofacial region (such as the root of the nose and the angle of the mandible), but also captures the global representation of the overall morphology, providing a rich multi-scale feature foundation for the high-precision soft and hard tissue mapping of the subsequent decoder module.
[0115] II. Decoder Module like Figure 3 As shown, in this embodiment, the decoder module adopts a four-level upsampling structure, and achieves multi-scale feature fusion through residual connections with the corresponding levels of the encoder module. Specifically: Feature fusion mechanism: At each upsampling level, the interpolated features are concatenated with the features of the corresponding level of the encoder module to enhance the transmission of geometric details; Sampling strategy: Inverse distance weighted interpolation is used to calculate the features of the target point as follows:
[0116]
[0117] in, for of The nearest neighbor set, As weight, It is a numerically stable term.
[0118] Upsampling point convolutional layer: Performs convolution operation on the interpolated point cloud, and simultaneously fuses residual connection features from the corresponding layer of the encoder module to enhance the transmission and fusion of multi-scale geometric information.
[0119] The final decoder outputs a 128-dimensional feature vector. The point cloud displacement field is generated by mapping through three fully connected layers (128->128->64->3). The final predicted point cloud is .
[0120] In other words, the bidirectional prediction model of soft and hard tissue morphology constructed in this invention extracts high-dimensional geometric features through downsampling at each level by the encoder module, and then upsampling at each level by the decoder module, and restores spatial details by combining multi-scale residual connections.
[0121] Therefore, the bidirectional prediction model for soft and hard tissue morphology constructed in this invention overcomes the information loss problem of unidirectional mapping by simultaneously optimizing the mapping relationship between "bone -> soft tissue" and "soft tissue -> bone" through a bidirectional branch design with parameter decoupling; it effectively preserves local geometric details through residual connections between the encoder module and the decoder module, solving the information attenuation problem in the downsampling process of traditional point cloud networks; it enhances the network's ability to model complex curved surface structures of the craniofacial region by adopting an adaptive spherical query radius and a self-attention module; and it maintains the topological structure of the input point cloud by predicting point cloud displacement rather than absolute coordinates, thus improving the anatomical rationality of the prediction results.
[0122] S205. Input the registered dataset after unifying the coordinate space into the bidirectional prediction model of soft and hard tissue morphology to train the bidirectional prediction model of soft and hard tissue morphology.
[0123] Before training the bidirectional prediction model for soft and hard tissue morphology, the training parameters need to be configured, including: Learning rate scheduling: The initial learning rate is set to 1×10. -2It adopts a cosine annealing decay strategy with hot restart.
[0124] Training rounds and batch processing: The total number of training rounds is 500, and the batch size is set to 4. To prevent overfitting, point cloud data augmentation strategies are adopted, including random rotation (±15°), scaling (0.9-1.1 times), and noise injection (standard deviation 0.01).
[0125] Optimizer configuration: Adam optimizer is used. Model Validation and Selection: The training process employs rigorous five-fold cross-validation for model evaluation and parameter selection.
[0126] Early Stopping Mechanism: To improve training efficiency, an early stopping strategy is set up, which means that training is terminated early when the validation set loss does not improve by more than 0.1% for 100 consecutive rounds.
[0127] In addition, the loss term needs to be configured, including: The bidirectional prediction model for soft and hard tissue morphology adopts a staged weighted loss function, and achieves a progressive training strategy from coarse to fine by dynamically adjusting the weight of each loss term.
[0128] Accordingly, the total loss consists of data loss and regularization loss:
[0129] in, For the total loss, For data loss, For regularization loss, These are dynamic weighting coefficients that change with the training rounds.
[0130] Data loss:
[0131] in: Data loss for the craniofacial bone-to-soft-tissue (S2F) branch; For the data loss of the soft tissue to craniofacial bone (F2S) branch; This represents the data loss due to the unidirectional mapping between craniofacial bones and soft tissues. This represents the shape matching loss of the unidirectional mapping between craniofacial bones and soft tissues. This represents the density matching loss of the unidirectional mapping between craniofacial bones and soft tissues; Shape weight; Density weights; The shape matching loss is the average of the nearest distances from each point in the predicted point cloud to the target point cloud. For density matching loss; To predict the nearest neighbors of each point in the point cloud The radius of each point; For each point in the target point cloud, the nearest neighbor is... The radius of each point; For training rounds; This represents the number of point clouds; This represents the total number of training rounds.
[0132] It should be noted that the present invention adopts a dynamic weight adjustment strategy: (1) shape weight With training rounds The weights are increased linearly from 0 to 1.0. In the early stage of training, the shape constraint is started with low weights to avoid training oscillations caused by initial parameter deviations. As training progresses, the weights are gradually increased to enhance the fitting accuracy of the overall craniofacial contour and skeletal morphology. (2) Density weights The first 60% of training sessions focus on overall shape convergence; after 60% of the training sessions, local density constraints are gradually introduced to finely adjust the point cloud distribution of anatomical details and improve the structural fidelity.
[0133] Regularization loss:
[0134]
[0135] in: Used to ensure geometric consistency in the mapping of soft and hard tissues, avoiding "unidirectional deformation"; For the point set Point set The displacement field; For the point set Point set The displacement field.
[0136] Regularization is activated incrementally. Regularization is not enabled in the first 20% of training rounds. Starting from the 20% progress, the weights are linearly increased to 0.1 to avoid parameter oscillations caused by excessive constraints in the early stages of training.
[0137] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for bidirectional prediction of soft and hard tissue morphology based on deep learning, characterized in that, include: Acquire preoperative cone-beam CT imaging data; The preoperative cone-beam CT image data is preprocessed, and the preprocessed preoperative cone-beam CT image data is input into a pre-trained deep learning segmentation model for segmentation to generate anatomical structures. A triangular mesh surface model is generated based on the anatomical structures. The triangular mesh surface model was modified to generate a craniofacial skeleton model for the surgical plan; Extract craniofacial bone point cloud data of the surgical plan from the craniofacial bone model described above; The craniofacial bone point cloud data of the surgical plan is input into a pre-trained bidirectional prediction model of soft and hard tissue morphology to generate predicted soft tissue point cloud data. The predicted soft tissue point cloud data is compared with the preoperative soft tissue morphology; Based on the comparison results, the predicted soft tissue point cloud data is optimized, and the optimized predicted soft tissue point cloud data is input into the bidirectional prediction model of soft and hard tissue morphology to generate the adjusted predicted craniofacial bone point cloud data. The training steps of the bidirectional prediction model for soft and hard tissue morphology include: collecting a basic dataset, which includes several pairs of facial 3D scan data and cone-beam CT image data; performing initial coarse registration on each pair of facial 3D scan data and cone-beam CT image data to form a registration dataset; performing secondary registration on all registration datasets to unify the coordinate space of the registration datasets; constructing the bidirectional prediction model for soft and hard tissue morphology; and inputting the registration dataset with unified coordinate space into the bidirectional prediction model for soft and hard tissue morphology to train the bidirectional prediction model for soft and hard tissue morphology. The steps for initial coarse registration of paired facial 3D scan data and cone-beam CT image data include: generating craniofacial bone point cloud data and soft tissue point cloud data based on the cone-beam CT image data; extracting the source marker point set of the facial 3D scan data and the target marker point set of the soft tissue point cloud data; calculating the source center coordinates of the source marker point set and the target center coordinates of the target marker point set; transforming the source center coordinates to a local coordinate system to generate source local coordinates, and transforming the target center coordinates to a local coordinate system to generate target local coordinates; constructing a covariance matrix based on the source local coordinates and target local coordinates; performing singular value decomposition on the covariance matrix; constructing an initial rotation matrix based on the singular value decomposition results; performing reflection correction on the initial rotation matrix to generate a reference rotation matrix; calculating the translation vector between the source center coordinates and the target center coordinates based on the reference rotation matrix; constructing a rigid transformation matrix based on the reference rotation matrix and the translation vector; and transforming the facial 3D scan data based on the rigid transformation matrix. The steps of the secondary registration include: searching for the nearest target marker in the target marker set to pair the source markers in the transformed source marker set to construct a registration point pair set; calculating the weighted distance of each registration point pair in the registration point pair set based on geometric distance and normal direction; solving for the optimal registration point pair using the least squares algorithm based on the weighted distance of each registration point pair; optimizing the rigid transformation matrix based on the optimal registration point pair; and performing a secondary transformation on the registration dataset based on the optimized rigid transformation matrix.
2. The method for bidirectional prediction of soft and hard tissue morphology based on deep learning as described in claim 1, characterized in that, The preprocessing steps for the preoperative cone-beam CT image data include: The preoperative cone-beam CT image data is converted to a new format to generate the target CT image data. Background data outside the orthognathic surgery area in the target CT image data is cropped out to generate baseline CT image data; The baseline CT image data are normalized in intensity according to a pre-defined queue-specific quantile normalization strategy.
3. The method for bidirectional prediction of soft and hard tissue morphology based on deep learning as described in claim 1, characterized in that, The step of generating a triangular mesh surface model based on the anatomical structure includes: Remove noisy areas from the anatomical structure; The anatomical structure after removing the noise region is converted into a triangular mesh surface model.
4. The method for bidirectional prediction of soft and hard tissue morphology based on deep learning as described in claim 1, characterized in that, The step of extracting craniofacial bone point cloud data of the surgical plan based on the craniofacial bone model of the surgical plan includes: Extract the outer surface structure of the craniofacial skeleton model of the surgical plan; Noise regions in the outer surface structure are removed to generate an optimized mesh structure; The optimized mesh structure is adaptively sampled to construct multiple initial point clouds; Multiple initial point clouds are spatially downsampled to construct multiple reference point clouds; The multiple reference point clouds are sorted according to a preset sorting rule; The sorted baseline point cloud is normalized to generate craniofacial skeletal point cloud data for surgical planning.
5. The deep learning-based bidirectional prediction method for soft and hard tissue morphology as described in claim 4, characterized in that, The step of extracting the outer surface structure of the craniofacial skeleton model of the surgical plan includes: A sampling scene is constructed in the world coordinate system based on the craniofacial skeleton model of the surgical plan; Calculate the reference intersection point between the multi-view sampling rays in the sampling scene and the craniofacial skeleton model of the surgical plan, and record the intersection point coordinates and triangle face ID of the reference intersection point; Extract valid intersection points from the reference intersection points; The triangular facet index is extracted based on the triangular facet ID of the effective intersection point, and the outer surface sub-mesh is constructed from the craniofacial skeleton model of the surgical plan based on the triangular facet index. The outer surface sub-mesh is topologically optimized to generate the outer surface structure.
6. The method for bidirectional prediction of soft and hard tissue morphology based on deep learning as described in claim 5, characterized in that, The steps for constructing the sampling scene in the world coordinate system based on the craniofacial skeleton model of the surgical plan include: Construct the axis-aligned bounding box of the craniofacial skeleton model of the surgical plan, and move the centroid of the craniofacial skeleton model of the surgical plan to the origin of the world coordinate system; A light source is positioned outside the craniofacial skeleton model of the surgical plan, and the position coordinates of the light source are (2d). a , 0, 0), where d a This is the diagonal length of the axis-aligned bounding box; Extract the four vertices with the smallest X coordinates from the eight vertices of the axis-aligned bounding box to construct a reference plane, expand the reference plane to form a sampling plane, and construct a regular mesh on the sampling plane; Starting from the position coordinates of the light source and ending at the vertices of the regular grid, a diverging cone-shaped light beam is constructed along the negative X-axis: Rotate around the origin as the center of rotation, and rotate around the Y-axis and Z-axis respectively to generate a rotation transformation matrix; The cone-shaped light beam is subjected to coordinate transformation according to the rotation transformation matrix to generate multi-view sampling light.
7. The method for bidirectional prediction of soft and hard tissue morphology based on deep learning as described in claim 1, characterized in that, The bidirectional prediction model for soft and hard tissue morphology includes a craniofacial bone-to-soft tissue branch and a soft tissue-to-craniofacial bone branch, both of which include an encoder module and a decoder module. The encoder module includes multiple stacked downsampling units, each of which includes a sampling layer, a grouping layer with an adaptive ball query radius, a self-attention layer, and a convolutional layer. The decoder module includes upsampling units that correspond one-to-one with the downsampling units and multiple fully connected layers, with residual connections between the downsampling units and their corresponding upsampling units.
Citation Information
Patent Citations
Plastic surgery postoperation physical appearance prediction method based on machine learning
CN113052864A
Bone surface bidirectional three-dimensional reconstruction and editing method and device, computer equipment and medium
CN121458911A
Oral denture digital design method based on AI intelligence
CN122490836A
