Non-rigid registration method and device based on 3D surface registration and storage medium

Abstract

The invention discloses a non-rigid registration method and device based on 3D surface registration and a storage medium. The method comprises the steps that a 3D image of craniocerebral CT and an intraoperative visible light face image are collected; extracting facial features of the 3D image of the craniocerebral CT by adopting a three-dimensional convolutional neural network, and constructing a CT facial feature point set; extracting facial features of the intraoperative visible light facial image by adopting a three-dimensional convolutional neural network, and constructing a visible light facial feature point set; establishing an initial corresponding relation matrix based on a standard topological structure of the CT facial feature point set and the visible light facial feature point set, and endowing each corresponding relation in the initial corresponding relation matrix with a reliability weight to obtain a face prior corresponding relation; and introducing a face prior corresponding relation as an initial parameter of the t distribution hybrid model, and outputting a registration result after iteratively optimizing the parameter. According to the method, two algorithms are combined for surgical navigation registration, so that a rapid and accurate registration effect is achieved, and the requirements of clinical surgical navigation are better met.

Description

Technical Field

[0001] This invention relates to a navigation and registration method for surgical robots, and more particularly to a non-rigid registration method, device, and storage medium based on 3D surface registration. Background Technology

[0002] Among surgical robots, neurosurgical robots are primarily used for the treatment of central nervous system diseases occurring in the brain and spinal cord. They are minimally invasive surgical devices that assist surgeons in positioning for nearly a hundred diseases, including cerebral hemorrhage, brain tumors, Parkinson's disease, epilepsy, and trigeminal neuralgia. Whether based on augmented reality or mixed reality and neurosurgical robot navigation technology, or utilizing a neurosurgical robot-assisted system, performing cerebral hemorrhage puncture surgery requires accurate image registration and three-dimensional visualization spatial positioning and navigation.

[0003] Current navigation technologies primarily rely on point matching, using markers to fuse image models with real-world space. This process requires multiple CT scans, delaying treatment, increasing costs, and the invasive placement of markers into the skull carries potential risks of scalp and skull hemorrhage and infection. While some studies have proposed surface registration, current surface registration methods suffer from significantly higher target point offsets than point registration, resulting in limited clinical applicability. Therefore, exploring precise, simple, rapid, and non-invasive innovative matching and fusion solutions is urgently needed in clinical practice. No reports have yet documented solutions utilizing multimodal 3D facial feature face alignment combined with non-rigid registration algorithms to address the surface registration challenges in mixed reality and neurosurgical robot navigation. During the surgery, visible light 3D facial imaging was used. Conventional point cloud registration combined with ICP registration algorithms for registration navigation suffers from poor matching accuracy and high time complexity due to its rigid computational nature, resulting in unsatisfactory clinical application results. Other improved registration methods primarily detect facial feature points within the face region and normalize the face based on these feature points, ensuring consistent scale and angle across the face region, facilitating feature extraction and face matching. The ultimate goal of face alignment is to locate the precise shape of the face within a known face bounding box, primarily achieved through optimization methods (convolutional neural networks, CNNs) or regression-based methods.

[0004] Currently, facial feature point output is generally performed using DLIB, an open-source facial feature extraction library that can extract 68 facial feature points (e.g., ...). Figure 1As shown in the figure, HOG and CNN feature extraction functions are used to extract facial features. However, the clinical application of facial DLIB still has two problems. First, facial feature points can be identified under natural lighting, but faces under CT / MRI images cannot be identified well. Second, the existing model is a 2D model, while CT / MRI images and real-world faces are 3D images.

[0005] Feature point registration in medical images is an important research direction in medical image analysis and processing. Currently, commonly used feature point descriptions mainly include point sets, feature curves, and feature surfaces. Point sets can describe arbitrary dimensions and shapes well, but it is difficult to establish correspondences between points. Feature curves and feature surfaces are obtained by fitting point sets. During the fitting process, spline interpolation is used to discard some outliers and irregularities, thereby improving the stability of registration. There are many non-rigid registration algorithms based on feature points. Currently, the most commonly used is the point set registration algorithm based on Gaussian mixture models. However, since the ideal Gaussian mixture model does not consider the influence of outliers and heavy-tailed points, the registration accuracy and robustness are affected. Although the traditional ICP (Iterative Closest Point) algorithm is a classic solution, it has certain limitations in terms of real-time performance, noise robustness, and multi-source data fusion. The NDT (Normal Distributions Transform) algorithm naturally possesses robustness to noise and outliers by dividing the point cloud into voxels and fitting a probability distribution. However, in sparse point clouds or non-rigid deformation scenarios, the traditional Gaussian distribution assumption will lead to registration failure.

[0006] A non-rigid registration algorithm for point sets based on the Student's-t mixture model (SMM) is proposed. This method treats one point set as the centroid set of the SMM and the other point set as the dataset, establishing a mixture probability model for point set registration. The expectation-maximization algorithm is then used to compute the closed-form solution of the non-rigid registration parameters. This method provides accurate and efficient non-rigid registration results and exhibits good robustness to noise and outliers.

[0007] Chinese patent CN113974831A discloses a medical image registration method for surgical navigation. It obtains feature points from facial images through a neural network and registers them with feature points obtained from infrared light. This is a traditional point cloud registration method, which has shortcomings such as heavy computational load, long registration time, and poor matching accuracy. Summary of the Invention

[0008] Objective of this invention: The objective of this invention is to provide a non-rigid registration method based on 3D surface registration, addressing the problems of computational complexity, long registration time, and poor matching accuracy in existing point cloud registration methods. Another objective of this invention is to propose a computer device and storage medium to solve the problem of how to execute a non-rigid registration method based on 3D surface registration.

[0009] Technical solution: The non-rigid registration method based on 3D surface registration described in this invention includes the following steps: Acquire 3D images of the brain from CT scans and intraoperative visible facial images; A three-dimensional convolutional neural network was used to extract facial features from 3D images of cranial CT scans, and a set of facial feature points from CT scans was constructed. A three-dimensional convolutional neural network was used to extract facial features from intraoperative visible light facial images, and a set of visible light facial feature points was constructed. An initial correspondence matrix is ​​established based on the standard topological structure of CT facial feature point set and visible light facial feature point set, and a reliability weight is assigned to each correspondence in the initial correspondence matrix to obtain the face prior correspondence; The prior correspondence of faces is introduced as the initial parameter of the t-distribution mixture model, and the registration result is output after iterative optimization of the parameters.

[0010] This invention replaces point registration with area registration, using a 3D CNN to extract 68 3D feature points of the face from CT and visible light images, establishing a facial feature database, and then... Image feature points are matched and fused using a point set non-rigid multi-mode feature registration algorithm based on a t-distribution hybrid model, thereby achieving an accurate, simple, fast, and non-invasive matching and fusion scheme.

[0011] Preferably, the acquisition of 3D images from cranial CT scans includes: A CT scanner was used to scan the brain to obtain CT images. Based on the CT images, a three-dimensional reconstruction of the head model was performed to obtain a 3D image of the brain CT scan. Intraoperative visible light facial images acquired include: During the surgery, a visible light 3D camera is used to capture the patient's face, acquiring color and depth images. The color and depth images are then registered so that every pixel in the effective area of ​​the color image is incorporated into the depth data, resulting in an intraoperative visible light facial image.

[0012] Preferably, the three-dimensional convolutional neural network includes a 3D data input layer, three layers of 3D convolutional layers and pooling layers, and finally outputs the 3D coordinates of facial key points through a fully connected layer.

[0013] Preferably, assigning a reliability weight to each correspondence in the initial correspondence matrix includes: Group the key points according to function or region to facilitate uniform weight allocation; Based on the application objectives and the classification of key points, assign initial weights to each key point or key point group; After assigning weights, the effect is verified through experiments, and the weights are adjusted according to the verification results until satisfactory performance is achieved. Dynamically adjust the weights of key points.

[0014] Preferably, the introduction of face prior correspondence as initial parameters of the t-distribution mixture model includes: Assume that the target point set Y is generated from the source point set X through a non-rigid transformation T, and is subject to noise interference from the t-distribution. The probability distribution defines the generation probability of the target point yj as a mixture t-distribution, as shown in the following formula: p(y j )=∑ i π ij ·t ν (y j ∣T(x i ),Σ i ,ν) Where, π ij For the mixing ratio based on prior correspondence, t ν Let ν represent the t-distribution, be the degree of freedom parameter, and define the initial corresponding probability matrix p, where p(y j ) represents y j With T(x) i The matching probability of x) i As the source point, Σ i This indicates summing over all parameters containing the index i.

[0015] Furthermore, the iterative optimization parameters include: By combining the prior reliability weights, the matching probability between points xi and yj is calculated, and the posterior probability wij is obtained. Spatial transformation update is performed by estimating the non-rigid transformation T using Bayesian linear regression. Optimize Σi and ν to fit the data distribution; Based on the convergence of the posterior probability wij during iteration, the reliability weight of the face prior correspondence matching is adjusted to weaken the influence of low reliability correspondences.

[0016] The iteration stops when the parameter change of the non-rigid transformation T is less than the threshold or the log-likelihood function converges.

[0017] Preferably, the registration result includes the optimal non-rigid transformation and the final correspondence between the registered source point set and the target point set.

[0018] A second aspect of the present invention discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0019] A third aspect of the present invention provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0020] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) This invention improves the traditional ICP point cloud registration algorithm by replacing the Gaussian distribution in NDT with the Student t distribution, which effectively suppresses the influence of heavy tail noise and other factors.

[0021] (2) This invention combines two algorithms (68-point calibration and t-distribution hybrid model) for surgical navigation registration, achieving fast and accurate registration, better meeting the needs of clinical surgical navigation, effectively shortening the treatment time and cost for patients with cerebral hemorrhage, reducing surgical complications, and has important clinical effects and application value. Attached Figure Description

[0022] Figure 1 An illustration of 68 feature points of a face; Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0024] like Figure 2 As shown, a non-rigid registration method based on 3D surface registration includes the following steps: (1) Acquire 3D images of the brain from CT scans and intraoperative visible facial images, specifically: A 128-slice CT scanner was used to perform a cranial scan to obtain CT images. The image parameters included an exposure of 3 mAs, a slice thickness of 5 mm, and a size of 512×512. Based on PACS, the DICOM format data was imported into the Medical Modeling and Design System software to perform three-dimensional reconstruction of the head model (skin, bones, hematoma, etc.) to obtain a 3D image of the cranial CT scan. During surgery, a visible light 3D camera is used to capture the patient's face, acquiring color and depth images. These images are then registered, ensuring that every pixel in the effective area of ​​the color image incorporates the depth data, resulting in an intraoperative visible light facial image. This invention utilizes RSV-A 3D vision technology to provide users with high frame rate, high resolution depth data streams and color image streams, applicable to object detection, 3D reconstruction, industrial sorting, and other applications.

[0025] (2) 3D Morphable Model (3DMM) is a statistical three-dimensional face modeling technique that generates a deformable general model by analyzing a large amount of face data. Currently, some open-source 3DMM models include BFM (BaselFace Model), FLAME, 3DDFA, etc. This invention is based on 3DDFA (3D Dense Face Alignment) and designs a dedicated 3D CNN network: including an input layer (3D data), three layers of 3D convolutional layers + pooling layers, and finally a fully connected layer (outputting the 3D coordinates of 68 facial key points); The above-mentioned 3D CNN network is used to extract facial features from the 3D image of cranial CT, and while detecting the face bounding box, it outputs 68 key facial feature points to construct a CT facial feature point set. (3) The above-mentioned 3D CNN network is used to extract facial features from the visible light facial images during the operation, and the detected 68 key facial feature points are output while detecting the face bounding box to construct a set of visible light facial feature points. 68 facial key feature points are annotated as follows Figure 1 As shown, each eyebrow has 5 keypoints, sampled evenly from the left to the right edge, totaling 5 × 2 = 10. The eyes have 6 keypoints: the left and right edges, and the upper and lower eyelids, sampled evenly, totaling 6 × 2 = 12. The lips have 20 keypoints, excluding the 2 at the corners of the mouth, divided into the upper and lower lips. The outer edges of the upper and lower lips each have 5 points sampled evenly, and the inner edges each have 3 points sampled evenly, totaling 20. The nose annotation adds 4 keypoints to the bridge of the nose, while the tip of the nose has 5 points sampled evenly, totaling 9 keypoints. The facial contours have 17 keypoints sampled evenly.

[0026] (4) Based on the standard topological structure of the CT facial feature point set and the visible light facial feature point set, an initial correspondence matrix is ​​established, and a reliability weight is assigned to each correspondence in the initial correspondence matrix to obtain the prior correspondence of the face, specifically: Based on the 68-point standard topology of dlib (e.g., the left eye corner is fixed at point 37), an initial correspondence matrix M is established, and a reliability weight is assigned to each match (e.g., key points such as the tip of the nose and the corner of the eye have higher weights). The specific weighting relationship considerations are as follows: In facial recognition, the detection and localization of key points (such as eyes, nose, and mouth) are crucial. 1. Determine the classification or grouping of key points. Group key points according to function or region to facilitate uniform weight allocation. For example: Key structural features, such as the chin and bridge of the nose, define the basic shape of the face.

[0027] Key facial features: such as eyebrows and mouth, used to express emotions.

[0028] Symmetry key points: such as the left and right eyes, left and right eyebrows, need to be constrained by symmetry.

[0029] Key details: such as the contour of the lips, used for fine-tuning.

[0030] 2. Assign initial weights based on importance Assign initial weights to each keypoint or group of keypoints based on the application objectives and keypoint classifications. The following factors may be considered: Discrimination: Key points that play a major role in recognition or matching are given higher weight. For example, eyes and nose have high discrimination and therefore higher weight.

[0031] Dynamics: Key points that change significantly in facial expressions or movements may be more important. For example, the mouth and eyebrows change a lot in facial expressions and therefore carry more weight.

[0032] Symmetry: Symmetrical keypoints may require similar weights. For example, the left and right eyes may have the same weight.

[0033] Stability: Some key points are more stable under different conditions (such as lighting and occlusion). For example, the tip of the nose may be more stable than the lips.

[0034] Example initial weight allocation (assuming a total weight of 1): Eye area (left eye + right eye): 0.3, left eye: 0.15, right eye: 0.15; Eyebrows (left + right): 0.1, left eyebrow: 0.05, right eyebrow: 0.05; Nose: 0.2; Mouth: 0.3; Chin: 0.1; Considering geometric constraints and correlations: The relative positions of certain key points can affect the weights. For example: The distance between the tip of the nose and the center of the eyes is relatively fixed in facial proportions, so the tip of the nose can be given higher weight to maintain the proportions.

[0035] Key points on the upper and lower lips may require higher weighting to maintain the shape of the mouth.

[0036] 3. Verification and iteration: After assigning weights, the effect was verified through experiments: Task performance evaluation: Test the impact of different weights on the final task (such as recognition accuracy, alignment error) on the validation set.

[0037] Sensitivity analysis: Adjust the weight of individual key points and observe performance changes to confirm their sensitivity.

[0038] Iterative optimization: Adjust the weights based on the verification results until satisfactory performance is achieved.

[0039] 4. Considerations for dynamic weights: The weights of key points may also need to be dynamically adjusted depending on the application. For example: Occlusion handling: If a key point is occluded, its weight can be temporarily reduced.

[0040] Facial expression changes: In facial expression recognition, the weights of different regions are adjusted according to the type of facial expression.

[0041] The initial weighting table is as follows (based on 68 facial key points):

[0042] (5) Introduce the prior correspondence of faces as the initial parameters of the t-distribution mixture model. After iterative optimization of the parameters, output the registration result. The point set registration problem is transformed into a probability density estimation problem. The core idea is to regard the target point set Y as generated by the source point set X through a non-rigid transformation T and subject to t-distribution noise interference. By maximizing the likelihood function, the correspondence between the transformation T and the point set is estimated at the same time.

[0043] Specifically: Model assumptions: Assume that the target point set Y is generated from the source point set X through a non-rigid transformation T, and is subject to t-distributed noise interference; (For details, please refer to the following literature: Zhou Zhiyong; Xue Weiqin; Zheng Jian; Kuai Duojie; Zhang Tao; Hu Su. Non-rigid registration algorithm for point sets based on t-distributed hybrid model. Optics and Precision Engineering. 2013, Vol. 21, No. 9, pp. 2405-2420) Introducing prior correspondences as initial parameters for the hybrid model avoids local optima.

[0044] The probability distribution defines the generation probability of the target point yj as a mixture t-distribution, as shown in the following formula: p(y j )=∑ i π ij ·t ν (y j ∣T(x i ),Σ i ,ν) Where, π ij For the mixing ratio based on prior correspondence, t ν Representing the t-distribution, ν is the degree of freedom parameter used to enhance robustness to outliers. An initial corresponding probability matrix p is defined, where p(y) j ) represents y j With T(x) i The matching probability of x) i As the source point, Σ i This indicates summing over all parameters containing the index i.

[0045] Iterative optimization of parameter optimization and registration: (i) Iterative optimization using the Expectation Maximization (EM) algorithm Set initial T and σ 2 ν, initialize the corresponding probability wij (represented by distance or feature similarity); Step E: Combining the prior reliability weights, calculate the matching probability between points xi and yj to obtain the posterior probability wij. The calculation formula is as follows:

[0046] Where k is the index that distinguishes different points, M represents the first M components, P is the mixed weight, and p is the target point y. j The generation probability, t v Let represent the t-distribution, T be the non-rigid transformation, ν be the degrees of freedom parameter, and d be the data dimension. M-step: Estimate the non-rigid transformation T using Bayesian linear regression, resist noise interference, and perform spatial transformation updates, specifically: Update transformation T:

[0047] Update noise variance σ 2 :

[0048] Optimize Σi and ν to fit the data distribution, and update the covariance and degrees of freedom; d is the data dimension.

[0049] Output: Optimal transformation T and corresponding relationship wij.

[0050] (ii) Dynamic adjustment of face prior correspondence Based on the convergence of the posterior probability wij during iteration, the reliability weight of the face prior correspondence matching is adjusted to weaken the influence of low reliability correspondences.

[0051] Termination conditions: The iteration stops when the parameter change of the non-rigid transformation T is less than the threshold or the log-likelihood function converges.

[0052] Output registration results: Optimal non-rigid transformation T; The final correspondence between the registered source point set T(X) and the target point set Y.

[0053] When applying the t-distribution mixture model for registration, due to its probabilistic nature, some corresponding point pairs still exhibit registration errors. To further improve the registration effect, facial prior relationships are incorporated to enhance accuracy and efficiency. Utilizing prior knowledge of faces and combining it with facial feature points, different face images are matched and registered, aligning them spatially. For example, in facial images, there are some known, fixed corresponding points (such as eyes, nose, and mouth). These points have certain relative positional relationships in different images, and the weights of different key points are not entirely the same.

Claims

1. A method of non-rigid registration based on 3D surface registration, characterized in that, The method comprises the following steps: collecting a 3D image of a brain CT and an intraoperative visible light facial image; extracting facial features of the 3D image of the brain CT by using a three-dimensional convolutional neural network, and constructing a CT facial feature point set; extracting facial features of the intraoperative visible light facial image by using a three-dimensional convolutional neural network, and constructing a visible light facial feature point set; establishing an initial correspondence matrix based on a standard topological structure of the CT facial feature point set and the visible light facial feature point set, and assigning a reliability weight to each correspondence in the initial correspondence matrix to obtain a face prior correspondence; introducing the face prior correspondence as an initial parameter of a t-distributed mixture model, and outputting a registration result after iterative optimization of the parameter.

2. The method of claim 1, wherein, The collecting of the 3D image of the brain CT comprises: performing a brain scan by using a CT machine to obtain a CT image, and performing three-dimensional reconstruction of a head model based on the CT image to obtain the 3D image of the brain CT; The collecting of the intraoperative visible light facial image comprises: During the operation, a visible light 3D camera is used to shoot the face of the patient to obtain a color image and a depth image, the color image and the depth image are registered so that each pixel in the effective area of the color image is integrated into the depth data, and the intraoperative visible light facial image is obtained.

3. The method of claim 1, wherein, The three-dimensional convolutional neural network comprises a 3D data input layer, three 3D convolutional layers and pooling layers, and finally outputs 3D coordinates of facial key points through a fully connected layer.

4. The method of claim 1, wherein, The assigning of the reliability weight to each correspondence in the initial correspondence matrix comprises: grouping the key points according to functions or regions to facilitate uniform weight distribution; assigning an initial weight to each key point or key point group according to the application target and the classification of the key points; after the weight is assigned, the effect is verified through experiments, the weight is adjusted according to the verification result until a satisfactory performance is achieved; dynamically adjusting the weight of the key points.

5. The method of claim 1, wherein, The introduction of the face prior correspondence as the initial parameter of the t-distributed mixture model comprises: assuming that a target point set Y is generated from a source point set X through a non-rigid transformation T and is interfered by t-distributed noise; the generation probability of a target point yj is defined as a mixed t-distribution, and the formula is as follows: p(y j )=∑ i πij·t ν (y j ∣T(x i ),Σ i ,ν) where πijis the mixing proportion based on the prior correspondence, t ν represents the t-distribution, v is the degree of freedom parameter, and define the initial correspondence probability matrix p, where p(y j ) represents the matching probability of y j and T(x i ), x i is the source point, and Σ i represents the summation of all parameters with the i subscript.

6. The method of claim 5, wherein, The iterative optimization of the parameter comprises: Combining the prior reliability weight, the matching probability of point x i and y j is calculated to obtain the posterior probability w ij ; estimating the non-rigid transformation T through Bayesian linear regression to update the spatial transformation; Optimization Sigma i and v to fit the data distribution; Based on the posterior probability w in the iteration ij Based on the convergence of the face prior correspondence, adjust the reliability weight of the face prior correspondence matching to weaken the influence of low reliability correspondence. stopping the iteration when the parameter change of the non-rigid transformation T is less than a threshold or the log-likelihood function converges.

7. The method of claim 1, wherein, The registration result comprises an optimal non-rigid transformation and a final correspondence between the source point set and the target point set after registration.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the method in any one of claims 1-7.

9. A computer storage medium, characterized in that The computer storage medium stores instructions, and the instructions make the computer execute the method in any one of claims 1-7 when executed on the computer.

Images