A three-dimensional reconstruction method of craniomaxillofacial region based on free-hand ultrasound

Abstract

A three-dimensional reconstruction method of the craniomaxillofacial region based on free-hand ultrasound is proposed, which includes: (1) real-time volume reconstruction: using a binocular camera to track the position of the ultrasound probe in real time, while adopting a two-stream neural network to segment the ultrasound image and reconstruct the voxel grid in real time; (2) voxel denoising: using morphological post-processing and template matching methods to eliminate outliers, and then converting the voxel model into a point cloud; (3) surface reconstruction: reconstructing the point cloud data into a polygonal mesh through a self-supervised implicit neural representation network to obtain a smooth bone surface.

Description

Technical Field

[0001] The present invention relates to the technical field of computer simulation, and in particular to a craniomaxillofacial three-dimensional reconstruction method based on free-hand ultrasound. Background Art

[0002] In modern medical practice, image-guided surgery (IGS) is a key technology that effectively assists surgeons in understanding the patient's condition by aligning preoperative images such as CT or MRI with the patient's anatomical structure during surgery, and displaying virtual images on a computer screen or projecting them onto the surgical field using augmented reality technology. This greatly improves the accuracy of information. Especially in the field of craniomaxillofacial repair, IGS has broad application prospects and is particularly important for surgeries such as midface reconstruction after trauma. Image registration is a key step in IGS, and the accuracy of registration is crucial to the overall effectiveness of surgical navigation. The acceptable registration error is usually no more than 1 mm.

[0003] Existing registration methods for craniomaxillofacial surgery navigation are mainly divided into two categories: feature marker-based and surface-based. Marker-based registration requires the placement of identifiable artificial markers on the subject's body, which are then associated with preoperative images. Commonly used markers include titanium screws implanted percutaneously into the bone, bite plates fixed to the upper and lower jaws, and small metal balls attached to the skin. In contrast, surface registration uses structured light 3D reconstruction to scan the human body surface exposed in the surgical area, obtain a 3D point cloud, and then perform shape matching using a surface registration algorithm. Summary of the Invention

[0004] All of the above methods have certain limitations. For example, titanium screw implantation is an invasive procedure that requires additional surgery to place markers before CT scanning, and the bone density of the implantation area needs to be carefully assessed to ensure sufficient fixation strength. The occlusal plate, which is a metal ball, is a personalized registration marker, and its production requires additional time and cost, and the process of collecting coordinates during surgery is also very cumbersome. In addition, although facial structured light 3D scanning is non-invasive, it can only scan directly exposed human surface tissue, and the registration accuracy may be reduced due to soft tissue deformation (such as post-traumatic swelling).

[0005] Ultrasound imaging has been widely used in clinical diagnosis, monitoring and analysis due to its portability, non-invasiveness, radiation-free nature, low cost and real-time capabilities. However, its potential in qualitative measurement of human body structure has not yet been explored. Therefore, the present invention aims to provide a non-invasive, accurate and real-time ultrasonic bone surface three-dimensional reconstruction solution for craniomaxillofacial surface reconstruction and registration, thereby improving the registration accuracy in craniomaxillofacial surgery. The method uses a stereo camera to optically track and acquire posture data of a 2D ultrasound image sequence, extracts the skull structure after deep learning segmentation and reconstructs it into a voxel grid, and finally refines it into a smooth polygonal mesh through an implicit neural representation network. This method can not only significantly improve the accuracy of surgical navigation registration, but also effectively solve the problems of invasiveness, low efficiency and accuracy existing in the existing technology.

[0006] (1) Method Overview

[0007] Embodiments of the present invention provide a free-hand 2D ultrasound-based craniofacial surface reconstruction framework to achieve non-invasive and high-precision registration for surgical navigation.

[0008] The method of the present invention mainly includes the following three parts: 1) Real-time volume reconstruction: using a binocular camera to track the position of the ultrasound probe in real time, while adopting a two-stream neural network to segment the ultrasound image and reconstruct the voxel grid in real time; 2) Voxel denoising: using morphological post-processing and template matching methods to eliminate outliers, and then converting the voxel model into a point cloud; 3) Surface reconstruction: finally, the point cloud data is reconstructed into a polygonal mesh through a self-supervised implicit neural representation network to obtain a smooth bone surface.

[0009] (2) Equipment

[0010] The equipment and operation methods used in the embodiments of the present invention are as follows Figure 1 As shown in the figure, the subject wears a head-mounted marker, and the computer software displays the binocular camera image, ultrasound image, and 3D reconstruction image in real time. The operator observes the ultrasound image and the real-time reconstruction process on the computer screen while operating the ultrasound probe to scan the subject's face to complete a real-time coarse reconstruction. De-noising, surface optimization, and fine reconstruction are then performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematic diagram of the equipment structure used in the embodiment of the present invention, wherein (a) is the equipment device and (b) is the operation method.

[0012] Figure 2 Schematic diagram of the STDC network structure in an embodiment of the present invention, where ARM represents the Attention Refine Module.

[0013] Figure 3This is a diagram of the FUNSR network structure in an embodiment of the present invention, where FC represents a fully connected layer.

[0014] Figure 4 Figure 2 shows the reconstruction results in the embodiments of the present invention ((c) and subsequent reconstruction results are obtained by processing the volume point cloud obtained by the morphological post-processing denoising method in (b): (a) real-time volume reconstruction, (b) morphological post-processing denoising, (c) marching cubes method surface reconstruction, (d) FUNSR surface reconstruction, (e) FUNSR+smoothing loss, (f) template matching denoising+FUNSR, (g) template matching denoising+FUNSR+smoothing loss.

[0015] Figure 5 These are diagrams showing the results of the model experiment in an embodiment of the present invention, where (a) is a skull model with implanted magnetic beads and a marker, and (b) is a probe used to measure the position of the magnetic beads.

[0016] Figure 6 Figure 1 shows the reconstruction results of the navigation experiment in an embodiment of the present invention, including (a) bone surface reconstruction results, (b) CBCT skull model and magnetic beads, (c) reconstructed surface and CT alignment, and (d) the position of the magnetic beads (semi-transparent red) in CT and the position of the magnetic beads (green) measured by the probe. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments and accompanying drawings. However, those skilled in the art will appreciate that the present invention is not limited to the accompanying drawings and the following embodiments.

[0018] The method of the present invention mainly includes the following three parts: (1) real-time volume reconstruction: using a binocular camera to track the position of the ultrasound probe in real time, while adopting a two-stream neural network to segment the ultrasound image and reconstruct the voxel grid in real time; (2) voxel denoising: using morphological post-processing and template matching methods to eliminate outliers, and then converting the voxel model into a point cloud; (3) surface reconstruction: finally, using a self-supervised implicit neural representation network to reconstruct the point cloud data into a polygonal mesh to obtain a smooth bone surface. Detailed description is as follows.

[0019] (1) Real-time volume reconstruction includes the following three steps.

[0020] 1.1. Stereoscopic positioning with binocular cameras

[0021] A binocular camera and X-corner markers (referred to as markers) are used to track the spatial position of the ultrasound probe, converting the two-dimensional ultrasound image into a three-dimensional volume. Considering that the subject's head may move during surgery, a head-mounted marker is also designed to bind the coordinate system to the subject's head position to ensure precise spatial positioning of the subject. The X-corner points of the marker are detected in the image captured by the binocular camera. Based on the invariance of the X-corner points in the y-axis and the parallax offset in the x-axis, the spatial position of the marker in the camera coordinate system and the coordinate transformation matrix between the camera coordinate system and the marker coordinate system are calculated.

[0022] For the process of ultrasound 3D reconstruction, it is necessary to determine the position of each ultrasound image in the subject's coordinate system, that is, the coordinate transformation matrix The transformation matrix from the image coordinate system to the ultrasound probe marker coordinate system is obtained by calibration Then, the transformation matrix from the image coordinate system to the subject's coordinate system is obtained through the following coordinate transformation:

[0023]

[0024] 1.2. Deep Learning Real-Time Segmentation

[0025] In order to extract the three-dimensional bone surface, the bone surface must first be extracted from the two-dimensional ultrasound image, so image segmentation is required. In skull structures, such as the zygomatic bone and zygomatic arch, these areas are thinner than the top of the skull (for example, the parietal bone or the frontal bone) and may not be clearly displayed on a portable color Doppler ultrasound device. Traditional segmentation methods such as threshold segmentation cannot accurately separate these bones from the background. Therefore, a deep learning segmentation method is needed. The present invention chooses to use an efficient two-stream neural network, a short-term densely connected network (STDC) to achieve real-time image segmentation.

[0026] The network structure is as follows Figure 2 As shown in the figure, a lightweight network structure is used to achieve real-time segmentation. To further improve segmentation performance, an S-curve algorithm is applied before segmentation to enhance the contrast of the ultrasound image. The image is first segmented into small blocks, and then contrast remapping is performed on each block separately. The pixel value r in the image block is first normalized and converted to the new pixel value r' using the following formula:

[0027]

[0028] Then denormalize from the range of [0, 1] to between the minimum and maximum pixel values ​​of the image block. Among them, α = 0.9642, β = 8.594*10 -4, γ = 0.4962, δ = 0.07598.

[0029] During the acquisition of ultrasound images and posture information, the operator needs to ensure that the markers are captured and recognized by the camera and that the acquired ultrasound images are clear and continuous. To allow the operator to review the scan data in real time, the 3D reconstruction method is divided into two parts: online voxel reconstruction and offline surface reconstruction. The voxel reconstruction stage focuses on real-time performance and allows for correction of deviations through repeated scans of the same area. The surface reconstruction stage focuses on generating a smooth and complete surface that is as close to the actual bone surface as possible.

[0030] 1.3 Voxel Reconstruction

[0031] After obtaining the segmented two-dimensional ultrasound image and the corresponding posture information, the two-dimensional image is reconstructed into three-dimensional voxels in real time. In the voxel reconstruction stage, in order to ensure real-time performance, a downsampled weighted pixel nearest neighbor (PNN) reconstruction algorithm is used. After obtaining the segmentation mask, the image is downsampled by 2 times, and the foreground part is selected as the region of interest (ROI) using a quadrilateral bounding box. Subsequently, each pixel within the ROI is converted into coordinates and assigned to the nearest voxel grid. Since the segmentation results cannot be completely accurate during the scanning and segmentation process, a weighted assignment algorithm is introduced to reduce the noise caused by misclassification of the segmentation algorithm. When resampling at the same spatial position, the voxel value at that position is updated using a weighted algorithm. dImage represents the downsampled image, V represents the voxel value corresponding to pixel (i, j), and W represents the weight value. The voxel value is updated as follows:

[0032]

[0033] At the same time, the weights are updated as follows:

[0034] W←W+1.

[0035] (2) Voxel denoising

[0036] After voxel reconstruction, a two-step post-processing step was used to remove noise from the volumetric model. The first step involved morphological post-processing. First, all voxels were traversed, and the 26 adjacent voxels were checked. If the number of adjacent non-zero voxels was less than 6, the voxel was considered an outlier and eliminated. Connected domain analysis was then performed on the entire voxel model, and domains with small numbers of voxels were eliminated, preserving the zygomatic bone, zygomatic arch, and forehead regions.

[0037] The second step involves template matching denoising. First, the voxel model is converted into a point cloud, using the voxel center as the coordinate. The ICP algorithm is then used to coarsely register this point cloud data with the CT scan. The distance error between each point in the point cloud and its best matching point in the CT scan is calculated. A threshold of 1 mm is selected, and points with a distance error exceeding this value are considered outliers and eliminated.

[0038] (3) Surface reconstruction

[0039] In the voxel reconstruction stage, even if the data has been subjected to a certain degree of denoising, the point cloud obtained after reconstruction will have the problem of uneven density due to the spatial intervals between US images during scanning. If traditional methods such as Marching Cubes are used for surface reconstruction, the obtained surface will be uneven or have holes, which does not conform to the true morphology of the bone surface. The present invention uses a self-supervised implicit neural representation (INR) network, FUNSR (Freehand 3D Ultrasound Neural Surface Reconstruction). By moving the three-dimensional query points sampled around the volumetric point cloud to iteratively learn the Sign Distance Function (SDF), the point cloud obtained in the previous stage is converted into a smooth polygonal mesh.

[0040] The structure of the network is as follows Figure 3 As shown. For the input point cloud, first generate a large number of query points Q within its spatial range. Use the generative adversarial learning model, with an MLP neural network f as the generator, to learn the query point q k The SDF value and gradient in Q. The query point is then projected to its nearest true point t using the predicted SDF value along the gradient or in the opposite direction of the gradient. k ∈P to calculate the self-supervised loss L self In addition, the direction loss L is added scc , preventing query points from oscillating around the zero level set during training. The adversarial discriminator D is then composed of an OSC-ADL module consisting of four fully connected multilayer perceptrons with a leaky ReLU activation function. The SDF values ​​learned from the network are input to this module, which then generates a confidence level for that SDF value. This enables the model to effectively reduce points that are close to the zero level set but do not belong to the surface.

[0041] For the voxel point cloud obtained by ultrasound scanning, there are problems such as low density, high noise, and uneven distribution. The surface obtained by directly reconstructing it using the FUNSR network will be uneven due to overfitting the shape of the point cloud. Therefore, the embodiment of the present invention adjusts the loss function of the network to correct this problem. A new smoothing loss is introduced, and the modulus g of the SDF gradient at a certain point is set i for

[0042]

[0043] Take the gradient of the gradient modulus, that is, the second-order derivative of SDF

[0044]

[0045] Assume the smoothing loss is

[0046]

[0047] This loss function ensures the smoothness of the reconstructed surface by penalizing drastic changes in SDF.

[0048] The loss function of the entire network, that is, the generator loss With the discriminator loss as follows:

[0049]

[0050] Among them, λ self ,λ scc and λ G is the loss weight, D(s) is the confidence value of the SDF prediction learned by D, and G and D are optimized alternately during training. s and s′ represent the predicted signed distance and the value of the zero scalar field (surface constraint), respectively. The size of the zero scalar field value is defined according to the batch size.

[0051] Reconstruction effect and accuracy verification

[0052] The above method was used to perform ultrasound scanning and facial bone surface reconstruction on 10 volunteers. The following surface registration error (SRE) was used as a criterion to measure the degree of match between the reconstruction result and the actual bone surface. The reconstructed surface and the volunteer's skull CBCT model were converted into point clouds P and Q respectively. The formula represents the root mean square error (RMS) of the distance between each point p in P and its best matching point q in Q, in millimeters. Where N P represents the number of points in P.

[0053]

[0054] Table 1 shows the SREs for 20 experiments with 10 volunteers. It can be seen that the error of real-time coarse reconstruction has been reduced to less than 1 mm, and after fine processing, the average error has been successfully reduced to less than 0.5 mm. In contrast, the traditional MarchingCubes method actually increases the error.

[0055] Figure 4 The polygonal mesh of the facial bone surface reconstructed by the present invention proves that the present invention can well extract the shape of the head bone surface.

[0056] Table 1 SRE of the volunteers’ facial scans, in mm

[0057]

[0058] The registration accuracy was verified using a model experiment. X-corner markers were fixed on a 3D-printed skull model, and 16 magnetic beads with a diameter of 2 mm were embedded in the model's maxilla, zygomatic bone, zygomatic arch, brow bone, and frontal bone. The shape and position of the magnetic beads were clearly visible in the CT image. A probe with a 2 mm diameter spherical groove at the tip was used to touch the magnetic beads on the model, and a binocular camera was used to record the position of the magnetic beads in the subject's space. The model was placed in a water tank and scanned with an ultrasound probe to reconstruct its surface contour in real time. After surface reconstruction and registration, the error between the magnetic bead position recorded by the probe and the magnetic bead position in the CT image was calculated to represent the alignment error between the image space and the subject space.

[0059] The following target registration error (TRE) is used to quantify the alignment error between the image space and the subject space:

[0060]

[0061] This formula represents the distance error between each pair of corresponding points p and q in the image space and the subject space, and ||·|| represents the L2 norm, that is, the Euclidean distance. P Indicates the number of points recorded by the probe.

[0062] Figure 6 The reconstruction results of the navigation experiment, the surface registration results, and the visualization of the magnetic bead marker positions are shown. The scanning range of the skull model includes the zygomatic bone, zygomatic arch, eye sockets, and upper jaw, basically covering the positions of the embedded magnetic beads in the model. The position of 15 of the 16 magnetic beads in the subject's space was measured using a probe. Table 2 shows the TRE measured in the navigation experiment. The average TRE for registration using direct voxel sampling of the point cloud was 1.115 mm, and after surface reconstruction, the average TRE was reduced to less than 1 mm. This experiment once again proves that the surface reconstruction method based on deep learning can reduce the impact of noise in real-time reconstruction and reconstruct a surface that is closer to the real bone surface, thereby obtaining a smaller TRE and improving registration accuracy.

[0063] Table 2 TRE of the model experiment, comparing the registration accuracy after real-time coarse reconstruction and surface reconstruction, unit is mm

[0064]

[0065]

Claims

1. A three-dimensional reconstruction method of craniomaxillofacial region based on free-hand ultrasound, characterized in that: The method includes: (1) real-time volume reconstruction: using a binocular camera to track the position of the ultrasound probe in real time, while adopting a two-stream neural network to segment the ultrasound image and reconstruct the voxel grid in real time; (2) voxel denoising: using morphological post-processing and template matching methods to eliminate outliers, and then converting the voxel model into a point cloud; (3) surface reconstruction: reconstructing the point cloud data into a polygonal mesh through a self-supervised implicit neural representation network to obtain a smooth bone surface; Step (1) Real-time volume reconstruction includes the following three steps: 1.

1. Stereoscopic positioning of binocular camera A binocular camera and X-corner markers were used to track the spatial position of the ultrasound probe, converting the two-dimensional ultrasound image into a three-dimensional volume. A head-mounted marker was designed to bind the coordinate system to the subject's head position. The X-corner points of the marker were detected in the image captured by the binocular camera. Based on the invariance of the X-corner points in the y-axis and the parallax offset in the x-axis, the spatial position of the marker in the camera coordinate system and the coordinate transformation matrix between the camera coordinate system and the marker coordinate system were calculated. Determine the position of each ultrasound image in the subject's coordinate system, that is, the coordinate transformation matrix ; Obtain the transformation matrix from the image coordinate system to the ultrasound probe marker coordinate system through calibration , and then obtain the transformation matrix from the image coordinate system to the subject's coordinate system through the following coordinate transformation: 1.

2. Real-time segmentation with deep learning Use a two-stream neural network, Short-Term Dense Concatenate network (STDC), to achieve real-time image segmentation; A lightweight network structure is used to achieve real-time segmentation. An S-curve algorithm is applied before segmentation to enhance the contrast of ultrasound images. The image is first divided into small blocks, and then contrast remapping is performed on each block separately. The pixel value r in the image block is first normalized and then converted to a new pixel value r' using the following formula: ； Then it is denormalized from the range of [0, 1] to between the minimum and maximum pixel values ​​of the image block; where α=0.9642, β=8.594*10 -4 ,γ=0.4962,δ=0.07598; The 3D reconstruction method is divided into two parts: online voxel reconstruction and offline surface reconstruction; 1.3 Voxel Reconstruction After obtaining the segmented two-dimensional ultrasound image and the corresponding posture information, the two-dimensional image is reconstructed into three-dimensional voxels in real time; the downsampled weighted pixel nearest neighbor (PNN) reconstruction algorithm is used in the voxel reconstruction stage; after obtaining the segmentation mask, the image is downsampled by 2 times, and the foreground part is selected as the region of interest (ROI) using a quadrilateral bounding box; then, each pixel within the ROI is converted into coordinates and assigned to the nearest voxel grid; when resampling at the same spatial position, the voxel value at that position is updated using a weighted algorithm; the downsampled image is represented by dImage, represents the voxel value corresponding to pixel (i, j), Represents the weight value, and the voxel value is updated as follows: ； At the same time, the weights are updated as follows: 。 2. The method according to claim 1, wherein: (2) Voxel denoising includes: After voxel reconstruction, a two-step post-processing step was used to remove noise from the volume model. The first step was morphological post-processing, which first traversed all voxels and checked the 26 voxels adjacent to the voxel. If the number of adjacent non-zero voxels was less than 6, the voxel was considered an outlier and eliminated. Then, a connected domain analysis was performed on the entire voxel model, and connected domains with small voxel numbers were eliminated, retaining the zygomatic bone, zygomatic arch, and forehead regions. The second step is template matching denoising. First, the voxel model is converted into a point cloud using the voxel center as the coordinate. Then, the ICP algorithm is used to coarsely align the point cloud data with the CT. The distance error between each point in the point cloud and its best matching point in the CT is obtained. 1 mm is selected as the threshold, and points with distance errors exceeding this value are considered outliers and eliminated.

3. The method according to claim 2, wherein: (3) Surface reconstruction includes: Using self-supervised implicit neural representation (INR network, FUNSR), the signed distance function (SDF) is iteratively learned by moving the 3D query point sampled around the volumetric point cloud, converting the point cloud processed in the previous stage into a smooth polygonal mesh. For the input point cloud, a large number of query points Q are first generated within its spatial range; the generative adversarial learning model is used to generate an MLP neural network. As a generator, learn the query point The SDF value in Q and the gradient; then, the query point uses the predicted SDF value to project to its nearest true point along the gradient or in the opposite direction of the gradient To calculate the self-supervised loss ; In addition, direction loss is added , and then an OSC-ADL module, which is four fully connected multi-layer perceptrons and Leaky ReLU activation functions, is used to form an adversarial discriminator. ;The SDF value learned from the network is input to this module, and then the confidence level of the SDF value is generated; Introduce a new smooth loss, set the modulus of the SDF gradient at a certain point for ； Take the gradient of the gradient modulus, that is, the second-order derivative of SDF ； Assume the smoothing loss is ； This loss function ensures the smoothness of the reconstructed surface by penalizing drastic changes in SDF; The loss function of the entire network, that is, the generator loss With the discriminator loss as follows: ； ； in, , and is the loss weight, is The confidence value of the learned SDF prediction, and Alternate optimization during training; and They represent the predicted signed distance and the value of the zero scalar field, respectively; the size of the zero scalar field value is defined according to the batch size.

Images