Real-time pose update method based on feature prediction and point cloud registration

Abstract

The application discloses a real-time pose updating method based on feature prediction and point cloud registration, and belongs to the technical field of image registration. The method comprises the following steps: extracting a bone surface point cloud covering at least a feature region and a neighboring region thereof from a three-dimensional model of a patient's bone tissue as a target point cloud for registration with a real-time ultrasound image; acquiring an ultrasound image collected by an ultrasound patch in real time to obtain a two-dimensional ultrasound image with a spatial position; using a pre-trained artificial intelligence model to predict the two-dimensional ultrasound image to obtain a bone-soft tissue interface feature, and extracting a plurality of points on the bone-soft tissue interface from the bone-soft tissue interface feature to form a source point cloud for registration; and registering the source point cloud to the target point cloud to realize real-time updating of a pose. The application can realize intraoperative pose monitoring through a real-time image stream under noninvasive conditions.

Description

Technical Field

[0001] This invention relates to the field of image registration technology, and in particular to a real-time pose update method based on feature prediction and point cloud registration. Background Technology

[0002] In surgical navigation systems, image registration technology is a necessary step to achieve precise navigation and positioning. Its core lies in establishing the spatial correspondence between the patient's intraoperative anatomical structures and preoperative images (such as CT and MRI), that is, determining the coordinate transformation between the two. High-precision and high-efficiency registration is a prerequisite for the surgical navigation system to accurately locate the target area and achieve precise surgical operations, directly affecting the safety and outcome of the surgery.

[0003] Traditional image registration in surgical navigation systems is a one-time process, and subsequent real-time pose updates often rely on optical or magnetic tracking. These methods typically require rigid fixation of the reference frame to the bone tissue, introducing additional trauma and limiting their clinical application. Utilizing real-time image streams available intraoperatively (such as continuous ultrasound sequences or X-ray fluoroscopy) for intraoperative pose monitoring and updates theoretically offers a non-invasive alternative. However, current technologies face significant challenges in using real-time image streams for surgical navigation: firstly, continuous image stream processing demands high real-time performance and stability. For example, automatically processing ultrasound image streams using edge detection algorithms cannot guarantee the extraction of correct image features; secondly, the features in continuous image streams are far less abundant than those in CT scans, making it difficult to meet the accuracy requirements of pose updates. These two factors lead to delayed pose updates, insufficient accuracy, or continuous error accumulation over time, failing to meet the stringent standards of surgical navigation.

[0004] In summary, there is an urgent need to develop a novel pose update method that can achieve intraoperative pose monitoring through real-time image streaming under non-invasive conditions. This requires not only innovative image acquisition methods to provide richer spatial information, but also the use of feature prediction and pose update methods that meet real-time requirements, thereby improving computational speed and robustness. Only by simultaneously addressing these two core challenges can the bottlenecks of existing technologies in terms of processing speed, accuracy, and real-time performance be effectively overcome, providing reliable core technological support for the next generation of non-invasive surgical navigation systems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a real-time pose update method based on feature prediction and point cloud registration that enables intraoperative pose monitoring through real-time image streams under non-invasive conditions.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A real-time pose update method based on feature prediction and point cloud registration is used in a surgical navigation system. The surgical navigation system includes an optical positioning system and an ultrasonic patch. The ultrasonic patch is applied to a characteristic region on the patient's skin surface, the characteristic region being an area with anatomical features. The ultrasonic patch has optical positioning markers that can be captured by the optical positioning system. The method is characterized by comprising:

[0008] Step S101: Extract bone surface point cloud covering at least the feature region and its adjacent region from the three-dimensional model of the patient's bone tissue as the target point cloud for registration with real-time ultrasound images, wherein the three-dimensional model is obtained by three-dimensional reconstruction using three-dimensional medical image data of the patient's bone tissue;

[0009] Step S102: Acquire the ultrasound image acquired in real time by the ultrasound patch to obtain a two-dimensional ultrasound image with spatial location;

[0010] Step S103: Use a pre-trained artificial intelligence model to predict the two-dimensional ultrasound image, obtain the bone-soft tissue interface features, and extract several points on the bone-soft tissue interface to form a source point cloud for registration.

[0011] Step S104: Register the source point cloud to the target point cloud to achieve real-time pose update.

[0012] The present invention has the following beneficial effects:

[0013] 1. Improved positioning accuracy: By using two-dimensional ultrasound images (i.e., ultrasound images in two or more different planes) to obtain more comprehensive spatial information, it helps to build a more accurate three-dimensional spatial model. Compared with the traditional single-plane imaging method, it can provide more accurate position and orientation information, thereby better realizing non-invasive surgical navigation.

[0014] 2. Enhanced Robustness: Utilizing artificial intelligence models to predict image features enables effective target identification and tracking in complex environments, reducing errors caused by noise, occlusion, and other factors. The AI ​​models can also learn feature representations for different scenarios, giving the system better adaptability and stability in various environments.

[0015] 3. Optimize real-time performance: By performing necessary post-processing on the prediction results of the artificial intelligence model, key features in the image can be quickly predicted, which can further improve the quality and reliability of the data, while ensuring the real-time performance of the system.

[0016] 4. Achieve efficient registration: The point cloud registration algorithm is used to complete real-time pose updates, which can quickly and accurately match the newly acquired data with the existing data, thereby achieving efficient pose tracking and meeting the real-time needs during surgery. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the usage state of the surgical navigation system in this invention;

[0018] Figure 2 for Figure 1 Schematic diagram of the structure of the ultrasound patch;

[0019] Figure 3 for Figure 2 A schematic diagram of an explosion structure in one direction, with additional illustration of the dressing;

[0020] Figure 4 for Figure 2 A schematic diagram of the explosion structure from another direction, with additional illustration of the dressing;

[0021] Figure 5 This is a flowchart illustrating the real-time pose update method based on feature prediction and point cloud registration of the present invention.

[0022] Figure 6 This is a schematic diagram of the real-time pose update method based on feature prediction and point cloud registration of the present invention.

[0023] Figure 7 This is a diagram showing the processing effect of steps A1-A2 in this invention;

[0024] Figure 8 This is a diagram showing the registration result of the ultrasound image using a dual-plane image in this invention. Detailed Implementation

[0025] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0026] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0027] In the prior art, patent application CN117503347A discloses an ultrasound-assisted registration system and method for orthopedic surgical navigation, which uses a real CT simulated ultrasound image (Fake US) and matches it with a real-time acquired ultrasound image (Real US). However, the generation of Fake US, whether using artificial intelligence methods or acoustic simulation methods, does not meet the real-time requirements. Patent application CN107736897A discloses an ultrasound registration and long bone repositioning device and method based on a six-degree-of-freedom parallel platform, which uses an edge detection method. However, ultrasound images have complex textures, poor standardization, and large individual differences. Because the edge detection method lacks reasoning ability, it easily extracts muscle textures, muscle-ligament interfaces, etc., but does not correctly converge to the features of the bone-soft tissue interface.

[0028] To address the shortcomings of existing technologies, this application proposes a pose update method for surgical navigation. Specifically, it introduces an artificial intelligence model to predict features from a real-time ultrasound image stream containing positioning markers, generating a point cloud of the bone-soft tissue interface. Then, an ICP-type registration algorithm, typically used for single-stage registration, is applied, compressing the registration computation time to real-time levels given a reasonable initial pose. This allows for the registration of the real-time acquired source point cloud with the target point cloud constructed from preoperative images, achieving dynamic real-time pose updates. This method enables precise surgical navigation under non-invasive conditions.

[0029] This invention provides a real-time pose update method based on feature prediction and point cloud registration for use in a surgical navigation system. The surgical navigation system includes an optical positioning system and an ultrasonic patch 1, as shown below. Figure 1-4 As shown, the ultrasonic patch 1 is used to be applied to a characteristic area on the patient's skin surface. The characteristic area is an area with anatomical features. The ultrasonic patch 1 is provided with an optical positioning mark 121 that can be captured by the optical positioning system.

[0030] The ultrasonic patch 1 can be constructed using existing technology, and its structure is briefly described below:

[0031] like Figure 2-4 As shown, the ultrasonic patch 1 of the present invention consists of three main parts: a visual positioning device, an ultrasonic device 14, and a coupling device. The visual positioning device includes a beacon base 11 and a beacon housing 12 sealed and fastened to the beacon base 11. The surface of the beacon housing 12 is provided with optical positioning marks 121. The ultrasonic device 14 is located inside the beacon housing 12 and is disposed on the beacon base 11.

[0032] When using, such as Figure 1As shown, the ultrasound patch 1 is fixed to the patient's skin surface by a coupling device. The visual positioning device and the ultrasound device 14 work together. Specifically, the ultrasound device 14 on the beacon base 11 is used to obtain the position information of the interface between hard tissue (such as femur, tibia, etc.) and soft tissue. The visual positioning device is used to cooperate with the visual positioning camera 2 to obtain the position information of the ultrasound patch 1. That is, the navigation system can perceive the position of the bone-soft tissue interface in the world coordinate system through the optical positioning mark 121 on the surface of the beacon shell 12 on the beacon base 11.

[0033] The optical positioning mark 121 can be any type of mark in the art, such as an ARUCO mark or an April tag. In the figure, reference numeral 123 is the optical calibration plate that carries the optical positioning mark 121, reference numeral 122 is the slot on the beacon housing 12 for mounting the optical calibration plate 123, reference numeral 124 is the notch on one side edge of the optical calibration plate 123 for engaging with a prying tool, and reference numeral 111 is the receiving groove 111 in the middle of the beacon base 11 for accommodating the ultrasonic device 14 and the bottom edge of the beacon housing 12.

[0034] The ultrasound device 14 includes an ultrasound transducer 141, an (built-in) ultrasound chip 142 electrically connected to the ultrasound transducer 141, and a power supply and transmission device 143 electrically connected to the ultrasound transducer 141 and the ultrasound chip 142. The ultrasound transducer 141 is the core component of the ultrasound patch 1, used to transmit ultrasound signals and receive echoes. The ultrasound chip 142 is the core computing and control unit, including at least an ultrasound signal processing chip, which is responsible for converting the received analog electrical signals from the ultrasound transducer 141 into digital signals and transmitting them to the surgical navigation system by the transmission device. The power supply and transmission device 143 is responsible for supplying power to the ultrasound device 14 and transmitting data between the surgical navigation system and the ultrasound device 14.

[0035] Ultrasonic patch 1 is used to obtain the location of the interface between hard tissue (such as the femur, tibia, etc.) and soft tissue from ultrasound signals or images. Through optical positioning markers on its surface, it enables the navigation system to perceive the position of the bone-soft tissue interface in the world coordinate system. The working principle of ultrasonic patch 1 is that ultrasound waves can penetrate soft tissue, but are strongly reflected when encountering hard tissue (such as bone). Therefore, a distinct high-signal shadow is presented at the interface between hard and soft tissue. Artificial intelligence models used for image segmentation tasks can predict the echo location of the hard-soft tissue interface, thereby accurately identifying the hard tissue surface beneath the ultrasound.

[0036] The coupling device is used to fix the ultrasound patch 1 to the patient's skin surface, including a gel fixation patch 131 and / or a dressing 132. The gel fixation patch 131 is disposed at the bottom of the beacon base 11, and the dressing 132 has an opening in the middle for the beacon housing 12 to pass through. The dressing 132 is pressed against the periphery of the beacon base 11, and the beacon base 11 can be effectively fixed by the dressing 132 on the periphery.

[0037] In existing technologies, registration in optical surgical navigation systems is completed in one step, and pose updates only require monitoring the position of the reference frame. However, because this invention innovates the image acquisition method, namely, the surgical navigation system adopts a new ultrasonic device (in the form of an ultrasonic patch with optical positioning markers), which is attached to the patient's skin surface and has no rigid connection with the bone, unlike existing technologies, each pose update in this invention requires iterative registration.

[0038] To solve this iterative registration problem, such as Figure 5-6 As shown, the method described in this invention includes:

[0039] Step S101: Extract bone surface point cloud covering at least the feature region and its adjacent region from the three-dimensional model of the patient's bone tissue as the target point cloud for registration with real-time ultrasound images, wherein the three-dimensional model is obtained by three-dimensional reconstruction using three-dimensional medical image data of the patient's bone tissue;

[0040] This step corresponds to Figure 6 The left frame in the image shows the image data. Before surgery, three-dimensional medical image data, including but not limited to CT, MRI, and spatial reconstruction results formed by multiple X-rays, can be acquired. Then, three-dimensional reconstruction of the patient's image data can be performed to obtain a three-dimensional model of the patient's bone tissue.

[0041] Bone surface point clouds covering at least the feature region and its adjacent region are extracted from the three-dimensional model and used as target point clouds for registration with real-time ultrasound images. In this way, by spatially constraining the extraction range and avoiding the inclusion of a large amount of irrelevant bone structure information, the computational cost and number of iterations for registration can be significantly reduced, ensuring that the registration process can be completed within an acceptable timeframe.

[0042] Step S102: Acquire the ultrasound image acquired in real time by the ultrasound patch to obtain a two-dimensional ultrasound image with spatial location;

[0043] In this invention, one or more regions with anatomical features are selected as feature regions from each anatomical site requiring surgical navigation. Preferably, the feature regions possess two characteristics: 1) the skin surface is relatively flat, making it easy for the ultrasound patch to adhere to the skin surface and facilitating stable coupling when acquiring ultrasound images during surgery; 2) there is bone tissue beneath the skin at the ultrasound image acquisition location, and the bone surface has relatively unique anatomical features, which can serve as a basis for feature matching in subsequent registration.

[0044] In this step, ultrasound imaging equipment (including but not limited to A-mode or B-mode ultrasound imaging) is used to acquire images from multiple angles and directions in the feature area, obtaining a set of two-dimensional ultrasound images or ultrasound shape data for subsequent feature extraction and pose estimation. The spatial position information of the two-dimensional ultrasound images is generated by the optical positioning system capturing optical positioning markers. This step corresponds to... Figure 6 The upper middle frame in the middle.

[0045] In the specific implementation of this step, during the surgical procedure, in order to update the coordinate transformation from the bone model coordinate system to the camera coordinate system in real time... From the current time step To a new time step At that time, a set of ultrasound images with spatial location is acquired, the ultrasound images being composed of B-mode ultrasound images in at least two planes within the characteristic region.

[0046] Step S103: Use a pre-trained artificial intelligence model to predict the two-dimensional ultrasound image, obtain the bone-soft tissue interface features, and extract several points on the bone-soft tissue interface to form a source point cloud for registration.

[0047] This step corresponds to Figure 6 The right-hand frame in the image transmits the two-dimensional ultrasound image obtained in step S102 to the trained artificial intelligence model in step S103 (i.e., Figure 6 A neural network model is used to predict bone-soft tissue interface features. After obtaining the bone-soft tissue interface prediction results, a series of post-processing steps are performed on the prediction mask to extract several points on the bone-soft tissue interface, forming a source point cloud for registration. This application does not limit the post-processing method for extracting the source point cloud; it can employ various methods readily conceived by those skilled in the art. A specific implementation method will be given below. The artificial intelligence model will be introduced first.

[0048] The propagation characteristics of ultrasound waves differ significantly across different tissue interfaces. Specifically, when ultrasound waves pass through soft tissues (such as fat, muscle, ligaments, and skin), the sound waves can propagate continuously with minimal energy loss because these tissues have similar acoustic impedances. However, because bone tissue has a much higher acoustic impedance than soft tissues, ultrasound waves are strongly reflected when they reach the bone surface. Therefore, the reflected signal from the bone surface appears as a bright, well-defined area of ​​strong echo in ultrasound images, which can be used to identify the location of the bone-soft tissue interface.

[0049] Therefore, in the ultrasound image described in step S102 above, the features of the bone-soft tissue interface can be clearly labeled. By training any artificial intelligence model applicable to image segmentation tasks, the bone-soft tissue interface features in the ultrasound image can be predicted.

[0050] Methods for labeling bone-soft tissue interface features may specifically include:

[0051] Labeling ultrasound images: In each ultrasound image, the boundary region of the bone-soft tissue interface is labeled, and a corresponding segmentation mask is generated to be used as training labels for the model.

[0052] The artificial intelligence model described herein, which can be used for image segmentation tasks, preferably employs a neural network model with end-to-end real-time segmentation capabilities, as this application requires the model to have high inference speed and real-time processing capabilities. This model takes a two-dimensional ultrasound image as input and can output a segmentation mask of target features in the image, making it suitable for real-time identification and extraction of structural features such as the bone-soft tissue interface.

[0053] As an optional embodiment, the artificial intelligence model includes:

[0054] The feature extraction module is used to receive the input image and extract multi-scale semantic feature information. It is constructed using a convolutional neural network (CNN) or a structure based on a self-attention mechanism.

[0055] The feature fusion module is used to fuse feature information from different scales or spatial locations, thereby enhancing the model's reasoning ability for complex structures and fuzzy boundaries. It is implemented using Feature Pyramid (FPN) or multi-layer Transformer encoder-decoder network.

[0056] The output prediction module is used to generate the location, category and corresponding segmentation mask of the feature region in the image after extracting and fusing features. It can typically include an object detection branch or a mask generation branch.

[0057] In a preferred embodiment, the artificial intelligence model is the YOLO-Seg model, specifically including:

[0058] The feature extraction module employs a lightweight convolutional neural network structure, preferably CSP-Darknet, to extract multi-scale features from the input two-dimensional ultrasound image.

[0059] The feature fusion module uses a combination of FPN and PAN to fuse feature maps of different scales from bottom to top and from top to bottom, enhancing the model's ability to perceive the bone-soft tissue boundary.

[0060] The output prediction module includes an object detection branch and a mask generation branch. The object detection branch outputs the target location and its category label, while the mask generation branch segments each target instance by generating a set of mask prototypes and corresponding mask coefficients, ultimately outputting an instance segmentation mask with appropriate resolution, featuring high inference speed and real-time processing capabilities.

[0061] In another preferred embodiment, the artificial intelligence model is a DET-R segmentation model, specifically including:

[0062] The feature extraction module uses residual networks (such as ResNet50) or Transformer structures to obtain deep semantic features of the input image;

[0063] The feature fusion module adopts a Transformer encoder-decoder structure based on cross-scale feature fusion. The encoder models the global context information of the image, and the decoder focuses on the potential target region through learnable query vectors to achieve spatial localization.

[0064] The output prediction module employs a region attention mechanism to generate segmentation results. It first extracts the contextual features of each candidate region, then performs pixel-level mask regression within the target region, outputting high-precision instance segmentation results. This model is suitable for medical image scenes with complex structures or blurred boundaries, combining segmentation accuracy with modeling robustness.

[0065] Step S104: Register the source point cloud to the target point cloud to achieve real-time pose update.

[0066] This step corresponds to Figure 6 The middle lower part of the related steps. As an optional embodiment, step S104 includes:

[0067] Step S1041: Based on the current time step The coordinate transformation matrix and the previous time step With the current time step Calculate the new time step based on the motion parameters between them. The initial iteration position;

[0068] Since the pose and motion parameters (velocity, acceleration, angular velocity, angular acceleration) from the previous time step can be obtained during pose update, and the pose changes within a single time step are limited, the default position should obviously not be used as the initial iteration position for re-registration. Instead, the position should be based on the current time step. The coordinate transformation matrix, and the previous time step With the current time step The motion parameters between the points are used to calculate a new initial iteration position (Initial Guess). This method ensures that the initial iteration position is very close to the true position, saving registration calculation time and improving accuracy.

[0069] Specifically, the new time step The coordinate transformation matrix corresponding to the initial iteration position is: The rotation matrix and translation vector are respectively:

[0070] ;

[0071] ;

[0072] The vector representation of motion parameters is: velocity acceleration angular velocity angular acceleration Exp and Log represent the exponential and diagonal mappings, respectively. The above formula represents the coordinate transformation matrix from the bone coordinate system to the ultrasound coordinate system. It means that, assuming the bone and skin continue to coordinate according to... The velocity and acceleration at any given moment are relative to the motion. Afterwards, The coordinate transformation matrix at time step.

[0073] As a simplified implementation method, when the relative motion between the skin and the tissue surface is not intense, the acceleration and angular acceleration terms can be omitted according to the actual engineering situation.

[0074] To further simplify, the velocity and angular velocity terms can be discarded, that is, the coordinate transformation matrix of the previous time step can be directly used as the initial iteration position of the new time step.

[0075] Step S1042: Starting from the initial iteration position, register the source point cloud to the target point cloud to obtain the coordinate transformation matrix from the bone coordinate system to the world coordinate system.

[0076] In the specific implementation of this step, the source point cloud is taken as the starting point from the initial iteration position. With target point cloud Re-registration is required.

[0077] Preferably, step S1042 includes:

[0078] Step S10421: with the aforementioned As the initial coordinate transformation matrix, iteratively solve for the new time step. coordinate transformation matrix ;

[0079] Preferably, the registration algorithm is the Point-to-plane ICP (point-to-plane nearest point iteration) method, based on the initial coordinate transformation matrix. Iteratively solve for the new time step coordinate transformation matrix .

[0080] Step S10422: Obtain the image at the time of image acquisition (i.e. Spatial localization of ultrasound images corresponding to a given time (e.g., time) to obtain new... Then, the coordinate transformation matrix from the new bone coordinate system to the world coordinate system is obtained as follows:

[0081] ;

[0082] in, This is the coordinate transformation matrix from the ultrasonic coordinate system to the world coordinate system.

[0083] Thus, through steps S1041-S1042, the coordinate transformation matrix from the bone coordinate system to the world coordinate system at each time step can be obtained, thereby achieving real-time pose update. Step S104 presents a method for real-time pose update using ultrasound image streams, and by reasonably defining the starting position of registration, the number of registration iterations is reduced, ensuring real-time navigation.

[0084] It is conceivable that step S102 may include the step of initial registration.

[0085] In this step, at the start of the surgery, a set of ultrasound patches is fixed to the patient's skin surface to generate a real-time stream of ultrasound images. Specifically, while keeping the patient relatively still, a set of ultrasound images with spatial positioning is acquired, consisting of B-mode ultrasound images in at least two planes within a characteristic region.

[0086] The ultrasound image or waveform is transmitted to the artificial intelligence model trained in step S103 to predict the bone-soft tissue interface features. After obtaining the bone-soft tissue interface prediction result, a series of post-processing steps are performed on the prediction mask (this application is not limited to this step; see the implementation method given below) to extract several points on the bone-soft tissue interface to form a source point cloud for registration. .

[0087] Initial registration of the patient's bone tissue was performed using the Point-to-plane ICP registration method to align the target point cloud. and ultrasound image source cloud Perform registration. The mathematical expression of the registration method can be as follows:

[0088] Registration with target area Using the centroid as the default starting position, iteratively optimize the following expression to solve for the error. Minimize coordinate transformation :

[0089] ;

[0090] in, Let R be the coordinate transformation matrix from the bone (model) coordinate system to the ultrasound coordinate system, where R and t are the rotation matrix and translation vector, respectively.

[0091] To obtain from the target point cloud Calculate each point The normal vector on the neighborhood is obtained by the following method:

[0092]

[0093] That is, the covariance matrix of the neighborhood points The eigenvector corresponding to the smallest eigenvalue.

[0094] Obtain the spatial localization of the ultrasound image at the time of image acquisition. The transformation matrix from the bone coordinate system to the world coordinate system, which can be obtained from each ultrasound patch, is as follows:

[0095] .

[0096] The following provides supplementary explanations of several details in the technical solution of this invention.

[0097] 1. Post-processing the prediction results of the artificial intelligence model.

[0098] To extract several points on the bone-soft tissue interface from the bone-soft tissue interface features / masks predicted by the artificial intelligence model and form a source point cloud for registration, some post-processing is required. One implementation method is as follows, that is, as an optional embodiment, such as... Figure 7 As shown, in step S103, several points are extracted on the bone-soft tissue interface to form a source point cloud for registration, including:

[0099] Step A1: Based on the region with a predicted probability higher than a set threshold in the segmentation mask output by the artificial intelligence model, the edge region is a strip region on the bone-soft tissue interface. By expanding or contracting at the pixel level, the width of the segmentation mask is made to be 3 to 10 pixels. The largest connected region of the segmentation mask is selected as the target mask to be sampled.

[0100] In this step, the threshold can be set flexibly according to the requirements, such as 0.7, 0.8, 0.9, etc., and the width of the segmentation mask is preferably 5-7 pixels.

[0101] Step A2: On the target mask, an adaptive sampling strategy based on local brightness extrema and spatial uniform distribution is used to extract feature points that constitute the source point cloud.

[0102] Preferably, step A2 includes:

[0103] Step A21: Starting from the left starting position of the segmentation mask, extract a region of a certain width along the interface contour direction. Within this region, slide to select multiple local neighborhood windows and calculate the average gray value of each neighborhood.

[0104] This step is used to extract a region of equal width, which is a region of a certain width along the outline of the interface. The width is preferably 7 to 11 pixels, and the local neighborhood window is preferably 3×3 pixels.

[0105] Step A22: Select the neighborhood with the largest average gray value as the current candidate region, and use the center pixel position of the neighborhood as the bone-soft tissue interface feature point;

[0106] This step is used to find the neighborhood with the highest brightness and to collect feature points. The neighborhood with the highest average gray value is the neighborhood with the strongest ultrasonic echo.

[0107] Step A23: Using the bone-soft tissue interface feature point as the center, set a neighborhood range with a radius of 7 to 11 pixels, and remove all pixels within this range from the mask;

[0108] The purpose of this step is to remove the sampled areas after obtaining the feature points.

[0109] Step A24: Perform connected component analysis on the remaining masks and select the connected region with the largest area as the new target mask;

[0110] Step A25: Repeat steps A21 to A24 until the maximum connected region width of the remaining mask is insufficient to form a valid sampling window.

[0111] In this way, through the steps of truncating equal-width regions, finding neighborhoods with maximum brightness, collecting feature points, and removing sampled regions, until the maximum connected region width of the remaining mask is insufficient to form an effective sampling window, a sufficient number of evenly spaced feature points can be extracted from the bone-soft tissue interface, thereby constructing a source point cloud representing the bone surface contour structure. .

[0112] 2. The ultrasound images are dual-plane images.

[0113] Preferred, such as Figure 8 As shown, in step S102 of this invention, the two-dimensional ultrasound image consists of B-mode ultrasound images in at least two planes within the feature region; that is, the ultrasound image includes B-mode ultrasound images of a meridional array and a latitudinal array that are perpendicular to each other. Thus, using two mutually perpendicular arrays allows for the acquisition of bone-soft tissue surface anatomical features in different directions. Simulation registration verification shows that the stability is far higher than using only a single linear array. This verification is as follows:

[0114] The embodiment generates a simulated three-dimensional bone surface model by superimposing random small protrusions on a Gaussian main peak, and simulates the continuous irregular movement between the ultrasound patch and the bone. A pair of orthogonal simulated linear array probes are used to scan in the sagittal and coronal planes respectively. Based on the positional relationship between the bone model and the linear array probes, a source point cloud P on the bone-soft tissue interface is simulated and generated. The source point cloud P on the bone-soft tissue interface is generated using the sagittal and coronal images at the same time. The real-time performance and accuracy of 200 consecutive pose updates between the source point cloud P and the target point cloud Q are tested when using only sagittal images, only coronal images, and mutually orthogonal sagittal and coronal images. Specific results are shown in Table 1.

[0115] Table 1. Comparison of registration performance among the three schemes

[0116] Image mode Average translation error (mm) Average rotation error (°) Average registration time (s) Number of times the error exceeded the standard (1 mm or 1°) Sagittal plane only 0.931 1.793 0.0242 108 / 200 only coronal plane 0.323 1.109 0.0017 96 / 200 Orthogonal double image 0.105 0.479 0.0017 13 / 200

[0117] As can be seen from the table, using two sets of mutually orthogonal ultrasound images, compared to using only one set of ultrasound images, can significantly reduce pose update errors and the number of frames exceeding the error limit, without increasing computation time. This embodiment aims to illustrate that the present invention, using ultrasound images from at least two planes within a feature region, can achieve good real-time registration and high accuracy.

[0118] 3. Acquire multiple sets of ultrasound image streams on the same bone.

[0119] Preferably, selecting multiple target point clouds Q on the same bone and fixing multiple sets of ultrasound patches / transducers at corresponding positions during surgery helps improve navigation accuracy and stability. Assuming there are multiple sets of continuous ultrasound image streams with positioning on the same bone, then according to step S104, starting from the first... The coordinate transformation from the bone coordinate system to the world coordinate system can be obtained from the group of images. The entire bone coordinate system is transformed into the world coordinate system to all The average value. The specific method for calculating the average value is as follows: for the th Rotation matrix of the group of images First, solve for the quaternion:

[0120] ;

[0121] in, , , , Calculate the average of the quaternions separately, i.e. Then, the averaged quaternion is transformed back into the rotation matrix, which is the average of the rotation transformation matrix, and still satisfies the orthogonality requirement:

[0122] ;

[0123] The average value of the translation vector is:

[0124] ;

[0125] The coordinate transformation from the bone coordinate system to the world coordinate system is as follows:

[0126] .

[0127] In other words, as another optional embodiment, in step S104, the coordinate transformation matrix from the bone coordinate system to the world coordinate system calculated for each ultrasound patch is averaged to obtain the final coordinate transformation matrix from the bone coordinate system to the world coordinate system.

[0128] Surgical navigation systems generally require obtaining the standard error of registration or navigation. The standard deviation of the angles used for registration by this method is:

[0129] ;

[0130] The standard deviation of the displacement is:

[0131] .

[0132] 4. Comparative Experiment 1

[0133] In one implementation, for the prediction of the bone-soft tissue interface in two-dimensional ultrasound images, the artificial intelligence model method in step S103 is compared with existing edge detection methods. The artificial intelligence model method is a YOLO-seg segmentation model based on deep learning, which directly outputs pixel-level segmentation results of the bone interface through multi-scale feature extraction and end-to-end training. The edge detection algorithm scans line by line from the lower boundary of the image, selects the first detected region where the gradient value of brightness change is greater than a certain threshold, and forms an interface envelope for the region through circle fitting. Further, using the interface envelope as the initial iteration position, the Active Contour method is used to converge towards the bone-soft tissue interface (i.e., the position with the largest gradient value of brightness change) until the envelope stabilizes. The performance of the two methods is quantitatively evaluated, using manually labeled data as a real reference, calculating the Dice similarity coefficient, Hausdorff distance, and single-frame prediction time of a set of image prediction results on the same test set.

[0134] Experimental results show that the YOLO-seg model has an average Dice coefficient of 0.92±0.03, a Hausdorff distance of 1.8±0.6 mm, and an average single-frame prediction time of 12 ms; while the traditional edge detection method has a Dice coefficient of 0.62±0.15, a Hausdorff distance of 6.2±2.7 mm, and an average single-frame prediction time of approximately 49 ms. The main problem with traditional methods is that they cannot determine whether the highlighted boundary is a bone-soft tissue interface based on the semantic features of the image, thus some images or regions converge to other irrelevant highlighted signals; and in terms of prediction speed, the time consumption is significantly increased when the boundary is difficult to converge, posing a substantial obstacle to real-time pose updates. The artificial intelligence model described in this invention significantly outperforms traditional methods in terms of prediction accuracy and boundary consistency, while its inference speed meets the requirements of real-time applications, demonstrating significant beneficial effects.

[0135] 5. Comparative Experiment Two

[0136] In another embodiment, to evaluate the impact of the initial iteration position based on kinematic priors on the ICP registration efficiency of the present invention, the source point cloud P and target point cloud Q of the bone-soft tissue interface in adjacent frames are registered using the ICP method described in step S104. On simulated bone surface test data, 200 frames of irregular bone surface motion are simulated to test the average registration time, and the following convergence criteria are set: a maximum number of iterations of 10,000, and a relative fitness convergence threshold of 2 × 10⁻⁶. - ³, the relative root mean square error convergence threshold (Relative_RMSE) is 2 × 10⁻⁶. - ³.

[0137] Experimental results show that, compared to ICP registration using only the previous frame position as input, the convergence time of this invention, based on kinematic priors, is reduced from 3.2 ± 1.1 ms to 1.7 ± 0.4 ms, and the non-convergence rate is reduced from 0.035 to 0.005. This test demonstrates that the initial iterative position generated using kinematic priors in this invention can reduce the ICP iteration burden and improve the pose update speed.

[0138] In summary, the present invention has the following beneficial effects:

[0139] 1. Improved positioning accuracy: By using two-dimensional ultrasound images (i.e., ultrasound images in two or more different planes) to obtain more comprehensive spatial information, it helps to build a more accurate three-dimensional spatial model. Compared with the traditional single-plane imaging method, it can provide more accurate position and orientation information, thereby better realizing non-invasive surgical navigation.

[0140] 2. Enhanced Robustness: Utilizing artificial intelligence models to predict image features enables effective target identification and tracking in complex environments, reducing errors caused by noise, occlusion, and other factors. The AI ​​models can also learn feature representations for different scenarios, giving the system better adaptability and stability in various environments.

[0141] 3. Optimize real-time performance: By performing necessary post-processing on the prediction results of the artificial intelligence model, key features in the image can be quickly predicted, which can further improve the quality and reliability of the data, while ensuring the real-time performance of the system.

[0142] 4. Achieve efficient registration: The point cloud registration algorithm is used to complete real-time pose updates, which can quickly and accurately match the newly acquired data with the existing data, thereby achieving efficient pose tracking and meeting the real-time needs during surgery.

[0143] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time pose update method based on feature prediction and point cloud registration for a surgical navigation system, the surgical navigation system comprising an optical positioning system and an ultrasonic patch, the ultrasonic patch being applied to a characteristic region on the patient's skin surface, the characteristic region being a region with anatomical features, the ultrasonic patch having optical positioning markers that can be captured by the optical positioning system, characterized in that... The method includes: Step S101: Extract bone surface point cloud covering at least the feature region and its adjacent region from the three-dimensional model of the patient's bone tissue as the target point cloud for registration with real-time ultrasound images, wherein the three-dimensional model is obtained by three-dimensional reconstruction using three-dimensional medical image data of the patient's bone tissue; Step S102: Acquire the ultrasound image acquired in real time by the ultrasound patch to obtain a two-dimensional ultrasound image with spatial location; Step S103: Use a pre-trained artificial intelligence model to predict the two-dimensional ultrasound image, obtain the bone-soft tissue interface features, and extract several points on the bone-soft tissue interface to form a source point cloud for registration. Step S104: Register the source point cloud to the target point cloud to achieve real-time pose update; In step S103, extracting several points on the bone-soft tissue interface to form a source point cloud for registration includes: Step A1: Based on the regions in the segmentation mask output by the artificial intelligence model with a predicted probability higher than a set threshold, the width of the segmentation mask is made to be 3 to 10 pixels by expanding or contracting at the pixel level, and the largest connected region of the segmentation mask is selected as the target mask to be sampled. Step A2: On the target mask, an adaptive sampling strategy based on local brightness extrema and spatial uniform distribution is used to extract feature points that constitute the source point cloud; Step A2 includes: Step A21: Starting from the left starting position of the segmentation mask, extract a region of a certain width along the interface contour direction. Within this region, slide to select multiple local neighborhood windows and calculate the average gray value of each neighborhood. Step A22: Select the neighborhood with the largest average gray value as the current candidate region, and use the center pixel position of the neighborhood as the bone-soft tissue interface feature point; Step A23: Using the bone-soft tissue interface feature point as the center, set a neighborhood range with a radius of 7 to 11 pixels, and remove all pixels within this range from the mask; Step A24: Perform connected component analysis on the remaining masks and select the connected region with the largest area as the new target mask; Step A25: Repeat steps A21 to A24 until the maximum connected region width of the remaining mask is insufficient to form a valid sampling window.

2. The real-time pose update method based on feature prediction and point cloud registration according to claim 1, characterized in that, In step S102, the two-dimensional ultrasound image is composed of B-mode ultrasound images in at least two planes within the feature region.

3. The real-time pose update method based on feature prediction and point cloud registration according to claim 1, characterized in that, In step S103, the artificial intelligence model is a neural network model with end-to-end real-time segmentation capability. The artificial intelligence model takes a two-dimensional ultrasound image as input and can output a segmentation mask of the target features in the image.

4. The real-time pose update method based on feature prediction and point cloud registration according to claim 3, characterized in that, The artificial intelligence model includes: The feature extraction module is used to receive the input image and extract multi-scale semantic feature information. It is constructed using a convolutional neural network or a structure based on a self-attention mechanism. The feature fusion module is used to fuse feature information from different scales or spatial locations, thereby enhancing the model's reasoning ability for complex structures and fuzzy boundaries. It is implemented using feature pyramids or multi-layer Transformer encoder-decoder networks. The output prediction module is used to generate the location, category, and corresponding segmentation mask of the feature regions in the image after extracting and fusing features.

5. The real-time pose update method based on feature prediction and point cloud registration according to any one of claims 1-4, characterized in that, Step S104 includes: Step S1041: Based on the current time step The coordinate transformation matrix and the previous time step With the current time step Calculate the new time step based on the motion parameters between them. The initial iteration position; Step S1042: Starting from the initial iteration position, register the source point cloud to the target point cloud to obtain the coordinate transformation matrix from the bone coordinate system to the world coordinate system.

6. The real-time pose update method based on feature prediction and point cloud registration according to claim 5, characterized in that, In step S1041, the new time step The coordinate transformation matrix corresponding to the initial iteration position is: The rotation matrix and translation vector are respectively: ； ； The vector representation of motion parameters is: velocity acceleration angular velocity angular acceleration Exp and Log represent the exponential and diagonal mappings, respectively. This is the coordinate transformation matrix from the bone coordinate system to the ultrasound coordinate system.

7. The real-time pose update method based on feature prediction and point cloud registration according to claim 6, characterized in that, Step S1042 includes: Step S10421: with the aforementioned As the initial coordinate transformation matrix, iteratively solve for the new time step. coordinate transformation matrix ; Step S10422: Obtain the spatial localization of the ultrasound image at the time of image acquisition, and obtain new... Then, the coordinate transformation matrix from the new bone coordinate system to the world coordinate system is obtained as follows: ； in, This is the coordinate transformation matrix from the ultrasonic coordinate system to the world coordinate system.

8. The real-time pose update method based on feature prediction and point cloud registration according to claim 5, characterized in that, The number of ultrasound patches is two or more. In step S104, the coordinate transformation matrix from the bone coordinate system to the world coordinate system calculated for each ultrasound patch is averaged to obtain the final coordinate transformation matrix from the bone coordinate system to the world coordinate system.

Images