A method, device, medium and robot for positioning a hematoma in a stroke surgery robot

Abstract

The application provides a stroke operation robot hematoma positioning method, device, medium and robot, which is used for realizing accurate positioning of stroke hematoma. The stroke operation robot hematoma positioning method comprises the following steps: acquiring an RGB image, a depth image and a CT image of a head face of a patient; pre-processing the RGB image; inputting the pre-processed image into a pre-constructed HA-Mask-Rcnn network to obtain a face mask; obtaining face point cloud according to a camera internal parameter and in combination with the face mask and the depth image, and performing fine segmentation on the face point cloud; obtaining head point cloud from the CT image, and performing coarse segmentation on the head point cloud; after registering the fine segmented face point cloud and the coarse segmented head point cloud, determining hematoma positioning information in combination with the head point cloud; and based on the hematoma positioning information, using a path planning algorithm to generate an obstacle avoidance path for the operation robot to reach the hematoma position.

Description

Technical Field

[0001] This invention belongs to the field of medical robots, specifically relating to a method, device, medium, and robot for hematoma localization in stroke surgery. Background Technology

[0002] Hematoma localization in stroke surgery robots is a core technology for improving the efficiency of stroke treatment. The key challenge in hematoma localization lies in accurately identifying the target region from complex brain tissue and mapping it to physical space. Traditional methods rely on doctors manually analyzing two-dimensional CT or MRI images, which suffers from large subjective errors and is time-consuming. Modern technology, however, achieves a breakthrough through deep learning-driven three-dimensional reconstruction: based on network architectures such as U-Net and 3D ResNet, it can automatically segment the hematoma contour and construct a high-precision three-dimensional model, especially demonstrating stronger recognition capabilities for lesions with low contrast or blurred boundaries.

[0003] Precise hematoma localization is one of the important tasks in stroke surgery. Summary of the Invention

[0004] This invention provides a method, device, medium, and robot for hematoma localization in stroke surgery, which can be used to achieve precise localization of stroke hematoma.

[0005] The technical solution of this application is as follows:

[0006] On the one hand, this application provides a method for hematoma localization in a stroke surgery robot, including:

[0007] Acquire RGB images, depth images, and CT images of the patient's head and face;

[0008] Preprocessing of RGB images;

[0009] The preprocessed image is input into a pre-built HA-Mask-Rcnn network to obtain a face mask;

[0010] Based on camera intrinsics, combined with face mask and depth image, face point cloud is obtained, and the face point cloud is finely segmented.

[0011] Head point cloud was obtained from CT images, and coarse segmentation was performed on the head point cloud;

[0012] After registering the finely segmented face point cloud and the coarsely segmented head point cloud, the hematoma localization information is determined by combining the head point cloud.

[0013] Based on the hematoma location information, a path planning algorithm is used to generate an obstacle avoidance path for the surgical robot to reach the hematoma location.

[0014] Preferably, the steps for preprocessing the RGB image include:

[0015] Filtering, grayscale conversion, and contrast enhancement are performed on RGB images.

[0016] Preferably, the step of inputting the preprocessed image into a pre-constructed HA-Mask-Rcnn network to obtain a face mask includes:

[0017] Multi-scale feature extraction is performed on the preprocessed image;

[0018] Channel attention processing is applied to the extracted multi-scale features to enhance channel information;

[0019] The multi-scale features after channel attention processing are then subjected to spatial attention processing to enhance spatial information.

[0020] Region suggestions are generated based on the enhanced multi-scale features;

[0021] The preprocessed image is processed by a gradient boosting decision tree (GBDT) network, and high-dimensional features of multiple facial key points are obtained by regression analysis after embedding them into a high-dimensional space.

[0022] Cross-attention processing is applied to multi-scale features, region proposals, and high-dimensional features to achieve feature enhancement.

[0023] Perform a pooling operation with no resolution loss on the features after cross-attention processing;

[0024] The features output by pooling are regressed and classified to finally output a face mask.

[0025] Preferably, the step of obtaining the face point cloud based on camera intrinsic parameters, combined with the face mask and depth image, includes:

[0026] Based on the face mask, depth information of the face region is extracted from the depth image;

[0027] The extracted depth information is combined with the pixel coordinates of the face mask, and then converted into three-dimensional spatial coordinates through the intrinsic parameter matrix of the depth camera to generate a face point cloud.

[0028] Preferably, the DBSCAN clustering algorithm is used for fine segmentation of the face point cloud, and the K-means++ clustering algorithm is used for coarse segmentation of the head point cloud.

[0029] Preferably, the step of registering the finely segmented face point cloud and the coarsely segmented head point cloud, and then combining the head point cloud to determine the relative pose of the face and the hematoma includes:

[0030] The fast 4PCS algorithm is used to align the finely segmented face point cloud and the coarsely segmented head point cloud into the same coordinate system, thereby achieving coarse registration of the face point cloud and the head point cloud.

[0031] Identify coarse localization information of hematoma in head point cloud;

[0032] The normal ICP algorithm is used to perform fine registration on the coarsely registered face point cloud and head point cloud.

[0033] In the finely registered head point cloud, the precise location information of the hematoma is obtained based on the coarse localization information of the hematoma.

[0034] Preferably, the RRT* algorithm is used to generate an obstacle avoidance path for the surgical robot to reach the hematoma location.

[0035] On the other hand, this application also provides a hematoma localization device for a stroke surgery robot, comprising:

[0036] The input module is used to acquire RGB images, depth images, and CT images of the patient's head and face;

[0037] The preprocessing module is used for preprocessing RGB images;

[0038] The HA-Mask-Rcnn module is used to input the preprocessed image into the pre-built HA-Mask-Rcnn network to obtain the face mask;

[0039] The face point cloud module is used to obtain face point clouds based on camera intrinsic parameters, combined with face masks and depth images, and to perform fine segmentation on the face point clouds.

[0040] The head point cloud module is used to obtain head point clouds from CT images and perform coarse segmentation on the head point clouds.

[0041] The registration algorithm module is used to register the finely segmented face point cloud and the coarsely segmented head point cloud, and then combine the head point cloud to determine the hematoma location information.

[0042] The obstacle avoidance path planning module is used to generate an obstacle avoidance path for the surgical robot to reach the hematoma location based on the hematoma location information and using a path planning algorithm.

[0043] On the other hand, this application also provides a medical robot, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0044] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0045] The beneficial effects of this application are as follows:

[0046] By using HA-Mask-RCNN and a fast registration algorithm (fast 4PCS + normal ICP), millimeter-level localization of hematoma is achieved, reducing surgical errors. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a hematoma localization and obstacle avoidance algorithm for a stroke surgical robot according to the present invention;

[0048] Figure 2 This is a schematic diagram of the image preprocessing module in S2 of the present invention;

[0049] Figure 3 This is a schematic diagram of the Mask-Rcnn algorithm flow in S3 of this invention;

[0050] Figure 4 This is a schematic diagram of the HA-Mask-Rcnn model structure in S3 of this invention;

[0051] Figure 5 This is a schematic diagram of the hybrid attention module in S3 of the present invention;

[0052] Figure 6 This is a schematic diagram of the channel attention algorithm of the hybrid attention module in S32 of the present invention;

[0053] Figure 7 This is a schematic diagram of the spatial attention algorithm of the hybrid attention module in S33 of the present invention;

[0054] Figure 8 This is a schematic diagram of the cross-attention algorithm for facial key points in the hybrid attention module of this invention, S37.

[0055] Figure 9 This is a schematic diagram of the radiometric invariant four-point set of the fast 4PCS algorithm in S61 of this invention;

[0056] Figure 10 This is a schematic diagram of the correspondence between four point sets based on affine invariance in the fast 4PCS algorithm of this invention S61;

[0057] Figure 11 This is a schematic diagram of the target point query based on spatial partitioning in the fast 4PCS algorithm of this invention S61;

[0058] Figure 12 This is a schematic diagram of the target point query based on angle information in the fast 4PCS algorithm of this invention S61;

[0059] Figure 13 This is a flowchart of the ICP registration process of the normal ICP algorithm in S62 of the present invention;

[0060] Figure 14 This is a schematic diagram of the path planning of the RTT* algorithm in S71 of the present invention;

[0061] Figure 15 This is a schematic diagram of the RTT* algorithm for reselecting the parent node in S71 of the present invention;

[0062] Figure 16 This is a schematic diagram of the RTT* algorithm rerouting in S71 of the present invention. Detailed Implementation

[0063] Reference Figure 1 This application provides a method for hematoma localization in a stroke surgery robot, comprising:

[0064] S1 acquires RGB images, depth images, and CT images of the patient's head and face.

[0065] The specific implementation method for acquiring camera images, camera depth maps, and head and hematoma CT scans is as follows:

[0066] The invention employs a camera, a depth camera, and a medical CT scanner to acquire RGB images, depth images, and CT images of the patient's face and head. The specific types of cameras, depth cameras, and medical CT scanners used are not limited in this embodiment.

[0067] S2, preprocessing the RGB image.

[0068] Input an RGB image, and through the preprocessing module, obtain the preprocessed image, such as... Figure 2 As shown.

[0069] In the preprocessing module, the RGB image is filtered, converted to grayscale, and its contrast is enhanced.

[0070] Specifically, in this embodiment of the invention, Gaussian filtering is used to preprocess the RGB image to adapt it to subsequent learning-based or traditional image algorithms.

[0071] For the RGB image after Gaussian filtering, a grayscale algorithm is used to convert the RGB image so that the facial key points can be extracted by machine learning algorithm. This invention uses a weighted average grayscale algorithm to obtain the grayscale image of the RGB image. The weighted average grayscale algorithm is as follows (1):

[0072]

[0073] In formula (1), R, G, and B represent the values ​​of red, green, and blue colors in the RGB image after Gaussian filtering, respectively.

[0074] For the grayscale RGB image, image contrast enhancement is used to increase the difference between different brightness levels, highlight the features of the edges, and reduce unimportant information in the image. This invention uses linear contrast stretching to enhance the contrast of the RGB image, as shown in formula (2):

[0075]

[0076] Where max and min are the minimum and maximum pixel values ​​in the grayscale RGB image, and input(x, y) is the pixel value at coordinates (x, y) in the grayscale RGB image.

[0077] S3, through the HA-Mask-Rcnn module, obtains the face mask from the preprocessed image obtained in S2.

[0078] The Mask-R-CNN algorithm flow of the HA-Mask-R-CNN module in this embodiment of the invention is as follows: Figure 3 As shown, it obtains the HA-Mask-Rcnn module by adding a hybrid attention mechanism to the existing Mask-Rcnn algorithm. A schematic diagram of the model structure of the HA-Mask-Rcnn module in this embodiment is shown below. Figure 4 As shown.

[0079] Optionally, the HA-Mask-Rcnn module includes a backbone network submodule, a channel attention submodule, a spatial attention submodule, a region proposal submodule, a GBDT submodule, a cross attention submodule, ROIAlign, a classification head, and a segmentation head submodule.

[0080] Spatial attention and channel attention are achieved by adding weighted learning networks to update the extracted spatial and channel features, making the model more sensitive to color and spatial features.

[0081] In addition, to improve the accuracy of edge segmentation, this embodiment of the invention also designs a cross-attention mechanism based on facial key points. It uses facial key point information as query value and performs cross-attention operation with image features to enhance image features and achieve a more robust face point cloud segmentation algorithm.

[0082] This invention adds a hybrid attention mechanism to the Mask-R-CNN module. This mechanism enhances the features extracted by the Backbone network in the traditional Mask-R-CNN module using keypoint features, thus achieving face mask extraction based on the HA-Mask-R-CNN algorithm. In this embodiment, the hybrid attention module is as follows: Figure 5 As shown.

[0083] Combination Figure 3 and Figure 4As shown, step S3 specifically includes:

[0084] S31, input the preprocessed image from S2 into the backbone network submodule for multi-scale feature extraction.

[0085] Combination Figure 3 The backbone network first receives the input image processed in step S2 and downsamples it layer by layer. After downsampling, features at different levels are extracted and then combined by feature concatenation. Subsequently, the network upsamples the concatenated features to enhance the resolution of the feature map and restore spatial details. Finally, the backbone network outputs a series of features with rich multi-scale information.

[0086] S32 performs channel attention processing on the extracted multi-scale features to enhance channel information.

[0087] Channel attention mechanisms allow models to focus more intently on analyzing high-dimensional color-related information; their structure is as follows: Figure 6 As shown, the channel attention submodule enhances the channel information of multi-scale features through the following steps:

[0088] S321, Max Pooling and AvgPool are performed on the multi-scale feature F to capture global information in different regions;

[0089] S322, the feature maps of max pooling (MaxPool) and average pooling (AvgPool) are processed by a multilayer perceptron (MLP); the MLP typically contains one or more fully connected layers to learn non-linear combinations of features;

[0090] S323, add the max pooling and average pooling feature maps processed by the multilayer perceptron (MLP) to obtain the fused representation;

[0091] S324, using the activation function (e.g., sigmoid or ReLU) performs a non-linear transformation on the fused features to generate channel attention weights. .

[0092] S325 will generate channel attention weights The weighted multiscale feature CA is obtained by multiplying the input multiscale feature F element by element.

[0093] Steps S321-S325 above are described using equations (3) and (4):

[0094]

[0095]

[0096] Where F represents the multi-scale feature output in step S31, and the activation function layer uses... This indicates that the max pooling operation uses Max Pool( The average max pooling operation uses Max Pool( Multilayer perceptron operations are represented by MLP(·). represents the generated channel attention weights, and CA represents the weighted multi-scale features.

[0097] S33 performs spatial attention processing on the multi-scale features after channel attention processing to enhance spatial information.

[0098] Spatial attention mechanisms determine the level of importance a model places on a particular region by adjusting the weights of spatial dimensions. Their structure is as follows: Figure 7 As shown, the spatial attention submodule enhances the channel information of multi-scale features through the following steps:

[0099] S331, Max Pooling and AvgPool are performed on the multi-scale features CA after weighting in S325 to obtain the maximum and average values ​​of spatial information, respectively;

[0100] S332, the pooled feature map is passed through a convolutional layer Conv 7×7 This convolutional layer is used to learn weights in the spatial dimension.

[0101] S333 uses an activation function σ2 (such as sigmoid) to perform a non-linear transformation on the output of the convolutional layer, generating spatial attention weights. ;

[0102] S334, the generated spatial attention weights The weighted feature map SA is obtained by element-wise multiplying the weighted multi-scale feature CA with the weighted feature map CA.

[0103] Steps S331-S334 above are described using equations (5) and (6):

[0104]

[0105]

[0106] In the formula, CA is the multi-scale feature CA after S325 weighting, and the activation function layer uses... This indicates that the max pooling operation uses Max Pool( The average max pooling operation uses Max Pool( ), This indicates digit-wise multiplication. Represents spatial attention weights. This represents the weighted feature map.

[0107] S34 generates region proposals based on the enhanced multi-scale features.

[0108] In this embodiment, the region proposal submodule uses the Region Proposal Network (RPN) to generate region proposals.

[0109] The Region Proposal submodule uses a Region Proposal Network (RPN) to generate region proposals on the spatially attention-processed feature map SA. The RPN slides a fixed-size anchor box across the feature map, predicting the bounding box offset and confidence score for each anchor box. These anchor boxes are a predefined set of bounding boxes that slide across different locations and scales on the feature map to cover all regions that may contain the target object.

[0110] Feature notation SA generated by S334 Where H and W are the height and width of the feature map, and C is the number of channels, for each position of the feature map. Generates preset multi-scale, multi-scale anchor boxes, each anchor represented as... , Indicates the center coordinates, This indicates the width of the Anchor. Indicates the height of the Anchor.

[0111] Apply a 3 to the feature map SA Three convolutional layers are used to obtain new feature maps. This is used to extract contextual features for each location. Apply 1 respectively 1. Convolution for classification and regression operations:

[0112] Classification operation: Output dimension is Each location ( Typically, there are 9 corresponding values. The foreground and background probabilities of each Anchor are calculated as follows: for each Anchor, the prediction p∈[0,1] represents the confidence level that it belongs to the foreground.

[0113] Regression operation: Output dimension is 4 for each position Value pairs The four coordinate offsets of each anchor Used to adjust the Anchor into a candidate region (Proposal).

[0114] For each Anchor, the candidate region is calculated using the following formulas (7)-(10):

[0115]

[0116]

[0117]

[0118]

[0119] Retain proposals with foreground probability p ≥ threshold (e.g., 0.7) in the classification branch, and filter out low-confidence background boxes. For all candidate proposals, sort them from highest to lowest foreground probability. Select the proposal with the highest probability and delete the remaining proposals with an intersection-union ratio ≥ threshold (e.g., 0.7) to avoid duplicate detection of the same target.

[0120] The final retained proposal is the Region of Interest (ROI) output by the RPN.

[0121] S35, the preprocessed image is processed by a gradient boosting decision tree (GBDT) network, and high-dimensional features of multiple facial key points are obtained by regression analysis after embedding them into a high-dimensional space.

[0122] Specifically, the preprocessed image in S2 is input into the GBDT submodule to obtain high-dimensional features obtained by embedding 68 facial key points into a high-dimensional space.

[0123] Gradient Boosting Decision Tree (GBDT) is an ensemble learning algorithm based on decision trees. It can solve complex regression, classification, and ranking problems by progressively optimizing the residuals predicted by the model. This invention uses a GBDT model to regress 68 key points of a face. The image in S2, which has been preprocessed by image filtering and contrast enhancement, is input and regressed by the GBDT model. The 68 key points of the face are embedded into the high-dimensional features obtained in the high-dimensional space. The algorithm will build multiple serial regression trees. Each regression tree will learn from the error of the regression tree obtained in the previous round of training. Therefore, in the process of multiple rounds of training of the regression tree, the output of the model will get closer and closer to the real label. The GBDT-based regression model will also gradually regress a clear and complete facial contour. The iterative formula (11) of the cascaded residual regression tree is as follows:

[0124]

[0125] in This represents the facial key point features regressed by the model after t iterations, and its representation is the coordinates of 68 key points; It is the learning rate, used to prevent overfitting. This is the regressor in round t. The regression method used in this invention trains parameters by minimizing the sum of squared errors between the predicted and true values ​​as the direction of gradient boosting. The input is the output of the previous round. And the preprocessed image I in S2, the output is the correction value learned based on the error of the previous round, i.e., in the formula .

[0126] S36 performs cross-attention processing on multi-scale features, region proposals, and high-dimensional features to achieve feature enhancement.

[0127] This invention employs cross-attention of facial key points, such as... Figure 8 As shown, the multi-scale features obtained through the backbone network submodule, channel attention submodule, and spatial attention submodule, the region proposals proposed by the region proposal submodule, and the high-dimensional features obtained by embedding the 68 keypoints regressed by the GBDT submodule into the high-dimensional space are enhanced by the cross-attention submodule.

[0128] The Query value Q is calculated by multiplying high-dimensional facial landmark features by a learnable parameter matrix. The key value K and value V are obtained by multiplying the multi-scale features by the learnable parameter matrix. By calculating the similarity between the Q and K values, we can obtain the degree of response of different feature vectors to a certain vector. This degree of response can be used as a weight to update the feature by applying it to the V value, which is called feature enhancement.

[0129] In the first stage of cross-attention, Q and K will be used to calculate their similarity. This invention uses the dot product method to calculate the similarity, as shown in formula (12):

[0130]

[0131] The output similarity vector is normalized to weights using the SoftMax function. As shown in equation (13):

[0132]

[0133] in This represents the similarity between the Q-vector and the i-th K-vector. This represents the total number of key vectors.

[0134] The final stage of the cross-attention mechanism weights V and the output of the previous stage, which is equivalent to highlighting the values ​​in the V vector that have a higher reactivity with the Q vector and weakening the values ​​in the V vector that have a lower reactivity with the Q vector, as shown in equation (14):

[0135]

[0136] This invention applies a cross-attention mechanism to a face segmentation model, where Q is generated from 68 facial key points output by the aforementioned GBDT, and K and V are generated from features output by the backbone network of Mask-Rcnn to enhance the features.

[0137] S37 performs a pooling operation without resolution loss on the features processed by the cross-attention mechanism.

[0138] The ROIAlign submodule is used to perform pooling operations on the features output by the cross-attention module without resolution loss.

[0139] Each ROI in the S36-enhanced feature map is divided into... There are three equal sub-regions (Bin). Within each Bin, a fixed number of sampling points are calculated using bilinear interpolation. The coordinates of these sampling points are floating-point numbers, directly interpolated from the feature map to obtain precise values, avoiding any rounding operations. For each Bin, the set of sampling points is... kR is the number of sampling points within each Bin. The pixel values ​​of the sampled points are pooled using max pooling to generate feature maps. As shown in equation (15):

[0140]

[0141] in This represents the output value of the Bin after max pooling.

[0142] S38 performs regression classification on the features output by pooling, and finally outputs the face mask.

[0143] The pooled output features are used for regression classification in the classification head and segmentation head sub-modules to finally output the face segmentation mask.

[0144] Classification Head: First, perform global feature extraction, and then extract the feature map output by S37. After dimensionality reduction through two 1024-dimensional fully connected layers, it becomes as shown in equation (16):

[0145]

[0146] This represents the feature map after dimensionality reduction. This represents the activation function. This refers to the dimensionality reduction of the first 1024-dimensional fully connected layer. This refers to the dimensionality reduction of the second 1024-dimensional fully connected layer.

[0147] Then, face classification is performed, with a fully connected layer connected to a Sigmoid activation function, outputting the face confidence score. Next, a separate fully connected layer is used to perform bounding box regression to optimize the face bounding box coordinates, and the original ROI bounding box is then used. Adjust to real face frame

[0148] Segmentation Head: First, feature processing is performed:

[0149] The feature map output by S37 Stack 4 The convolutional layer is downsampled as shown in equation (17):

[0150]

[0151] Upsampling is performed to restore the resolution, and the feature map size is expanded to twice by transpose convolution to ensure mask accuracy, as shown in equation (18):

[0152]

[0153] Perform pixel-by-pixel classification to obtain the probability that each pixel value belongs to a human face. As shown in equation (19):

[0154]

[0155] Then, mask generation and post-processing are performed:

[0156] For each pixel, if If the value is greater than 0.5, it is identified as a face (value 1); otherwise, it is background (value 0), and a binary face mask is generated.

[0157] S4, based on camera intrinsic parameters, combines the face mask and depth image to obtain the face point cloud.

[0158] Specifically, S4 includes:

[0159] S41, based on the face mask, extracts the depth information of the face region from the depth image.

[0160] A face mask is a binary image that marks the location of the face region. By applying this mask to a depth image, the depth values ​​corresponding to the face region can be extracted from the depth image. The purpose of this step is to filter out the depth information related to the face from the depth image, ignoring the depth information of the background and other non-face regions.

[0161] Applying the face mask obtained in step S3 to the depth image means multiplying the depth value of each pixel in the depth image by the corresponding face mask pixel value; in this way, only the depth value of the face region is retained, and the depth value of the background region is set to an invalid value.

[0162] After the above processing, only the depth information of the face region is retained in the depth image.

[0163] S42 combines the extracted depth information with the pixel coordinates of the face mask, and converts it into three-dimensional spatial coordinates through the intrinsic parameter matrix of the depth camera to generate a face point cloud.

[0164] After extracting the depth information of the face region, this depth information needs to be combined with the pixel coordinates of the face mask. Using the intrinsic parameter matrix of the depth camera, the two-dimensional pixel coordinates are converted into three-dimensional spatial coordinates. The purpose of this step is to "upgrade" the depth information of the face region from the two-dimensional space of the depth image to three-dimensional space, thereby generating face point cloud data.

[0165] Specifically, for a point P, its coordinates on the imaging plane can be represented as (x, y, z), where x and y are pixel values, z represents the depth of point P (i.e., the distance from point P to the camera), and the points projected onto the imaging plane are represented as... This means that to obtain the three-dimensional spatial coordinates of the depth camera coordinate system... Then, the transformation can be performed using equations (20) and (21):

[0166]

[0167]

[0168] in, This represents the point projected onto the imaging plane. The imaging screen refers to the pixel coordinates of the depth image, i.e., (x, y). This is the intrinsic parameter matrix of the depth camera; It is the inverse of the intrinsic parameter matrix of the depth camera; This represents a three-dimensional spatial coordinate point in the depth camera coordinate system, where the three-dimensional coordinates consist of (x, y, z).

[0169] S43, the obtained face point cloud is finely segmented based on the DBSCAN clustering algorithm.

[0170] Since the resulting face point cloud contains useless point clouds, this invention utilizes a clustering algorithm for fine regression analysis. Specifically, this embodiment uses the DBSCAN clustering algorithm to cluster point cloud clusters and ultimately extracts the point cloud cluster with the largest number of point clouds as the final fine segmentation data output for the face point cloud. The obtained face point cloud is then finely segmented using the clustering algorithm, which removes useless point clouds and retains the point clouds representing the facial features.

[0171] S5. Obtain head point cloud from CT images.

[0172] The head CT scan was reconstructed using 3DSlicer to obtain the head point cloud, and then the head point cloud was coarsely segmented.

[0173] In this embodiment of the invention, a K-means++ clustering algorithm is used to coarsely segment the head point cloud, thereby improving the clustering quality and the stability of point cloud segmentation.

[0174] S6. After registering the face point cloud and head point cloud, the relative positions of the face and hematoma are determined by combining the head point cloud.

[0175] The registration algorithm module includes a fast 4PCS algorithm submodule and a normal ICP algorithm submodule.

[0176] Step S6 includes:

[0177] S61, the fast 4PCS algorithm submodule uses the fast 4PCS algorithm to align the finely segmented face point cloud and the coarsely segmented head point cloud into the same coordinate system, thus achieving coarse registration of the face point cloud and the head point cloud.

[0178] Specifically, the 4PCS registration algorithm is a matching algorithm based on a two-dimensional affine invariant four-point set. The 4PCS algorithm finds four coplanar points (a, b, c, d) in a reference point cloud as feature descriptions. Assuming there is a set of coplanar points X, these points are only coplanar but not collinear; connecting each pair of these four points will yield an intersection point e; for example... Figure 9 Two ratios can be calculated from the intersection point e, as shown in (22) and (23):

[0179]

[0180]

[0181] Where a, b, c, d represent Figure 9 The four points in the text.

[0182] For any 4 points in the baseline point cloud (head point cloud), the edge ratio is obtained. At any two points in the point cloud (face point cloud) to be registered From the above formula, we can obtain 2 (Depend on (Calculated), 2 Point (by) (Calculated). Specifically: Calculation The intersection of the extension of the connecting line with the lines connecting other points in the reference point cloud; among them, for ,calculate The intersection of the lines connecting other points in the reference point cloud is obtained. ;for ,calculate The intersection of the lines connecting other points in the reference point cloud is obtained. .

[0183] like Figure 10 As shown in the diagram on the right, for two line segments (4 points), if any one of the points in the first line segment... With any one of the second line segments If the two line segments overlap or are within a very small distance, then the four endpoints of these two line segments can be considered to be the corresponding points of four points in the reference point cloud, such as... Figure 10 .

[0184] The 4PCS algorithm uses a specific set of congruent points for coarse registration; however, this algorithm requires finding and traversing all sets of 4 points that satisfy rotation invariance, resulting in a time complexity of O(n log n). Where n represents the size of the point cloud, and k represents the number of candidate sets of congruent 4 points. The biggest speed bottleneck of the 4PCS algorithm is... To reduce the time complexity, this embodiment of the invention improves the query steps of the output-sensitive algorithm for matching point pairs by using the Smart Indexing method to reduce the query complexity. Smart Indexing optimizes the query from two aspects: block query and limiting the query perspective.

[0185] This invention first employs a block-based spatial point query method, dividing the input dataset into several small data blocks and performing matching calculations on each block separately, such as... Figure 11 As shown.

[0186] Furthermore, the selected set of four points may only satisfy the distance relationship among the four points but not the angle relationship. To solve the above two problems, this invention also adds a restriction to the query angle. For a point that has found a pair of coplanar points... If the intersection of the four pairs of points is e, then when searching for other pairs of points, we will use e as the center. For radius, To draw a circle using a compass to represent the angles of the lines of intersection of four points, as shown below. Figure 12As shown. Similarly, an extra width of ε is added to the drawn circular range to find as many candidate point pairs as possible, preventing the optimal solution from being missed. This completes the fast 4PCS algorithm.

[0187] The above fast 4PCS method can greatly reduce the time complexity of registration.

[0188] S62 identifies coarse localization information of hematoma in the head point cloud.

[0189] S63, the Normal ICP algorithm submodule uses the Normal ICP algorithm to perform fine registration on the coarsely registered face point cloud and head point cloud.

[0190] Specifically, the basic idea of ​​the normal ICP algorithm is to analyze the face point cloud. and head point cloud One-to-one matching and minimizing their distance, in the formula and These represent the coordinates of the face point cloud and the head point cloud, respectively. and Let X represent the number of face point clouds and P represent the number of head point clouds. The goal of the ICP algorithm is to calculate a rotation matrix and a translation vector that minimizes the error between point cloud sets X and P.

[0191] Normal ICP registration can be completed through the following steps S631-S635:

[0192] S631, Preprocessing to calculate normal vectors: For face point cloud X and head point cloud P, the normal vector of each point is calculated using the method of fitting a plane to a local neighborhood, which represents the local surface orientation of the face and head point clouds.

[0193] S632, Find corresponding point pairs: For each point in the face point cloud X In the head point cloud P, find the point with the closest spatial distance and whose normal vector angle is less than a threshold (e.g., 45 degrees). ) points This serves as the corresponding point. If the normal constraint is not satisfied, the point is ignored or the next nearest point is found to avoid incorrect matching.

[0194] S633, construct the objective function E, as shown in equation (24):

[0195]

[0196] in, This represents a point in the head point cloud P. This represents a point in the face point cloud X. R represents the rotation matrix, and t is the translation vector. Indicating face dot cloud The normal vector, Indicates head point cloud The normal vector, This represents the normal error weighting parameter.

[0197] The optimal transformation T from face point cloud to head point cloud is found by minimizing E.

[0198] S634, Optimize the transformation matrix: Use the Gauss-Newton optimization algorithm to iteratively update the rotation and translation increments, as shown in equation (25):

[0199]

[0200] in This represents the pose transformation increment, which includes translation and rotation. This represents the translation increment along the x-axis. This represents the translation increment along the y-axis. This represents the translation increment along the z-axis. This represents the rotation increment along the x-axis. This represents the rotation increment along the y-axis. The rotation increment along the z-axis is used to minimize the objective function E, continuously adjusting the position and pose of the face point cloud.

[0201] S635, apply the obtained transformation T to the face point cloud X to update the face point cloud position.

[0202] Repeat the above steps until the change in the transformation matrix is ​​less than the threshold, then stop the iteration. The final transformation matrix T achieves the registration of the face point cloud X to the head point cloud P, transforming the face point cloud into the coordinate system of the head point cloud, thus completing the precise alignment between the two.

[0203] The above-mentioned normal ICP algorithm is used to achieve precise registration of face point clouds and head point clouds.

[0204] S64: Based on the coarse localization information of the hematoma, the precise localization information of the hematoma is obtained from the finely registered head point cloud.

[0205] S7, based on hematoma location information, uses a path planning algorithm to generate an obstacle avoidance path for the surgical robot to reach the hematoma location.

[0206] The Path Redirecting (RRT) algorithm is a classic robot path planning algorithm. It finds efficient, collision-free paths by randomly generating sampling points in 3D space and comparing them with existing sampling points. Because collision detection is included in the sampling process, it can handle constrained task scenarios more efficiently. RRT path planning is as follows: Figure 14 As shown.

[0207] In practical search tasks, the RRT algorithm often has too much randomness and it is difficult to obtain the optimal solution. Therefore, the RRT* algorithm adds the operations of reselecting parent nodes and rewiring to optimize the above problems, thereby optimizing the path of the RRT algorithm to a certain extent and obtaining higher quality results.

[0208] In this embodiment of the invention, the RRT* algorithm based on parent node reselection and rewiring optimization is used for path planning.

[0209] Specifically, the goal of reselecting parent nodes is to optimize the parent nodes of existing nodes. First, each existing node is traversed, and the cost is constructed using the path distance from the root node to the current node and the distance between the current node and the target node. This yields the cost from the root node to the current node. Simultaneously calculate the cost from all unvisited nodes y to that node. ,for Calculate its distance to the current node. And find the point closest to the current node. Then, similarly for the untraversed nodes... Get the root node to The cost Calculate each that can become The parent node's node Take the distance function The smallest node becomes the new parent node, such as... Figure 15 .

[0210] Rewiring is used to optimize the path from existing nodes to new nodes. Let the most recently generated node in the current round be... , Nearby point ,calculate and Path cost The step of selecting the connection with the minimum path cost is equivalent to reducing unnecessary connections in the path, making the agent's movement more direct and efficient. Figure 16 .

[0211] Finally, the obstacle avoidance path planning of the stroke surgery robot was completed using the RRT* algorithm.

[0212] The final step was to complete the hematoma localization and obstacle avoidance path planning for the stroke surgery robot.

[0213] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0214] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0215] It should also be noted that, in this document, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used for ease of description and simplification of the invention, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, relational terms such as "first" and "second" are used to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations, nor should they be construed as indicating or implying relative importance. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements does not include those elements, but also includes other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. In the absence of further restrictions, an element defined by the phrase "includes a..." does not preclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0216] The technical solution provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand this invention, and the content of this specification should not be construed as a limitation of this invention. Furthermore, for those skilled in the art, there will be different forms of changes in the specific implementation methods and application scope based on this invention. It is neither necessary nor possible to exhaustively list all implementation methods here, but obvious changes or modifications derived therefrom are still within the protection scope of this invention.

Claims

1. A method for positioning a hematoma in a stroke surgery robot, characterized by, include: Acquire RGB images, depth images, and CT images of the patient's head and face; Preprocessing of RGB images; The preprocessed image is input into a pre-built HA-Mask-Rcnn network to obtain a face mask; Based on camera intrinsics, combined with face mask and depth image, face point cloud is obtained, and the face point cloud is finely segmented. Head point cloud was obtained from CT images, and coarse segmentation was performed on the head point cloud; After registering the finely segmented face point cloud and the coarsely segmented head point cloud, the hematoma localization information is determined by combining the head point cloud. Based on the hematoma location information, a path planning algorithm is used to generate an obstacle avoidance path for the surgical robot to reach the hematoma location. The steps to obtain the face mask by inputting the preprocessed image into the pre-constructed HA-Mask-Rcnn network include: Multi-scale feature extraction is performed on the preprocessed image; Channel attention processing is applied to the extracted multi-scale features to enhance channel information; The multi-scale features after channel attention processing are then subjected to spatial attention processing to enhance spatial information. Region suggestions are generated based on the enhanced multi-scale features; The preprocessed image is processed by a gradient boosting decision tree (GBDT) network, and high-dimensional features of multiple facial key points are obtained by regression analysis after embedding them into a high-dimensional space. Cross-attention processing is applied to multi-scale features, region proposals, and high-dimensional features to achieve feature enhancement. Perform a pooling operation with no resolution loss on the features after cross-attention processing; The features output by pooling are regressed and classified to finally output a face mask. After registering the finely segmented face point cloud and the coarsely segmented head point cloud, the steps for determining the relative pose of the face and hematoma by combining the head point cloud include: The fast 4PCS algorithm is used to align the finely segmented face point cloud and the coarsely segmented head point cloud into the same coordinate system, thereby achieving coarse registration of the face point cloud and the head point cloud. Identify coarse localization information of hematoma in head point cloud; The normal ICP algorithm is used to perform fine registration on the coarsely registered face point cloud and head point cloud. In the finely registered head point cloud, the precise location information of the hematoma is obtained based on the coarse localization information of the hematoma. Using the fast 4PCS algorithm, the finely segmented face point cloud and the coarsely segmented head point cloud are aligned to the same coordinate system. The steps to achieve coarse registration of the face point cloud and the head point cloud include: Four coplanar and non-collinear points (a, b, c, d) are selected from the head point cloud as feature descriptions. The intersection point e is obtained by connecting each pair of the four points (a, b, c, d) through the intersection point e. The first ratio r1 and the second ratio r2 are calculated through the intersection point e. Based on the first ratio r1, the second ratio r2, and any two points q1 and q2 in the face point cloud, calculate the intersection point e1 of the line connecting the first point q1 to other points in the head point cloud, and calculate the intersection point e2 of the line connecting the second point q2 to other points in the head point cloud; in the face point cloud, for any two line segments, any e1 in the first line segment coincides with any e2 in the second line segment.

2. The stroke surgery robot hematoma localization method according to claim 1, characterized in that, The steps for preprocessing RGB images include: Filtering, grayscale conversion, and contrast enhancement are performed on RGB images.

3. The stroke surgery robotic hematoma localization method of claim 1, wherein, The steps to obtain the face point cloud based on camera intrinsics, combined with the face mask and depth image, include: Based on the face mask, depth information of the face region is extracted from the depth image; The extracted depth information is combined with the pixel coordinates of the face mask, and then converted into three-dimensional spatial coordinates through the intrinsic parameter matrix of the depth camera to generate a face point cloud.

4. The stroke surgery robotic hematoma localization method of claim 1, wherein, The DBSCAN clustering algorithm was used for fine segmentation of the face point cloud, and the K-means++ clustering algorithm was used for coarse segmentation of the head point cloud.

5. The stroke surgery robotic hematoma localization method of claim 1, wherein, The RRT* algorithm is used to generate an obstacle avoidance path for the surgical robot to reach the hematoma.

6. A hematoma localization device for a stroke surgery robot, characterized in that, include: The input module is used to acquire RGB images, depth images, and CT images of the patient's head and face; The preprocessing module is used for preprocessing RGB images; The HA-Mask-Rcnn module is used to input the preprocessed image into the pre-built HA-Mask-Rcnn network to obtain the face mask; The face point cloud module is used to obtain face point clouds based on camera intrinsic parameters, combined with face masks and depth images, and to perform fine segmentation on the face point clouds. The head point cloud module is used to obtain head point clouds from CT images and perform coarse segmentation on the head point clouds. The registration algorithm module is used to register the finely segmented face point cloud and the coarsely segmented head point cloud, and then combine the head point cloud to determine the hematoma location information. The obstacle avoidance path planning module is used to generate an obstacle avoidance path for the surgical robot to reach the hematoma location based on the hematoma location information and using a path planning algorithm. The steps to obtain the face mask by inputting the preprocessed image into the pre-constructed HA-Mask-Rcnn network include: Multi-scale feature extraction is performed on the preprocessed image; Channel attention processing is applied to the extracted multi-scale features to enhance channel information; The multi-scale features after channel attention processing are then subjected to spatial attention processing to enhance spatial information. Region suggestions are generated based on the enhanced multi-scale features; The preprocessed image is processed by a gradient boosting decision tree (GBDT) network, and high-dimensional features of multiple facial key points are obtained by regression analysis after embedding them into a high-dimensional space. Cross-attention processing is applied to multi-scale features, region proposals, and high-dimensional features to achieve feature enhancement. Perform a pooling operation with no resolution loss on the features after cross-attention processing; The features output by pooling are regressed and classified to finally output a face mask. After registering the finely segmented face point cloud and the coarsely segmented head point cloud, the steps for determining the relative pose of the face and hematoma by combining the head point cloud include: The fast 4PCS algorithm is used to align the finely segmented face point cloud and the coarsely segmented head point cloud into the same coordinate system, thereby achieving coarse registration of the face point cloud and the head point cloud. Identify coarse localization information of hematoma in head point cloud; The normal ICP algorithm is used to perform fine registration on the coarsely registered face point cloud and head point cloud. In the finely registered head point cloud, the precise location information of the hematoma is obtained based on the coarse localization information of the hematoma. Using the fast 4PCS algorithm, the finely segmented face point cloud and the coarsely segmented head point cloud are aligned to the same coordinate system. The steps to achieve coarse registration of the face point cloud and the head point cloud include: Four coplanar and non-collinear points (a, b, c, d) are selected from the head point cloud as feature descriptions. The intersection point e is obtained by connecting each pair of the four points (a, b, c, d) through the intersection point e. The first ratio r1 and the second ratio r2 are calculated through the intersection point e. Based on the first ratio r1, the second ratio r2, and any two points q1 and q2 in the face point cloud, calculate the intersection point e1 of the line connecting the first point q1 to other points in the head point cloud, and calculate the intersection point e2 of the line connecting the second point q2 to other points in the head point cloud; in the face point cloud, for any two line segments, any e1 in the first line segment coincides with any e2 in the second line segment.

7. A medical robot, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The steps of implementing the method of any one of claims 1 to 5 when the processor executes a computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of the method of any one of claims 1 to 5.

Images