Real-time motion compensation robot puncture control method based on respiration prediction

Abstract

The invention relates to the technical field of liver puncture control, in particular to a real-time motion compensation robot puncture control method based on respiration prediction. The method comprises the following steps: acquiring breathing phase data based on a breathing sensor at the thorax and abdomen of a patient, and determining a breathing resting period based on the breathing phase data; collecting a liver ultrasonic image and a liver lesion template image of the patient in the respiratory resting period, and determining the position of a liver lesion target spot based on the liver ultrasonic image and the liver lesion template image; based on historical breathing cycle data and real-time breathing phase data, a dynamic trajectory model of the liver moving along with breathing is constructed through a motion prediction algorithm, and the liver displacement trajectory in the subsequent breathing cycle is predicted; the optimal puncture path is determined based on the predicted liver displacement track and the liver lesion target position, and the robot is driven to puncture based on the optimal puncture path, so that the accuracy of positioning the liver lesion position can be improved, the puncture path deviation is reduced, and the puncture operation risk is reduced.

Description

Technical Field

[0001] This invention relates to the field of liver puncture control technology, specifically to a real-time motion compensation robot puncture control method based on respiratory prediction. Background Technology

[0002] Liver biopsy or interventional therapy are commonly used diagnostic and treatment methods in clinical practice. However, because the liver shifts significantly with respiration, traditional puncture methods rely on the patient's breath-holding or manual judgment, which leads to problems such as inaccurate positioning, large deviations in the puncture path, and high surgical risks. Especially under real-time ultrasound guidance, the target position drift caused by respiratory motion can significantly affect puncture accuracy and surgical success rate.

[0003] Some studies have attempted to compensate for respiratory motion through respiratory gating technology or image-based real-time tracking, but most of these methods rely on passive responses to the current respiratory state and lack the ability to predict respiratory trends, resulting in system response lag and difficulty in achieving high-precision positioning in dynamic environments.

[0004] Therefore, there is an urgent need for a robotic puncture control method that can integrate multi-source respiratory information and achieve respiratory motion prediction and real-time compensation in order to improve puncture accuracy and surgical safety under the influence of respiration. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time motion compensation robot puncture control method based on respiratory prediction: aiming to solve the technical problems of inaccurate positioning of liver lesions, large deviation of puncture path, and high surgical risk in existing liver puncture control methods.

[0006] A real-time motion-compensated robotic puncture control method based on respiratory prediction, the method comprising: Respiratory phase data is obtained using respiratory sensors on the patient's chest and abdomen, and the respiratory resting period is determined based on the respiratory phase data. Liver ultrasound images and liver lesion template images were acquired from patients during the respiratory resting period, and the location of liver lesion target points was determined based on the liver ultrasound images and liver lesion template images; Based on historical respiratory cycle data and real-time respiratory phase data, a dynamic trajectory model of liver movement during respiration is constructed using a motion prediction algorithm to predict the liver displacement trajectory in subsequent respiratory cycles. The optimal puncture path is determined based on the predicted liver displacement trajectory and the target location of the liver lesion, and the robot is driven to perform puncture based on the optimal puncture path.

[0007] Furthermore, determining the respiratory resting period based on respiratory phase data specifically includes the following process: Acquire the patient's respiratory phase data, wherein the respiratory phase data includes at least respiratory displacement data, instantaneous velocity data derived from the displacement data, and instantaneous acceleration data derived from the displacement data; Based on respiratory displacement data, peak and trough points in the respiratory cycle are identified to segment continuous respiratory cycles. Based on the respiratory motion characteristics of the current cycle or the most recent historical cycle, a decision threshold is dynamically calculated. The decision threshold is a velocity threshold dynamically calculated based on the ratio of the average amplitude to the average cycle duration of the most recent N historical respiratory cycles. Determine whether the respiratory phase data at the current moment simultaneously meets the following conditions: First condition: The respiratory displacement data is within the high or low range of the dynamic interval defined by the peak displacement and valley displacement of the current respiratory cycle. The high range is defined as the interval extending downward from the peak displacement by a first predetermined proportion, and the low range is defined as the interval extending upward from the valley displacement by a second predetermined proportion. Second condition: The absolute value of the instantaneous velocity data is less than the dynamically calculated velocity threshold; The third condition is that the absolute value of the instantaneous acceleration data is less than the preset acceleration threshold. When the first, second, and third conditions are met simultaneously, a continuous verification mechanism is triggered: respiratory phase data is continuously monitored during a predetermined verification period, and the current respiratory resting period is determined and a corresponding determination signal is output only when the first, second, and third conditions are continuously met during the verification period.

[0008] Furthermore, determining the location of liver lesion target points based on liver ultrasound images and liver lesion template images specifically includes the following process: Image features were extracted from liver ultrasound images and liver lesion template images to obtain contrast features and template features. The comparison features and template features are subjected to feature interaction comparison processing to obtain reference features after feature interaction comparison processing. Location of liver lesions in liver ultrasound images is determined based on reference features.

[0009] Furthermore, image feature extraction is performed on liver ultrasound images and liver lesion template images to obtain contrast features and template features. Specifically, this includes the following processes: Liver ultrasound images Liver lesion template image ,in, Represents the set of real numbers. , Images ,image height, , Images ,image The width is 3, where 3 represents the three color channels of the image (RGB). Will and Each image is split into multiple non-overlapping image patch sequences, denoted as follows: , ,in, , These are the dimensions of the image patches, The size of each image patch is represented, all image patches are flattened, and the embedding representation of the liver ultrasound image is obtained through a linear mapping function. Embedded representation of liver lesion template images ,in, , The length of the embedded representation, , For the dimension of the embedded representation, and The input is fed into the Transformer encoding layer for feature extraction. The extracted contrast features and template features are: , .

[0010] Furthermore, the comparison features and template features are subjected to feature interaction comparison processing to obtain reference features after feature interaction comparison processing, including the following processes: based on and Generate graph structure ,in, This is a set of nodes representing the characteristic information of liver lesions in a liver ultrasound image. , For the number of nodes, This refers to the edge set representing the feature information of liver lesions in the liver lesion template image. The number of sides is ; compute nodes The size of the d-th eigenvalue : ; in, For nodes Features Represents a node Degree centrality; based on Calculate edges Importance probability : ,in, , The range of values ​​is , This indicates the operation of finding the maximum value. ; in, , for The maximum value, for The average value, This is a probability-limited threshold; Importance probability Edges with an importance probability below a preset threshold in the graph structure Deleting the graph results in a graph structure that has undergone information interaction processing. For graph structures that have undergone information interaction processing Feature extraction is performed again to obtain reference features.

[0011] Furthermore, determining the location of liver lesions in liver ultrasound images based on reference features specifically includes the following process: Step 1, refer to the features Mapped to an embedded representation of the liver ultrasound image via a fully connected layer. Consistent dimensions, resulting in mapped reference features ,calculate and Embedded features for each image patch The cosine similarity is used to filter image patches whose cosine similarity is greater than a cosine similarity threshold, forming a set of image patches associated with lesions. ; Step two, according to the non-overlapping segmentation rules of ultrasound image blocks, divide... Each lesion-associated image patch is mapped back to the pixel coordinate system of the original liver ultrasound image to obtain the diagonal pixel coordinates of each lesion-associated image patch; the boundary extrema of the coordinates of all lesion-associated image patches are calculated to determine the coarse localization rectangular region of the lesion. ; Step 3: Call the liver anatomy template to... Structural matching was performed on ultrasound images within the region, excluding non-lesion structural regions with a difference score below 0.6, to obtain a fine-grained candidate region for lesions. ;right After denoising with a 3×3 Gaussian filter, the edges are extracted using the Canny edge detection algorithm. The gaps are then filled using morphological closing operations on 5×5 rectangular structuring elements to obtain a complete and detailed contour of the lesion region. ; Step 4: Calculate using the geometric center method The center coordinates of the minimum bounding rectangle The weighted center coordinates were calculated using the density-weighted method. The location of the liver lesion target point is obtained by calculating the mean of the two center coordinates. , .

[0012] Furthermore, based on historical respiratory cycle data and real-time respiratory phase data, a dynamic trajectory model of liver movement during respiration is constructed using a motion prediction algorithm. The prediction of liver displacement trajectory in subsequent respiratory cycles specifically includes the following processes: Collect complete data for the most recent H historical respiratory cycles. The historical respiratory cycle data includes respiratory displacement data, instantaneous velocity data, instantaneous acceleration data, peak displacement, trough displacement, and cycle duration characteristics for each cycle. Using the peak point of the respiratory cycle as a reference, align all historical respiratory cycle data along the time axis and divide each cycle into four phase intervals: inspiratory phase, end-inspiratory resting phase, expiratory phase, and end-expiratory resting phase. Based on the location of liver lesion targets determined during the respiratory resting period, the displacement coordinates of liver targets in different phase intervals of each historical respiratory cycle are extracted using image tracing technology to establish a "respiratory phase-liver displacement" mapping relationship library; and the historical cycle data are used to form a training dataset. A bidirectional LSTM network was used as the core of the motion prediction algorithm. The training dataset was divided into training set and validation set in a 7:3 ratio. The mean squared error (MSE) was used as the loss function. The Adam optimizer was used to iteratively train the model until the loss function of the validation set converged, thus obtaining the initial liver motion trajectory model. The initial liver motion trajectory model data is fused and the model is dynamically corrected to obtain the corrected dynamic trajectory model. Real-time respiratory phase data is input into the corrected dynamic trajectory model to output the liver displacement trajectory in subsequent respiratory cycles.

[0013] Furthermore, the structure of the bidirectional LSTM network includes: an input layer with a dimension of 9, a hidden layer with 256 neurons, and an output layer with a dimension of 2.

[0014] Compared to existing solutions, the beneficial effects achieved by this invention are: More accurate determination of respiratory resting period: By combining respiratory displacement, instantaneous velocity and acceleration data, and introducing dynamic threshold and continuous verification mechanism, the accuracy and robustness of respiratory resting period determination are significantly improved, providing a stable time window for image acquisition and path planning.

[0015] High lesion localization accuracy: Employing a feature extraction and matching mechanism based on Transformer and graph structure interaction, it can accurately identify and locate lesion areas from ultrasound images. Combined with image block mapping and structure matching strategies, it effectively eliminates interference from non-target areas and improves the reliability of target localization.

[0016] Strong respiratory motion prediction capability: Based on a bidirectional LSTM network, a dynamic trajectory model of the liver is constructed, which integrates multi-cycle historical data and real-time phase information. It can accurately predict the displacement trend of the liver in subsequent respiratory cycles and achieve prospective motion compensation.

[0017] Superior puncture path planning: By combining the predicted liver displacement trajectory with the lesion target location, the optimal puncture path is generated and executed precisely through a robotic system, which significantly reduces puncture errors caused by respiratory movements and improves the safety and success rate of liver puncture. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of a real-time motion compensation robot puncture control method based on respiratory prediction according to an embodiment of the present invention. Figure 2 This is a system block diagram of a respiratory resting period determination system according to an embodiment of the present invention; Figure 3 This is a flowchart of another real-time motion compensation robot puncture control method based on respiratory prediction according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating a real-time motion compensation robot puncture control method based on respiratory prediction, according to an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0022] This embodiment provides a real-time motion compensation robot puncture control method based on respiratory prediction. Figure 1 This is a flowchart illustrating a real-time motion compensation robot puncture control method based on respiratory prediction, according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S101: Obtain respiratory phase data based on respiratory sensors on the patient's chest and abdomen, and determine the respiratory resting period based on the respiratory phase data; Step S102: Acquire liver ultrasound images and liver lesion template images of patients during the respiratory resting period, and determine the location of liver lesion target points based on liver ultrasound images and liver lesion template images; It is worth noting that the liver lesion template image contains images of liver lesion information, which, with the patient's consent, are used by those skilled in the art in the method disclosed in this embodiment.

[0023] Step S103: Based on historical respiratory cycle data and real-time respiratory phase data, a dynamic trajectory model of liver movement with respiration is constructed using a motion prediction algorithm to predict the liver displacement trajectory in subsequent respiratory cycles. Step S104: Determine the optimal puncture path based on the predicted liver displacement trajectory and the target location of the liver lesion, and drive the robot to perform puncture based on the optimal puncture path.

[0024] In summary, this invention acquires respiratory phase data using respiratory sensors on the patient's chest and abdomen, and determines the respiratory resting period based on the respiratory phase data; it acquires liver ultrasound images and liver lesion template images of the patient during the respiratory resting period, and determines the target location of the liver lesion based on the liver ultrasound images and liver lesion template images; based on historical respiratory cycle data and real-time respiratory phase data, it constructs a dynamic trajectory model of liver movement with respiration using a motion prediction algorithm, and predicts the liver displacement trajectory in subsequent respiratory cycles; based on the predicted liver displacement trajectory and the target location of the liver lesion, it determines the optimal puncture path, and drives the robot to perform puncture based on the optimal puncture path. This can improve the accuracy of liver lesion location, reduce puncture path deviation, and lower the risk of puncture surgery.

[0025] In some embodiments, Figure 2 This is a system block diagram of a respiratory resting period determination system according to an embodiment of the present invention, as shown below. Figure 2 As shown, the system includes: Data acquisition module: Used to acquire raw patient respiratory phase data (respiratory signals).

[0026] Signal preprocessing module: Filters the raw patient respiratory phase data.

[0027] Feature extraction module: Calculates respiratory phase data in real time, including but not limited to: respiratory displacement data, instantaneous velocity data (first derivative of displacement), and instantaneous acceleration data (second derivative of displacement). Normalized phase (Φ): Normalizes the current respiratory cycle to 0° to 360°.

[0028] Resting period determination module: Executes the following steps to determine the respiratory resting period based on respiratory phase data.

[0029] Output module: Outputs the respiratory resting period to the doctor's interface and the robot control system.

[0030] The steps for determining the resting respiratory period based on respiratory phase data are as follows: Step 1: Respiratory cycle segmentation and baseline establishment; The system continuously monitors respiratory signals and segments a complete respiratory cycle (e.g., from the end of the previous expiration to the current expiration) by identifying the maximum (peak, corresponding to the end of inspiration) and minimum (trough, corresponding to the end of expiration) points of displacement. The system maintains a sliding window of length N (e.g., N=5) to record historical data for the most recent N cycles, including: the average amplitude of the cycle (A_avg), the duration of the cycle (T_avg), and the displacement values ​​between the peak and trough.

[0031] Step 2: Constructing multi-parameter joint decision conditions; The first condition, namely the displacement condition, is that the current instantaneous displacement D(t) must be within the high (e.g., 15%) or low (e.g., 85%) range of the dynamic interval formed by the peak and valley displacements within its period, ensuring that the needle tip is near the two extreme positions of the respiratory motion. The second condition is that the absolute value of the current instantaneous velocity V(t) must be less than an adaptive velocity threshold V_threshold. This threshold is not fixed, but dynamically calculated based on the average amplitude and average period duration of the most recent N cycles. ,in, This is an empirical coefficient, which is set to 0.1 in the context of liver biopsy.

[0032] The third condition is that the absolute value of the current instantaneous acceleration A(t) must be less than a small acceleration threshold A_threshold. This condition ensures that the breathing motion is not only slow but also in a smooth "turning" phase, excluding moments when the velocity is zero but about to accelerate. The acceleration threshold A_threshold is set as a proportion of the dynamic velocity threshold V_threshold, as shown in the formula: V_threshold; where It is another empirical coefficient. This can be determined through retrospective analysis of clinical data; its physical meaning can be understood as "the permissible change in velocity per unit time," with a typical initial value range possibly around 0.5. Up to 2 between.

[0033] Step 3: Continuity and Stability Verification: When the above three conditions are met for the first time, the system does not immediately determine that the resting period has started. Instead, it initiates a short verification period (e.g., 100-300 milliseconds). During this verification period, the above combined conditions must be continuously met in order to finally trigger the "resting period start" signal. This effectively prevents false triggering caused by signal jitter.

[0034] Step 4: Output and target locking of the resting period window: Once the "resting period begins" signal is confirmed, the system outputs the respiratory resting period to the doctor's interface and the robot control system.

[0035] In some embodiments, Figure 3 This is a flowchart of another real-time motion compensation robot puncture control method based on respiratory prediction according to an embodiment of the present invention, as shown below. Figure 3 As shown, determining the target location of liver lesions based on liver ultrasound images and liver lesion template images specifically includes the following process: Step S301: Extract image features from liver ultrasound images and liver lesion template images to obtain contrast features and template features; Liver ultrasound images Liver lesion template image ,in, Represents the set of real numbers. , Images ,image height, , Images ,image The width is 3, where 3 represents the three color channels of the image (RGB). Will and Each image is split into multiple non-overlapping image patch sequences, denoted as follows: , ,in, , These are the dimensions of the image patches, The size of each image patch is represented, all image patches are flattened, and the embedding representation of the liver ultrasound image is obtained through a linear mapping function. Embedded representation of liver lesion template images ,in, , The length of the embedded representation, , For the dimension of the embedded representation, and The input is fed into the Transformer encoding layer for feature extraction. The extracted contrast features and template features are: , .

[0036] Step S302: Perform feature interaction comparison processing on the comparison features and template features to obtain reference features after feature interaction comparison processing; The process of performing cross-feature comparison processing on the contrast features and template features to obtain reference features after cross-feature comparison processing includes the following steps: based on and Generate graph structure ,in, This is a set of nodes representing the characteristic information of liver lesions in a liver ultrasound image. , For the number of nodes, This refers to the edge set representing the feature information of liver lesions in the liver lesion template image. The number of sides is ; compute nodes The size of the d-th eigenvalue : ; in, For nodes Features Represents a node Degree centrality; based on Calculate edges Importance probability : ,in, , The range of values ​​is , This indicates the operation of finding the maximum value. ; in, , for The maximum value, for The average value, The probability limit threshold is set to 0.95 in this embodiment.

[0037] Importance probability Edges with an importance probability below a preset threshold in the graph structure Deleting the graph results in a graph structure that has undergone information interaction processing. For graph structures that have undergone information interaction processing Re-extract features to obtain reference features: the graph structure after information interaction processing. The input is fed into the Transformer encoding layer, which uses a multi-head self-attention mechanism to capture the correlation information within the features and outputs reference features.

[0038] The Transformer encoding layer includes an input projection layer, whose core function is to transform the graph structure. The node features are mapped to the dimensions adapted by the Transformer encoding layer to solve the problem of incompatibility between graph structure features and Transformer input dimensions. The input dimension is set to 192 and the output dimension is set to 384. The pre-layer normalization module uses layer normalization, and the normalization dimension is the feature dimension. The residual connection adds the output of the multi-head self-attention mechanism to the output of the pre-layer normalization module element-wise, and the output dimension is still 384. The post-layer normalization module has the same structure as the pre-layer normalization module.

[0039] Step S303: Determine the location of liver lesion target points in liver ultrasound images based on reference features.

[0040] Specifically, in step one, the reference features are... Mapped to an embedded representation of the liver ultrasound image via a fully connected layer. Consistent dimensions, resulting in mapped reference features ,calculate and Embedded features for each image patch The cosine similarity is used to filter image patches whose cosine similarity is greater than a cosine similarity threshold, forming a set of image patches associated with lesions. The cosine similarity threshold ranges from 0.7 to 0.85.

[0041] Step two, based on the non-overlapping segmentation rules of the ultrasound image blocks described above, divide... Each lesion-associated image patch is mapped back to the pixel coordinate system of the original liver ultrasound image to obtain the diagonal pixel coordinates of each lesion-associated image patch; the boundary extrema of the coordinates of all lesion-associated image patches are calculated to determine the coarse localization rectangular region of the lesion. ; Specifically, the diagonal pixel coordinates include the top-left pixel coordinates. and bottom right pixel coordinates Calculate the boundary extreme values ​​of the coordinates of all image blocks associated with the lesions: the coordinates of the upper left corner of the coarse localization region. ,in, This is the set of x-coordinates of the top-left pixel of the image patch associated with the lesion. This is the set of the top-left pixel coordinates of the image patch associated with the lesion; min is the minimum value operation; and the bottom-right coordinates of the coarsely located region are... , This is the set of x-coordinates of the lower right pixel of the image patch associated with the lesion. This is the set of bottom-right pixel coordinates of the image patch associated with the lesion, where `max` is the maximum value to determine the coarse rectangular region for lesion localization. This area contains the core extent of the lesion.

[0042] Step 3: Call the liver anatomy template to... Structural matching was performed on ultrasound images within the region, excluding non-lesion structural regions with a difference score below 0.6, to obtain a fine-grained candidate region for lesions. ;right After denoising with a 3×3 Gaussian filter, the edges are extracted using the Canny edge detection algorithm. The gaps are then filled using morphological closing operations on 5×5 rectangular structuring elements to obtain a complete and detailed contour of the lesion region. In the Canny edge detection algorithm, the high threshold setting range is 80-100, and the low threshold setting range is 40-60. Specifically, calculation The degree of difference between the inner pixels and the features of the liver anatomy template "normal liver tissue" and "vascular structure": Will The region is traversed using a 3×3 pixel sliding window. The center pixel of each window is used as the target pixel to extract the pixel set features within the window (completely consistent with the feature types of the liver anatomical structure template): grayscale features: mean grayscale value, variance, and histogram peak value of pixels within the window; texture features: GLCM entropy, contrast, correlation, and energy of pixels within the window; if the template is a vascular structure, additional morphological features of pixels within the window are extracted (e.g., whether there is a slender distribution, the number of local branch points). The extracted target pixel features are normalized in the [0,1] interval to obtain... The pixel feature vector F1 within the region is assigned feature weights: grayscale features account for 40% (mean 20%, variance 15%, peak 5%), texture features account for 60% (entropy 20%, contrast 20%, correlation 10%, energy 10%), and vascular morphological features account for an additional 30% (slenderness ratio 20%, branch point density 10%). Single feature differences are calculated: for each feature dimension, absolute difference normalization is used to calculate the difference. The formula is: ,in, Let i be the feature value corresponding to the i-th feature dimension. Let be the template feature value corresponding to the i-th feature dimension, where one feature dimension corresponds to one template feature. Then, let all ... Multiply the corresponding feature weights and sum all the resulting product values ​​to obtain the feature difference score. Regions with a feature difference score below 0.6 are excluded (i.e., non-lesion structures such as blood vessels and capsules are excluded); regions with a difference score of 0.6 or higher are retained to obtain the fine-grained candidate lesion regions.

[0043] Step 4: Calculate using the geometric center method The center coordinates of the minimum bounding rectangle The weighted center coordinates were calculated using the density-weighted method. The location of the liver lesion target point is obtained by calculating the mean of the two center coordinates. , .

[0044] Specifically, calculation is performed using the geometric center method. The center coordinates of the minimum bounding rectangle : Obtain the complete fine outline of the lesion region Complete set of pixel coordinates Where m represents the fine contour of the lesion region. The number of complete pixel coordinates, traversed Given all pixel coordinates, solve for the boundary extrema along the x and y axes: Minimum value in the x-axis direction: (x-coordinate of the leftmost pixel of the outline); Maximum value in the x-axis direction: (x-coordinate of the rightmost pixel of the outline); Minimum value in the y-axis direction: (x-coordinate of the leftmost pixel of the outline); Maximum value in the y-axis direction: (x-coordinate of the leftmost pixel of the outline); The rectangle formed by the above extreme values ​​is... Find the minimum bounding rectangle, and calculate the center coordinates of the minimum bounding rectangle based on the boundary extrema: , .

[0045] The coordinates of the weighting center are calculated using the density-weighted method. : fine contour of the lesion area Perform internal filling processing to obtain the coordinate set of all pixels enclosed by the contour (including contour pixels and internal pixels), denoted as . b is Total number of pixels inside, For the b-th pixel; The RGB value of each pixel is converted to a grayscale value using a weighted average method. ;in, Let be the grayscale value of the i-th pixel. , and These are the red, green, and blue channel values ​​for the i-th pixel, respectively; The grayscale values ​​of all pixels are inversely normalized and mapped to the weight range [0,1] to eliminate the influence of differences in grayscale value ranges. The normalization formula is as follows: ;in, The maximum grayscale value within the region. Minimum grayscale value within the region Let be the normalized grayscale value of the i-th pixel; Weights are assigned based on normalized gray values. : ;in, This is the weight amplification factor (with a value of 1.2-1.5 to balance weight differences and calculation stability). The base weight is set to 0.1 to avoid information loss due to some high grayscale pixels having a weight of 0.

[0046] Calculate the weighted center coordinates based on the pixel coordinates and their corresponding weights. : ; ;in, Let b be the total weight of the pixels.

[0047] In some embodiments, based on historical respiratory cycle data and real-time respiratory phase data, a dynamic trajectory model of liver movement during respiration is constructed using a motion prediction algorithm to predict the liver displacement trajectory in subsequent respiratory cycles. This process specifically includes the following steps: Data preprocessing and feature alignment: Collect complete data of the most recent N historical respiratory cycles (N≥5, N is a positive integer). The historical respiratory cycle data includes respiratory displacement data, instantaneous velocity data, instantaneous acceleration data, peak displacement, valley displacement and cycle duration features of each cycle; Based on the peak point of the respiratory cycle, align all historical respiratory cycle data on the time axis and divide each cycle into four phase intervals: inspiratory phase, end-inspiratory resting phase, expiratory phase and end-expiratory resting phase.

[0048] Liver motility-related feature extraction: Based on the location of liver lesions determined during the respiratory resting phase, the displacement coordinates of liver targets in different phase intervals of each historical respiratory cycle are extracted using image tracing technology. (t is the time node within the cycle), establish a mapping relationship library of "respiratory phase-liver displacement"; calculate the dynamic characteristics of liver displacement in each historical cycle, including displacement amplitude, average movement velocity, acceleration change rate, and displacement similarity between adjacent cycles in each phase interval, screen stable historical cycle data with similarity ≥ 0.8 to form a training dataset, and remove abnormal respiratory cycles (such as cycle mutations caused by coughing or deep breathing).

[0049] Dynamic trajectory model construction and training: A bidirectional LSTM (Long Short-Term Memory) network is used as the core of the motion prediction algorithm, constructing a three-layer network structure: the input layer has a dimension of M (M=9, containing dynamic features of respiratory displacement, instantaneous velocity, and instantaneous acceleration), the hidden layer has 256 neurons, and the output layer has a dimension of 2 (corresponding to the x and y displacement coordinates of the liver). The training dataset is divided into training and validation sets in a 7:3 ratio. The mean squared error (MSE) is used as the loss function, and the Adam optimizer (learning rate set to 0.001) is used to train the model. Iterative training is performed until the validation set loss function converges (convergence threshold set to 10). -4 This yields an initial liver motion trajectory model.

[0050] Real-time data fusion and dynamic model correction: Phase data of the current respiratory cycle is acquired in real time, and features are extracted according to preprocessing rules to obtain real-time respiratory feature vectors. These real-time respiratory feature vectors are input into an initial trajectory model, which outputs a predicted displacement sequence of the liver within the current cycle. Simultaneously, the actual displacement coordinates of the liver are captured in real time using ultrasound images, and the deviation between the predicted and actual displacements is calculated. : ;in, This represents the actual displacement along the x-axis. This represents the actual displacement along the y-axis. This represents the predicted displacement along the x-axis. This is the predicted displacement along the y-axis. If... For distances >0.5mm, an online adaptive learning mechanism is used to correct model parameters: Real-time respiratory feature vectors and actual displacement coordinates are used as new training samples to fine-tune the hidden layer weights (learning rate reduced to 0.0001, single iteration), updating to obtain a dynamic trajectory model adapted to the current patient's respiratory state; if ≤0.5mm, keep the model parameters unchanged.

[0051] Based on the modified dynamic trajectory model, input the complete feature data of 3-5 consecutive respiratory cycles that have been collected, and predict the liver displacement trajectory at each time point (time step set to 10ms) in the next 1-2 complete respiratory cycles.

[0052] In some embodiments, determining the optimal puncture path based on the predicted liver displacement trajectory and the target location of the liver lesion, and driving the robot to perform puncture based on the optimal puncture path specifically includes the following process: Puncture path constraint settings: Call the three-dimensional anatomical structure template of the liver (including the coordinate information of key obstacle avoidance structures such as portal vein, hepatic vein, gallbladder, etc.) and set path constraint rules: the shortest distance between the puncture path and the obstacle avoidance structure is ≥3mm, the puncture angle (angle with the normal of the liver surface) is in the range of 0°-60°, and the puncture depth does not exceed ±1cm of the lesion center; at the same time, combined with the range of motion of the robot puncture arm (joint rotation ±90°, extension stroke 0-200mm), the physical reachability boundary of the path is limited; Dynamic target coordinate mapping: Based on the output liver displacement trajectory, the dynamic target coordinates of the liver at each time node (time step 10ms) in the subsequent respiratory cycle are extracted, and a "time-dynamic target coordinate" mapping table is established to clarify the real-time position change pattern of the target during the puncture process. Optimal puncture path planning: Starting from the initial positioning point of the robotic puncture arm (puncture start point, determined by the doctor based on the patient's position), and ending at the coordinates of each dynamic target point in the mapping table, the A* path planning algorithm (with the heuristic function set as a weighted sum of Euclidean distance and obstacle avoidance cost, with a weight ratio of 1:2) is used to calculate the feasible puncture path corresponding to each time node. The path with the shortest path length, lowest obstacle avoidance cost, and satisfaction of the constraints is selected as the optimal puncture path, which includes the time-position sequence of the entire puncture process. ( As a time node, (Coordinates of the corresponding puncture needle tip position). Puncture motion command generation: The time-position sequence of the optimal puncture path is converted into joint motion commands of the robotic puncture arm to ensure that the puncture needle tip moves along the planned path and that the speed is equal in magnitude and direction to the liver displacement speed (achieving dynamic synchronization compensation); the command update frequency is consistent with the respiratory signal acquisition frequency (50-100Hz) to ensure motion synchronization. Real-time synchronous puncture execution: After receiving the motion command, the robot starts the puncture. The force sensor installed on the puncture arm monitors the puncture resistance in real time, and the optical positioning sensor at the needle tip (positioning accuracy ±0.1mm) provides feedback on the actual needle tip position.

[0053] Figure 4 This is a flowchart illustrating a real-time motion compensation robot puncture control method based on respiratory prediction, according to an embodiment of the present invention. Figure 4 As shown, this puncture control method includes the following procedures: Patient respiratory signal perception Starting point: Respiratory motion signals are continuously acquired through respiratory sensors attached to the patient's chest and abdomen.

[0054] Output: Real-time respiratory phase data (such as peak inspiratory time, end-expiratory time, etc.) are obtained, which serves as the timing reference for the entire system.

[0055] respiratory resting period target calibration Trigger: The system identifies the "resting period" with the smallest respiratory movement amplitude based on the real-time respiratory phase.

[0056] Imaging and Calibration: During the brief, stable resting period, the system triggers ultrasound imaging to acquire a clear image of the liver. Based on this image, the physician or the system precisely calibrates the location of the target liver lesion. This location serves as the "static" target reference for the puncture.

[0057] Liver motility prediction model Learning and Modeling: The system uses historical respiratory cycle data and real-time respiratory phases to learn through motion prediction algorithms.

[0058] Prediction: The algorithm constructs a dynamic trajectory model of the liver's movement with respiration, and uses this model to predict the real-time displacement trajectory of the liver (including the target site) in the next one or several respiratory cycles.

[0059] Robot path planning and control Information fusion: The system fuses the calibrated static target location with the predicted future liver displacement trajectory. This means the system knows where the target is now and where it will move to in the next second.

[0060] Path calculation and execution: Based on the fused dynamic information, an optimal puncture path is calculated that can compensate for respiratory movements and ensure accurate intersection of the needle tip and the moving target.

[0061] Final action: Based on this path, the control system drives the robot (robotic arm) to perform a precise puncture action in real time, ultimately completing the surgery.

[0062] It is worth noting that the above formulas are all dimensionless calculations, and the preset parameters in the formulas should be set by those skilled in the art according to the actual situation.

[0063] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0065] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time motion compensated robot puncture control method based on respiratory prediction, characterized in that, The method comprises: acquiring breathing phase data of a patient based on a chest and abdomen respiration sensor to obtain breathing phase data, and determining a breathing rest period based on the breathing phase data; acquiring an ultrasound image of a liver of the patient in the breathing rest period and a liver lesion template image, and determining a liver lesion target position based on the ultrasound image of the liver and the liver lesion template image; based on historical breathing cycle data and real-time breathing phase data, constructing a dynamic trajectory model of the liver with respiratory motion through a motion prediction algorithm to predict a liver displacement trajectory in a subsequent breathing cycle; determining an optimal puncture path based on the predicted liver displacement trajectory and the liver lesion target position, and driving a robot to puncture based on the optimal puncture path.

2. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 1, characterized in that, Determining the breathing rest period based on the breathing phase data specifically comprises the following processes: acquiring breathing phase data of a patient, wherein the breathing phase data at least includes breathing displacement data, instantaneous speed data derived from the displacement data, and instantaneous acceleration data derived from the displacement data; based on the breathing displacement data, identifying peak points and valley points in the breathing cycle to segment continuous breathing cycles; based on the breathing motion characteristics of the current cycle or the most recent historical cycle, dynamically calculating a decision threshold, wherein the decision threshold is a speed threshold dynamically calculated based on the ratio of the average amplitude to the average cycle length of the last N historical breathing cycles; determining whether the breathing phase data at the current time meets the following conditions simultaneously: a first condition: the breathing displacement data is in a high range or a low range of a dynamic interval defined by the peak displacement and the valley displacement of the current breathing cycle, the high range is defined as an interval extending downward from the peak displacement by a first predetermined proportion, and the low range is defined as an interval extending upward from the valley displacement by a second predetermined proportion; a second condition: the absolute value of the instantaneous speed data is less than the dynamically calculated speed threshold; a third condition: the absolute value of the instantaneous acceleration data is less than a preset acceleration threshold; when the first condition, the second condition and the third condition are met simultaneously, triggering a persistent verification mechanism: continuously monitoring the breathing phase data within a predetermined verification period, and only when the first, second and third conditions are continuously met within the verification period, finally determining that the current is in the breathing rest period and outputting a corresponding determination signal. 3.The real-time motion compensated robotic puncture control method based on respiratory prediction of claim 1, wherein, Determining the liver lesion target position based on the ultrasound image of the liver and the liver lesion template image specifically comprises the following processes: performing image feature extraction on the ultrasound image of the liver and the liver lesion template image to obtain contrast features and template features; performing feature interaction comparison processing on the contrast features and the template features to obtain reference features after feature interaction comparison processing; determining the liver lesion target position in the ultrasound image of the liver based on the reference features.

4. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 3, characterized in that, Performing image feature extraction on the ultrasound image of the liver and the liver lesion template image to obtain contrast features and template features specifically comprises the following processes: Liver ultrasound images Liver lesion template image ,in, Represents the set of real numbers. , Images ,image height, , Images ,image The width is 3, where 3 represents the three color channels of the image (RGB). Will be split into a plurality of non-overlapping image block sequences respectively, denoted as and respectively. , wherein, , is the dimension number of the image block respectively. , represents the size of each image block, and the flattened processing of all image blocks is performed to obtain the embedding representation of the liver ultrasound image and the embedding representation of the liver lesion template image respectively through a linear mapping function. , is the length of the embedding representation. , is the dimension of the embedding representation. and are input into the Transformer encoding layer for feature extraction, and the extracted contrast features and template features are , .

5. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 4, characterized in that, performing feature interaction comparison processing on the contrast features and the template features to obtain reference features after feature interaction comparison processing comprises the following processes: Based on And Generate a graph structure Wherein, The node set representing the liver lesion feature information of the liver ultrasound image is a node set , The number of nodes is The edge set representing the liver lesion feature information in the liver lesion template image is an edge set , and the number of edges is ; Computing node In the size of the d-th eigenvalue : ; wherein, is a feature of a node , denotes the degree centrality of a node . Based on The importance probability of the edge : wherein, , The value range of each of the importance probability of the edge , represents a maximum value operation, ;​ wherein, , is the maximum value of, is the average value of, is a probability limit threshold value; importance probability edges with importance probabilities lower than a preset importance probability threshold are deleted from the graph structure obtained after information interaction processing the graph structure obtained after information interaction processing re-performs feature extraction to obtain reference features 6. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 5, characterized in that, determining the liver lesion target position in the ultrasound image of the liver based on the reference features specifically comprises the following processes: Step 1, refer to the features Mapped to an embedded representation of the liver ultrasound image via a fully connected layer. Consistent dimensions, resulting in mapped reference features ,calculate and Embedded features for each image patch The cosine similarity is used to filter image patches whose cosine similarity is greater than a cosine similarity threshold, forming a set of image patches associated with lesions. ; Step two, according to the non-overlapping segmentation rule of the ultrasound image block, the ultrasound image block is segmented into a plurality of sub-image blocks Each lesion associated image block is mapped back to the pixel coordinate system of the original liver ultrasound image to obtain the diagonal pixel coordinates of each lesion associated image block; the boundary extreme value of the coordinates of all lesion associated image blocks is calculated to determine the rough positioning rectangular region of the lesion ; Step three, call liver anatomy template to The ultrasound image in the region is matched with the structure, non-lesion structure regions with a difference less than 0.6 are excluded, and a fine lesion candidate region is obtained ; After 3x3 Gaussian filter denoising , the edges are extracted by using the Canny edge detection algorithm, the gaps are filled by morphological closing operation of a 5x5 rectangular structural element, and a complete fine lesion region contour is obtained ; Step four, calculate the weighted center coordinate by density weighted method The center coordinate of the minimum circumscribed rectangle , calculate the weighted center coordinate by density weighted method , calculate the mean value of the two center coordinates to obtain the target position of the liver lesion , .

7. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 1, characterized in that, Based on historical respiratory cycle data and real-time respiratory phase data, a dynamic trajectory model of liver movement with respiration is constructed through a motion prediction algorithm, and the liver displacement trajectory in the subsequent respiratory cycle is predicted, which specifically includes the following processes: Collect complete data of the last H historical respiratory cycles, including respiratory displacement data, instantaneous velocity data, instantaneous acceleration data, peak displacement, valley displacement and cycle length characteristics of each cycle; take the peak point of the respiratory cycle as the reference, align all historical respiratory cycle data on the time axis, and divide each cycle into four phase intervals: inspiration, inspiration end rest, expiration and expiration end rest; Based on the target position of the liver lesion determined in the respiratory rest period, the liver target displacement coordinates in different phase intervals of each historical respiratory cycle are extracted through image tracing technology, and a "respiratory phase-liver displacement" mapping relationship library is established; the historical cycle data is used to form a training data set; A bidirectional LSTM network is used as the core of the motion prediction algorithm, the training data set is divided into training set and validation set in a 7:3 ratio, the mean square error MSE is used as the loss function, and the Adam optimizer is used for iterative training until the validation set loss function converges, to obtain an initial liver motion trajectory model; The initial liver motion trajectory model data is fused and the model is dynamically corrected to obtain a corrected dynamic trajectory model, and real-time respiratory phase data is input into the corrected dynamic trajectory model to output the liver displacement trajectory in the subsequent respiratory cycle.

8. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 1, characterized in that, The structure of the bidirectional LSTM network includes: the input layer dimension is 9, the hidden layer is set to 256 neurons, and the output layer dimension is 2. 9.The real-time motion compensated robotic puncture control method based on respiratory prediction of claim 1, wherein, Driving the robot to puncture based on the optimal puncture path specifically includes the following processes: Puncture path constraint condition setting: call the liver three-dimensional anatomical structure template, set the path constraint rule: the shortest distance between the puncture path and the obstacle structure is greater than or equal to 3mm, set the puncture angle range, and the puncture depth is not more than 1.2 times the vertical distance from the lesion center to the liver surface; at the same time, combined with the motion range of the robot puncture arm, the physical reachable boundary of the path is limited; Dynamic target point coordinate mapping: based on the output liver displacement trajectory, the liver dynamic target point coordinates at each time node in the subsequent respiratory cycle are extracted, a "time-dynamic target point coordinate" mapping table is established, and the real-time position change rule of the target point in the puncture process is determined; Optimal puncture path planning: taking the initial positioning point of the robot puncture arm as the starting point and the dynamic target point coordinates in the mapping table as the ending point, the A* path planning algorithm is used to calculate the feasible puncture path corresponding to each time node; the path with the shortest length, the lowest obstacle avoidance cost and meeting the constraint conditions is selected as the optimal puncture path; Puncture motion instruction generation: the time-position sequence of the optimal puncture path is converted into joint motion instructions of the robot puncture arm to ensure that the puncture needle tip moves according to the planned path, and the speed and direction are the same as the liver displacement speed.

10. The real-time motion compensated robotic puncture control method based on respiratory prediction according to claim 9, wherein, The update frequency of the puncture motion instruction is consistent with the respiratory signal acquisition frequency to ensure the motion synchronization.

Images