Vision-based robot vascular suture force feedback estimation method, medium and system

Abstract

The invention belongs to the field of medical robots, and particularly relates to a robot blood vessel suturing force feedback estimation method, medium and system based on vision, and the method comprises the following steps: collecting and preprocessing a high-frequency dynamic image of a blood vessel region and a suturing tool; performing time-frequency analysis on the image; after the image is converted to a frequency domain, a motion mode of the cardiovascular system is extracted; constructing a mechanical model based on the extracted motion mode and a prior biomechanical model of the cardiovascular system; and combining the extracted motion modal information with the mechanical response of the mechanical model through a space-time information fusion algorithm, and estimating the contact force when the suture tool is in contact with the blood vessel. Compared with the prior art, the problem that a force feedback estimation method based on vision is difficult to apply in a dynamic environment in the prior art is solved. According to the scheme, more accurate force control and operation are achieved, and the safety and the success rate of the robot vascular suturing operation can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical robots, and particularly relates to a method, medium and system for force feedback estimation of robotic vascular suture based on vision. Background Art

[0002] During the process of robotic vascular suture, the complexity of the surgical environment remains a major challenge.

[0003] Firstly, the morphology and position of blood vessels require extremely high precision. Blood vessels usually present complex curved shapes and vary significantly among different patients. This requires the suture tool to be able to accurately locate and adjust the operating posture at any time to adapt to the different geometric shapes of blood vessels. However, the blood vessel wall itself is extremely fragile, and the force applied during the operation must be extremely delicate to avoid perforating or otherwise damaging the blood vessel. Excessive force may cause tearing or bleeding of the blood vessel wall, while too little force may result in insecure suturing, thereby affecting the surgical effect. Therefore, the suture tool needs to have a highly sensitive force feedback system to ensure precise control of the magnitude and direction of the force, especially in complex and delicate operations.

[0004] In addition, the dynamic factors during the operation make precise operation more difficult. For example, physiological phenomena such as the beating of the heart, the movement of breathing, and the pulsation of blood flow continuously change the surgical environment. Among them, blood vessels contract and expand periodically due to the beating of the heart, and the blood flow fluctuations further affect the shape and elasticity of blood vessels. At the same time, during the operation, the contact state between the suture tool and the blood vessel will also change at any time. These dynamic factors require a higher precision of operation. In this case, relying on traditional visual imaging technology and static image analysis often fails to meet the real-time and delicate force control requirements.

[0005] Currently, the force feedback technology in robotic vascular suture mainly relies on several methods. The most common method is to integrate a force sensor into the suture tool and use the sensor to detect the force when the tool contacts the blood vessel. This method can provide force feedback in real time. However, its application also has certain limitations. Firstly, the integration of the force sensor will increase the complexity of the suture tool, which may affect the flexibility of the tool, especially during fine operations. Secondly, the sensor may increase the weight or rigidity of the tool, affecting the comfort and precision of operation. In addition, the cost of the sensor itself is relatively high, increasing the overall cost of the system and limiting its wide application.

[0006] Another commonly used method is indirect force estimation, which calculates the contact force by analyzing the bending degree of the tool or the current of the driving motor. Although this method is easy to implement, its accuracy and reliability are still problematic. The change in the tool's bending degree is often limited by the elastic properties of the material and cannot directly reflect the actual contact force. Especially in a dynamic surgical environment, the change in bending may be affected by other factors. The change in motor current can only provide a rough estimate of the force and is difficult to accurately reflect the actual contact force between the suture tool and the blood vessel. Therefore, although the indirect force estimation method has certain application value, it cannot provide sufficient support for the precise control of force application.

[0007] There is also a method that relies on the operator's tactile feedback. Although this method has been widely used in traditional surgeries and can provide certain real-time feedback through the operator's intuition and experience, the limitations of this method are also very obvious. The accuracy of tactile feedback highly depends on the operator's experience and feel. Differences in tactile sensitivity among different operators may lead to inconsistent operation results, and tactile feedback is difficult to quantify, making it impossible to precisely control the magnitude and application direction of the force, resulting in poor repeatability of the operation. For robotic vascular suturing, relying on the operator's tactile feedback cannot meet the requirements of high precision and high consistency.

[0008] With the progress of visual force estimation technology, in recent years, researchers have begun to explore introducing this technology into robotic vascular suturing. Visual force estimation technology calculates the contact force between the operating tool and the tissue by using image sensors and computer vision algorithms. There have been some research progresses in the application of this technology in robotic-assisted surgery. For example, the SurgeMOD method proposed by the Liu Hongbin team achieved good results in laparoscopic surgery by combining visual information and force estimation; CN118279257A discloses a pure visual force estimation method based on deep learning. It recovers the material information from the collected ultrasound images and models the kinematic features using the optical flow map to achieve real-time force estimation; CN118866370A discloses a guide wire shape force estimation method for a vascular intervention surgical robot, which estimates the three-dimensional shape of the guide wire and the contact force with the blood vessel wall according to medical image information. However, successfully applying visual force estimation technology to robotic vascular suturing still faces many challenges. First, the shape of the blood vessel is complex and in dynamic change. Especially affected by physiological factors such as the heartbeat and breathing of the blood vessel, its three-dimensional movement and deformation make force estimation based on static images more difficult. The periodic contraction and dilation of the blood vessel, the pulsation of blood flow, and the movement of surrounding tissues will all lead to unstable force feedback and increase the difficulty of image analysis.

[0009] Secondly, existing visual imaging technologies, especially those used in robotic vascular suturing, usually struggle to provide sufficient detail and texture information, with relatively low image resolution, which affects the estimation of subtle contact forces. Due to the lack of sufficient detail, the exact position of the contact between the blood vessel and the tool in the image is difficult to distinguish, thereby affecting the accuracy of force feedback. In addition, existing imaging technologies, especially two-dimensional images, lack depth information, making the precise estimation of the three-dimensional contact situation between the tool and the blood vessel complex. At the same time, the minute changes in blood vessels, especially the flexible deformation of blood vessels during the operation, require precise depth information to support force estimation. Since two-dimensional imaging cannot directly capture the three-dimensional shape of blood vessels, the limitations of visual force estimation methods in this regard are also very significant.

[0010] Therefore, although visual force estimation technology shows good prospects in some medical surgery fields, especially in laparoscopic surgery, its effective application in robotic vascular suturing still requires overcoming numerous technical challenges. How to provide precise and real-time force feedback in a dynamic and complex surgical environment remains the focus of current research. Summary of the Invention

[0011] The purpose of the present invention is to provide a visual-based force feedback estimation method, medium, and system for robotic vascular suturing to solve at least one of the above problems, so as to address the issue that existing visual-based force feedback estimation methods are difficult to apply in dynamic environments, especially the lack of image details in vascular suturing. This solution combines more efficient imaging technologies, deep learning methods, and real-time feedback mechanisms to achieve more precise force control and operation, and can improve the safety and success rate of robotic vascular suturing.

[0012] The purpose of the present invention is achieved through the following technical solutions:

[0013] The first aspect of the present invention discloses a visual-based force feedback estimation method for robotic vascular suturing, including the following steps:

[0014] S1: Collect high-frequency dynamic images of the blood vessel area and the suturing tool, and preprocess the images;

[0015] S2: Perform time-frequency analysis on the preprocessed images; after converting the preprocessed images to the frequency domain, extract the motion modes of the cardiovascular system, and the motion modes include cardiac pulsation, respiratory movement, blood flow fluctuation, blood vessel elastic deformation, and contact deformation between the suturing tool and the blood vessel;

[0016] S3: Based on the extracted motion modes and the prior biomechanical model of the cardiovascular system, construct a mechanical model that considers the periodic motion, local deformation, hemodynamic effects of the blood vessel system, and mechanical interaction between the blood vessel and the suturing tool;

[0017] S4: Combine the extracted motion modality information with the mechanical response of the mechanical model through a spatio-temporal information fusion algorithm to estimate the contact force when the suture tool contacts the blood vessel.

[0018] Preferably, in step S1, the high-frequency dynamic image is obtained by X-ray and / or ultrasonic imaging, and the high-frequency dynamic image is an HDR image; the preprocessing includes one or more of denoising, artifact removal, contrast enhancement, illumination correction, and Gaussian filtering.

[0019] Preferably, in step S2, it includes one or more of the following:

[0020] i) The time-frequency analysis is performed by wavelet transform;

[0021] ii) The preprocessed image is transformed to the frequency domain through fast Fourier transform;

[0022] iii) Extracting the motion modality of the cardiovascular system is performed by multi-modal analysis, including principal component analysis and independent component analysis, and dynamically adjusting the parameters in the extraction process through an adaptive mode decomposition method.

[0023] Preferably, in step S3, it includes one or two of the following:

[0024] i) Adopt a multi-physics field coupling simulation technology based on the mechanical model to simulate the three-dimensional deformation and mechanical response of the blood vessel under dynamic conditions. The multi-physics field coupling simulation technology includes computational fluid dynamics and finite element analysis;

[0025] ii) Through the way of solving inverse problems, adjust and optimize the parameters of the mechanical model through experimental data feedback.

[0026] Preferably, in step S4, the spatio-temporal information fusion algorithm is: a joint model based on a deep convolutional neural network and a temporal recursive neural network; the estimation improves the accuracy of dynamic estimation through adaptive Kalman filtering, and online optimizes the mechanical model through recursive least squares method.

[0027] Preferably, the method further includes step S5:

[0028] Optimize the parameters of the mechanical model based on the data collected during the operation through an adaptive mechanism and / or autonomous learning; the adaptive mechanism and / or autonomous learning include deep reinforcement learning and meta-learning.

[0029] Preferably, in step S5, the parameters of the mechanical model are optimized according to the new data obtained during the operation through incremental learning technology.

[0030] Preferably, the method further includes step S6:

[0031] Dynamically adjust the mechanical model during the surgical process through multimodal information fusion; the multimodal information includes the collected images and force sensor data.

[0032] The second aspect of the present invention discloses a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the force feedback estimation method for vision-based robotic vascular suture as described in any one of the above.

[0033] The third aspect of the present invention discloses a force feedback estimation system for vision-based robotic vascular suture, including:

[0034] An image acquisition module for acquiring high-frequency dynamic images of the vascular region and the suture tool;

[0035] An algorithm analysis module for estimating the contact force when the suture tool contacts the blood vessel based on the high-frequency dynamic images acquired by the image acquisition module through the force feedback estimation method for vision-based robotic vascular suture as described in any one of the above;

[0036] A robot control module for feeding back the contact force estimated by the algorithm analysis module to the surgical robot.

[0037] The force feedback estimation method proposed by the present invention:

[0038] 1. Multi-scale coupling analysis: By combining multi-physics field simulation and finite element analysis, accurately model the complex mechanical behavior of the blood vessel, and fuse global and local features through a multi-scale method to improve the accuracy of force estimation.

[0039] 2. Spatiotemporal fusion analysis: Adopt a spatiotemporal fusion analysis method to enhance the processing ability of dynamic complex motion patterns, especially for the estimation of blood vessel deformation and the contact force of the suture tool.

[0040] 3. Adaptive learning and dynamic optimization: Through deep reinforcement learning and meta-learning mechanisms, the system can optimize the parameters of the mechanical model in real time, thereby continuously improving the accuracy and adaptability of force estimation.

[0041] 4. Multimodal information fusion: Comprehensively consider image, mechanical and physiological data to provide comprehensive decision-making support, and further enhance the stability and adaptability of the system.

[0042] 5. Real-time feedback and precise control: Adopt a hybrid control strategy based on deep learning and model predictive control (for surgical robots) to ensure high-precision operation of the suture tool in a complex environment.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. High-precision force estimation: It can estimate the contact force between the suture tool and the blood vessel in real time and accurately, providing precise tactile feedback for the operator, and greatly improving the precision and safety of the surgery.

[0045] 2. Strong adaptability: It can adaptively adjust according to the individual differences of different patients and the surgical stages, improving the adaptability of the model.

[0046] 3. Real-time dynamic optimization: Through the adaptive learning mechanism, continuously optimize the parameters of the mechanical model to ensure that the force estimation accuracy is always in the optimal state.

[0047] 4. Wide applicability: This method is not only applicable to robotic vascular suturing, but can also be extended to other high-precision minimally invasive surgery fields, providing strong technical support for various robot-assisted surgeries. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic structural diagram of a force feedback estimation system for vision-based robotic vascular suturing;

[0049] Figure 2 is a schematic flow diagram of a force feedback estimation method for vision-based robotic vascular suturing. DETAILED DESCRIPTION OF THE INVENTION

[0050] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0051] For matters not described below, the prior art can be adopted.

[0052] Embodiment 1

[0053] A force feedback estimation method for vision-based robotic vascular suturing adopts a multi-level, multi-modal, multi-scale algorithm framework, integrating advanced image processing, mechanical modeling, and deep learning technologies, aiming at high precision, high adaptability, and high real-time performance to improve the force estimation ability in robotic vascular suturing. The specific method is as Figure 2 shown, including the following steps:

[0054] S1. Image acquisition and preprocessing

[0055] A high - frame - rate imaging system, such as an ultra - high - speed multi - dimensional X - ray or multi - modal ultrasound imaging system, is used to collect high - frequency dynamic images of the vascular region and the suturing tool in real - time. During the data acquisition process, high - dynamic - range imaging (HDR) technology is combined to further capture fine tissue features. The collected images are then pre - processed by one or more of the operations such as denoising, artifact removal, contrast enhancement, and illumination correction, and the noise in the low - quality images is removed through the Gaussian filtering algorithm to ensure that the image data has the best quality.

[0056] In this embodiment, to collect high - frequency dynamic images of the vascular region and the suturing tool, a high - frame - rate imaging system is preferably used, such as the X - ray fluoroscopy system commonly used in clinics (such as the GE Innova series or Siemens Artis series) or the multi - modal ultrasound imaging system (such as Philips EPIQ 7 or GE LOGIQ E10). The acquisition frequency of these systems is set to 15 - 50 FPS according to the device performance, which can meet the real - time imaging requirements. During the data acquisition process, high - dynamic - range imaging (HDR) technology is combined to enhance the dynamic range of the image by adjusting the exposure parameters or multi - frame synthesis, and further capture the fine tissue features of the blood vessels and the suturing tool (such as blood vessel wall texture and tool edge). The collected images are then pre - processed by one or more of the operations including denoising, artifact removal, contrast enhancement, and illumination correction. Specifically, Gaussian filtering (standard deviation σ = 0.5) can be used for denoising to remove high - frequency noise, eliminate the noise and artifacts in the low - quality images, ensure that the image data has the best quality, and provide a reliable basis for subsequent analysis.

[0057] S2. Multi - domain motion pattern analysis and mode extraction

[0058] Time - frequency analysis is performed on the pre - processed images through multi - scale wavelet transform to extract the high - frequency motion features in the images.

[0059] After converting the image to the frequency domain using the fast Fourier transform (FFT), a multi - modal analysis method combining traditional principal component analysis (PCA) and independent component analysis (ICA) is used to further extract the motion modes of the cardiovascular system, including cardiac pulsation, respiratory motion, blood flow fluctuation, blood vessel elastic deformation, and the contact deformation between the suturing tool and the blood vessel. To adapt to the physiological differences of different patients and the complexity of the surgical stage, the present invention introduces an adaptive mode decomposition method (AMD), which automatically identifies and extracts various motion modes by dynamically adjusting the parameters in the mode extraction algorithm, thereby improving the flexibility of the method and its adaptability to individual differences.

[0060] In this embodiment, time-frequency analysis is performed on the preprocessed image. Preferably, a multi-scale time-frequency analysis method based on wavelet transform is adopted to extract the high-frequency motion features in the image. Wavelet transform is a classical signal processing technology that hierarchically analyzes the image signal through multi-level decomposition, and can effectively separate high-frequency motion features (such as the pulsation of blood vessels and blood flow fluctuations) and low-frequency background components. Specifically, this method uses wavelet basis functions (such as Daubechies or Symlets wavelets) to decompose the image, and extracts features in different frequency ranges step by step, such as cardiac pulsation (0.5 - 2 Hz), respiratory motion (0.1 - 0.5 Hz), and blood flow fluctuations (1 - 5 Hz). The image is then transformed into the frequency domain through fast Fourier transform (FFT), and combined with principal component analysis (PCA) and independent component analysis (ICA) to further extract the motion modes of the cardiovascular system, including cardiac pulsation, respiratory motion, blood flow fluctuations, elastic deformation of blood vessels, and contact deformation between the suture tool and the blood vessel. To enhance the adaptability of the method, the present invention introduces adaptive mode decomposition (AMD), and optimizes the mode extraction process by dynamically adjusting decomposition parameters (such as the number of decomposition levels and frequency thresholds).

[0061] S3. High-precision mechanical model construction and multi-scale coupling analysis

[0062] Based on the extracted motion modes and the prior biomechanical model of the cardiovascular system, a multi-scale and high-dimensional image space mechanical model is constructed. This model takes into account the periodic motion of the vascular system, local deformation, hemodynamic effects, and the mechanical interaction between blood vessels and suture tools. On this basis, multi-physics field coupling simulation technology is adopted, combined with computational fluid dynamics (CFD) and finite element analysis (FEA), to accurately simulate the three-dimensional deformation and mechanical response of blood vessels under different motion modes. To further improve the accuracy of the model, the present invention also integrates inverse problem solving technology, and adjusts and optimizes the parameters of the mechanical model in real time through experimental data feedback.

[0063] Among them, multi-scale means that the model simultaneously considers the macroscopic scale (the periodic motion of the vascular system and hemodynamic effects, such as the overall displacement caused by cardiac pulsation) and the microscopic scale (local deformation and the contact force between the suture tool and the blood vessel, such as the elastic deformation of the blood vessel wall); through multi-scale analysis, global and local features are integrated to improve the comprehensiveness of force estimation. Multi-dimensional means the multi-dimensionality of the model input and output, including three-dimensional image data (spatial coordinates x, y, z), time-series motion modes (time dimension t), and biomechanical parameters (such as elastic modulus, blood flow velocity, shear stress), forming a comprehensive mechanical response space.

[0064] S4. Motion-mechanics multi-modal fusion and real-time force estimation

[0065] Through a multi-channel spatio-temporal information fusion algorithm (e.g., a joint model based on a deep convolutional neural network (CNN) and a temporal recurrent neural network (RNN)), the extracted image modality information is combined with the dynamic response of the mechanical model to estimate the contact force in real time when the suture tool contacts the blood vessel. This algorithm combines the feature extraction ability of deep learning with the high accuracy of physical modeling, and further enhances the processing ability of complex blood vessel motion patterns through spatio-temporal fusion analysis. During the real-time estimation process, the present invention adopts a Kalman Filtering algorithm based on an adaptive time window to improve the accuracy of dynamic estimation, and online optimizes the mechanical model by the Recursive Least Squares (RLS) method to ensure that the force estimation is always in an optimal state.

[0066] Specifically, in this embodiment, the spatio-temporal information fusion algorithm is a joint model based on a deep convolutional neural network (CNN) and a temporal recurrent neural network (RNN). Specifically, the CNN is used to extract spatial features from high-frequency dynamic images, such as blood vessel boundaries, suture tool positions, and local deformations; the RNN (preferably a long short-term memory network LSTM) is used to process temporal features, such as the periodic pulsation of blood vessels, blood flow fluctuations, and contact deformation dynamics. The force estimation process improves the accuracy of dynamic estimation through an Adaptive Kalman Filter, specifically by dynamically adjusting the process noise covariance and the measurement noise covariance to optimize the real-time tracking of the contact force. The parameters of the mechanical model are online optimized by the Recursive Least Squares (RLS) method to adapt to the changes in the blood vessel state during the operation. Subsequently, the estimated contact force is transmitted to the surgical robot through a feedback control algorithm, and a hybrid control framework combining Model Predictive Control (MPC) and a neural network is used to finely adjust the applied force and direction of the suture tool to ensure high-precision operation.

[0067] S5. High-efficiency force feedback and robot system control

[0068] The real-time estimated contact force is transmitted to the robot system (such as a surgical robot) through a feedback control algorithm for assisting surgical operations. This system combines an advanced real-time feedback mechanism to finely adjust the applied force and direction of the suture tool through a Force Sensing Control strategy and Torque Control. To further optimize the response time and control accuracy during the operation, the present invention designs a hybrid control framework based on Model Predictive Control (MPC) and a neural network to achieve efficient control of the suture tool and ensure high-precision operation in different blood vessel morphologies and complex environments.

[0069] S6. Adaptive Learning Mechanism and Dynamic Optimization

[0070] The present invention also includes an adaptive learning mechanism that continuously optimizes the parameters of the mechanical model based on the data collected during the surgery. Through Deep Reinforcement Learning and Meta-Learning algorithms, it can not only improve the adaptability of the mechanical model but also continuously self-learn and adjust according to the specific physiological characteristics and surgical stages of each patient. In addition, the present invention also uses Incremental Learning technology to gradually optimize the mechanical model according to the new data obtained during the surgery, ensuring that the model continuously approaches the optimal state and improving the surgical accuracy and safety.

[0071] S7. Multi-modal Information Fusion and Global Decision Support

[0072] To further enhance the robustness and accuracy of the force estimation method, the present invention also introduces multi-modal information fusion technology, including image information, mechanical information, and physiological parameter information (such as heart rate, blood pressure, etc.). By combining a deep learning model based on images (e.g., Convolutional Neural Network CNN) with the mechanical model, this method can comprehensively consider all relevant information and thus provide global decision support. Through this decision-making mechanism, this method can not only achieve real-time force estimation but also automatically adjust the surgical strategy according to the current operating state, thereby providing scientific suggestions for the operator and improving the surgical efficiency and safety.

[0073] Embodiment 2

[0074] The robotic system mentioned in Embodiment 1 is as Figure 1 shown, and it includes an image acquisition module, an algorithm analysis module, a robot control module, and a surgical robot. Among them, the image acquisition module is used to acquire high-frequency dynamic images of the blood vessel area and the suture tool; the algorithm analysis module is used to execute the steps given in Embodiment 1 to estimate the contact force when the suture tool contacts the blood vessel; the robot control module is used to feedback the contact force estimated by the algorithm analysis module to the surgical robot and control the surgical robot according to the estimation result.

[0075] Embodiment 3: Force Estimation Method for Robotic Vascular Suturing Based on Image and Deep Learning

[0076] Background and Principle Explanation:

[0077] In this embodiment, a deep learning algorithm is combined with image processing technology to estimate the contact force between the catheter and the blood vessel during robotic vascular suture by analyzing the real-time acquired blood vessel image data. The system accurately estimates the contact force through a combination of high-frame-rate imaging, convolutional neural network (CNN), and long short-term memory network (LSTM), and provides real-time feedback to the robot.

[0078] Steps and parameters:

[0079] Image data acquisition and preprocessing: Use a GE LOGIQ P9 ultrasound device to acquire real-time blood vessel images. The imaging parameters are set as follows:

[0080] Imaging frequency: 30 FPS

[0081] Resolution: 1920×1080

[0082] Image processing: Remove high-frequency noise (Gaussian filtering, standard deviation of 0.5)

[0083] Contrast enhancement: Use CLAHE (Contrast Limited Adaptive Histogram Equalization)

[0084] Feature extraction using a deep convolutional neural network (CNN): After preprocessing, the image is input into a convolutional neural network for feature extraction. Use the ResNet-50 architecture, with the following specific settings:

[0085] Input size: 224×224

[0086] Convolutional layer: 64 convolutional kernels (3×3), stride of 2

[0087] Pooling layer: Max pooling, pooling window 3×3, stride of 2

[0088] Activation function: ReLU

[0089] After feature extraction, modal decomposition and data dimensionality reduction are performed. Modal decomposition and data dimensionality reduction are carried out on the high-dimensional features extracted by the CNN. The principal component analysis (PCA) and independent component analysis (ICA) methods are used, which are consistent with the multi-modal analysis in step S2 of the aforementioned embodiment 1. Specifically, PCA is used for dimensionality reduction, retaining the main feature components (usually the first 5-10 principal components); ICA is used for decomposing independent modes to further separate the relevant features of blood vessel movement (such as pulsation, deformation). Temporal feature extraction and LSTM modeling: The temporal features of blood vessel movement are extracted through an LSTM network. The configuration of the LSTM model is as follows:

[0090] Input: Extracted temporal features, time step of length 128

[0091] Hidden layer: 2 LSTM layers, with 256 neurons in each layer

[0092] Activation function: Tanh

[0093] Optimizer: Adam optimizer, learning rate set to 0.001

[0094] The LSTM network is used to capture the periodic and aperiodic motion characteristics of blood vessels.

[0095] Mechanical model and force estimation: Based on the extracted temporal features, combined with the finite element mechanical model, estimate the force of the catheter in contact with the blood vessel. Assume that the blood vessel model is an elastic thin film structure and use linear elasticity theory for modeling:

[0096] F = -λtr(ε)I - 2με

[0097] where F is the contact force, λ = 0.3MP and μ = 0.1MPa are the Lame constants of the blood vessel wall, ε is the strain tensor, and I is the unit matrix.

[0098] Force feedback and robot control: The estimated contact force is fed back to the robot control system (such as Intuitive Surgical Da Vinci Xi). The robot control system adjusts the catheter position and force according to the real-time feedback through the PID control algorithm to ensure the accuracy of the suture operation. Specifically, the contact force between the tool and the blood vessel is estimated through the spatio-temporal information fusion algorithm (a joint model based on CNN and LSTM). The real-time estimated contact force is transmitted to the robot system (such as Intuitive Surgical Da Vinci Xi) through the feedback control algorithm for assisting the surgical operation. To optimize the response time and control accuracy, the present invention adopts a hybrid control framework based on model predictive control (MPC) and neural network; specifically, MPC predicts the blood vessel movement and contact force changes according to the mechanical model and calculates the optimal control input (tool position and strength); the neural network corrects the prediction in real time to enhance the system's connection to complex environments. This framework ensures that the tool maintains the operation under different blood vessel morphologies and dynamic conditions (force < 0.05N, position error < 0.1mm).

[0099] Force estimation optimization and model adjustment: During the operation, the system performs online learning through real-time data and makes adaptive adjustments through reinforcement learning, using the Q-learning algorithm for optimization. Assume that the current contact force error is:

[0100] ∈(t) = F target -F estimated

[0101] The system continuously optimizes the contact force estimation model by updating the policy:

[0102]

[0103] Wherein:

[0104] s t : The current state, representing the characteristics of the surgical environment at the current moment, including the vascular motion mode (extracted from step S2, such as the pulsation frequency), the current contact force estimation value, and the image features (extracted from step S4).

[0105] a t : The current action, representing the parameter adjustment strategy of the force estimation model, such as adjusting the elastic modulus increment of the mechanical model (range ±10 MPa) or the correction value of the contact force estimation (range ±0.1 N).

[0106] a′: The candidate action for the next state, representing the selectable parameter adjustment strategy at the next moment s t+1 .

[0107] Q t (s t , a t ): The state-action value function at the current moment, representing the expected cumulative reward for performing the action a t under the state s t .

[0108] Q t+1 (s t , a t ): The updated state-action value function.

[0109] r t : The immediate reward, defined as the negative value of the error r t = -∣∈(t)∣, encouraging the reduction of the contact force error.

[0110] α = 0.1: The learning rate, controlling the update step size.

[0111] γ = 0.9: The discount factor, balancing the immediate and long-term rewards.

[0112] Through this iterative update, the system continuously optimizes the force estimation model, making F estimated gradually approach F target .

[0113] Example 4: Force Estimation for Robotic Vascular Suturing Based on Finite Element Analysis and Deep Reinforcement Learning

[0114] Background and Principle Explanation:

[0115] This example combines finite element analysis (FEA) and deep reinforcement learning (DRL) to estimate the catheter-vessel contact force in real time during robotic vascular suturing, further improving the accuracy of robot control. By combining simulation and actual operation, this example optimizes the force estimation and feedback mechanism.

[0116] Steps and Parameters

[0117] Finite Element Model Construction and Vascular Mechanics Simulation: The blood vessel is modeled as a three-dimensional model containing elastic materials, and the simulation is carried out using ANSYS 2023R1 software. The material properties of the blood vessel are as follows:

[0118] Elastic Modulus: E = 100 MPa

[0119] Poisson's Ratio: ν = 0.4

[0120] Initial Thickness: 1 mm

[0121] Maximum Strain: 0.05

[0122] The rigidity of the catheter is simulated by combining linear elastic theory and nonlinear contact mechanics.

[0123] Specifically, based on the extracted motion modes and the prior biomechanical model of the cardiovascular system, a multi-scale, high-dimensional image space mechanics model is constructed. This model considers the periodic motion of the vascular system, local deformation, hemodynamic effects, and the mechanical interaction between the blood vessel and the suturing tool (catheter). The rigidity of the catheter is simulated by combining linear elastic theory and nonlinear contact mechanics, and the specific method is as follows:

[0124] Linear Elastic Theory: Assume that the catheter material is an isotropic elastic body, and calculate its rigid response under small deformations based on Hooke's Law. Given Poisson's ratio ν = 0.4, initial thickness 1 mm, and elastic modulus E (range 50 - 150 MPa), calculate the rigid parameters of the catheter under maximum strain ∈ = 0.05 through the stress-strain relationship σ = E.

[0125] Nonlinear Contact Mechanics: Consider the contact area between the catheter and the blood vessel, and simulate the nonlinear contact behavior using Hertz contact theory. Assume that the catheter is cylindrical (diameter 2 mm), and the blood vessel surface is approximately an elastic plane, and calculate the nonlinear changes of the contact force F and the contact area with strain.

[0126] Combination Method: Integrate linear elastic theory (global rigidity) and nonlinear contact mechanics (local contact) through finite element analysis (FEA), and couple the overall deformation of the catheter and the local response of the contact area in the same model. Use experimental data (such as force sensor feedback) to calibrate the simulation results to ensure the accuracy of rigidity estimation.

[0127] The simulation results are used to optimize the mechanical model parameters and improve the accuracy of contact force estimation.

[0128] Image and Sensor Data Acquisition: The vascular images were acquired using a Philips EPIQ 7 ultrasound device, and the acquisition parameters were set as follows:

[0129] Ultrasound frequency: 5 MHz

[0130] Resolution: 1280×1024

[0131] Imaging mode: Color Doppler

[0132] Acquisition frequency: 50 FPS

[0133] The sensor data includes the force sensor signal at the catheter tip, and the sampling frequency is 500 Hz.

[0134] Deep Reinforcement Learning Model Training: In this embodiment, a deep reinforcement learning (DRL) model was used to optimize the force estimation, and the training process included the following settings:

[0135] State space: Includes vascular images, catheter position, and force sensor data

[0136] Action space: Adjust the catheter position and apply force

[0137] Reward function: The smaller the force error, the greater the reward. The formula is as follows:

[0138] R t =-|F target -F estimated |

[0139] Network structure: Training was performed using a deep Q-network (DQN), and the DQN model configuration was as follows:

[0140] Number of network layers: 3 fully connected layers, with 512 neurons in each layer

[0141] Activation function: ReLU

[0142] Optimizer: Adam, learning rate is 0.001

[0143] Real-time Force Estimation and Feedback Regulation: The optimized model through deep reinforcement learning estimates the force of catheter-vessel contact in real time, and transmits the estimation result to the robot control system (such as KUKA LBR iiwa) through ROS (Robot Operating System), and ensures that the applied force is within a safe range through precise feedback control.

[0144] The robot system adjusts the catheter's attitude and force according to the real-time feedback, avoiding complications such as excessive compression of blood vessels or perforation.

[0145] Example 5: Application of Multimodal Image Fusion and Adaptive Force Estimation in Robotic Vascular Suturing

[0146] Background and Principle Explanation:

[0147] In this embodiment, a multi-modal image fusion technology and an adaptive force estimation method are utilized. By combining multiple imaging techniques through a deep learning model, the contact force between the catheter and the blood vessel is estimated. By integrating image data and force sensor data, the mechanical model is dynamically adjusted during the operation to ensure the accuracy of the operation.

[0148] Steps and Parameters:

[0149] Multi-modal Image Data Acquisition: The combination of GE Discovery IGS 730 (X-ray) and Hitachi HI VISION Preirus (ultrasound) is used to perform real-time blood vessel image acquisition. The specific parameters are as follows:

[0150] X-ray Imaging: Ray dose 1.5 mGy, resolution: 2048×2048, acquisition frequency: 15 FPS

[0151] Ultrasound Imaging: Frequency: 7.5 MHz, resolution: 1920×1080, acquisition frequency: 50 FPS

[0152] Image Fusion and Feature Extraction: 3D U-Net is used for image segmentation and feature extraction. Image fusion is performed through a convolutional neural network (CNN) to combine X-ray and ultrasound data, and the boundary, morphology, and deformation features of the blood vessel are extracted.

[0153] Network Layer Configuration:

[0154] Convolutional Layer: 3 layers, with the number of filters in each layer being 64, 128, and 256

[0155] Max Pooling Layer: 2×2

[0156] Activation Function: Leaky ReLU

[0157] Force Estimation and Adaptive Adjustment: An adaptive weighted fusion method is adopted to adjust the weight of the mechanical model according to the real-time image data. The force estimation formula is as follows:

[0158] F contact = α1·F image + α2·F sensor

[0159] Among them, α1 and α2 are adaptive weights, which are dynamically adjusted based on the current image quality and sensor feedback.

[0160] After the force sensor data (such as using ATI Mini40) is fused with the image data, the contact force is estimated for real-time feedback to the robot control system.

[0161] Example 6: Real-time Force Estimation Based on the Combined CNN-LSTM Model and Adaptive Kalman Filter

[0162] Background and Principle Explanation:

[0163] In this example, for the spatio-temporal information fusion algorithm proposed in step S4, the real-time force estimation ability of the combined model based on the deep convolutional neural network (CNN) and long short-term memory network (LSTM) in robot vascular suturing is verified, and the mechanical model is optimized by combining the adaptive Kalman filter and the recursive least squares method (RLS) to ensure the accuracy and stability of force estimation in a dynamic environment.

[0164] Steps and Parameters:

[0165] 1. Image Acquisition and Preprocessing

[0166] Use a Philips EPIQ 7 ultrasound imaging device to collect high-frequency dynamic images of the vascular region and the suturing tool. The parameters are as follows: Ultrasound frequency: 7.5 MHz Resolution: 1280×1024 Acquisition frequency: 30 FPS

[0167] For preprocessing, Gaussian filtering (standard deviation σ = 0.5) is used to remove high-frequency noise, and contrast-limited adaptive histogram equalization (CLAHE, clipping threshold 0.01) is used to enhance the image contrast to ensure that the vascular boundaries and tool positions are clearly visible.

[0168] 2. Motion Mode Extraction

[0169] Perform multi-scale wavelet transform (using Daubechies wavelet, decomposition level 4) on the preprocessed images to extract high-frequency motion features (such as vascular pulsation and tool contact deformation). The images are transformed to the frequency domain through the fast Fourier transform (FFT), and the motion modes of the cardiovascular system, including cardiac pulsation, respiratory motion, blood flow fluctuation, vascular elastic deformation, and contact deformation, are extracted by combining principal component analysis (PCA, retaining the first 5 principal components) and independent component analysis (ICA). Adaptive mode decomposition (AMD) optimizes mode extraction by dynamically adjusting the frequency threshold (range 0.1 - 5 Hz) to adapt to individual patient differences.

[0170] 3. Mechanical Model Construction

[0171] Based on the extracted motion modes and a priori biomechanical models of the cardiovascular system, a mechanical model is constructed, considering the periodic motion, local deformation, and hemodynamic effects of blood vessels. Computational fluid dynamics (CFD, Reynolds number Re = 500, implemented by ANSYS Fluent) and finite element analysis (FEA, elastic modulus E = 100 MPa, Poisson's ratio ν = 0.4, implemented by ANSYS Mechanical) are used to simulate the three-dimensional deformation and mechanical response of blood vessels. The inverse problem is solved by optimizing the model parameters with experimental data (such as force sensor feedback, sampling frequency 500 Hz).

[0172] 4. Spatiotemporal Information Fusion and Force Estimation

[0173] Implement the spatiotemporal information fusion algorithm in step S4, specifically the CNN-LSTM joint model:

[0174] CNN part: Adopt the ResNet-50 architecture, with an input image size of 224×224, including 64 3×3 convolutional kernels (stride 2), a maximum pooling window of 3×3 (stride 2), and the activation function is ReLU, which is used to extract spatial features (such as blood vessel boundaries and tool positions).

[0175] LSTM part: The input temporal feature length is 128 (corresponding to a 4-second image sequence), including 2 layers of LSTM networks, with 256 neurons in each layer, the activation function is Tanh, and the optimizer is Adam (learning rate 0.001), which is used to capture the periodic and aperiodic motion characteristics of blood vessels.

[0176] Force estimation optimizes the dynamic accuracy through adaptive Kalman filtering, and the parameter settings are as follows: process noise covariance: Q = 0.01, measurement noise covariance: R = 0.1.

[0177] State transition matrix: Based on the linear approximation of blood vessel motion, it is dynamically updated.

[0178] The parameters of the mechanical model are optimized online by the recursive least squares method (RLS), with a forgetting factor λ = 0.99 and an initial covariance matrix P = 100I (I is the identity matrix), ensuring that the model adapts to the changes in the blood vessel state during surgery.

[0179] The output contact force estimate value (unit: Newton, range 0 - 5 N) is compared with the force sensor data, and the error is controlled within ±0.05 N.

[0180] 5. Force Feedback and Robot Control

[0181] The estimated contact force is fed back to the surgical robot (such as Intuitive Surgical Da Vinci Xi) through the ROS (Robot Operating System) interface. The control system adopts a model predictive control (MPC) framework with the following parameters: prediction horizon: 10 steps (corresponding to 0.33 seconds), control horizon: 2 steps, objective function: minimize the force error and tool displacement.

[0182] A 3-layer fully connected neural network (with 512 neurons in each layer and ReLU activation) is combined to optimize the control strategy, ensuring high-precision operation of the suture tool under complex vascular morphologies (position error < 0.1 mm, force error < 0.05 N).

[0183] Example 7: Real-time force feedback and tool control based on a hybrid control framework of MPC and neural network

[0184] Background and principle description:

[0185] This example focuses on the "hybrid control framework based on model predictive control (MPC) and neural network" proposed in step S4 and verifies its application effect in robotic vascular suturing. This framework combines the predictive optimization ability of MPC and the dynamic correction ability of neural network to achieve precise control of the suture tool.

[0186] Steps and parameters:

[0187] Image acquisition and preprocessing

[0188] Use a Philips EPIQ 7 ultrasound imaging system to acquire images of blood vessels and suture tools:

[0189] Frequency: 7.5 MHz;

[0190] Resolution: 1280×1024;

[0191] Acquisition frequency: 30 FPS;

[0192] The preprocessing uses Gaussian filtering (σ = 0.5) and CLAHE (clipping threshold 0.01).

[0193] Contact force estimation

[0194] Use a CNN-LSTM joint model to estimate the contact force: CNN: ResNet-50, input 224×224, 64 3×3 convolutional kernels, ReLU activation.

[0195] LSTM: sequence length 128, 2 layers with 256 neurons, Tanh activation, Adam optimization (learning rate 0.001).

[0196] Optimized estimation using adaptive Kalman filtering (Q = 0.01, R = 0.1) and recursive least squares method (λ = 0.99), and output the real-time contact force (range 0 - 5N).

[0197] The hybrid control framework implements model predictive control (MPC): Prediction model: Based on the mechanical model in step S3 (vascular elastic modulus E = 100 MPa, blood flow velocity 1 m / s), predict the change of contact force in the next 10 steps (0.33 seconds).

[0198] Control objective: Minimize the force error and tool displacement, and the objective function is defined as:

[0199]

[0200] where, F ref is the target force, F pred is the predicted force, Δu is the change of control input, N p = 10 (prediction horizon), N c = 2 (control horizon), λ = 0.1 (weight coefficient).

[0201] Optimization algorithm: Use a quadratic programming (QP) solver (such as quadprog in MATLAB), and the update frequency is 30 Hz.

[0202] Output: The displacement adjustment amount of the tool (range ±0.5 mm) and the force adjustment amount (range ±0.5 N).

[0203] Neural network correction: Network structure: A 3-layer feedforward neural network with 512 neurons in each layer, ReLU activation, and linear activation in the output layer.

[0204] Input: The current contact force, vascular motion mode (extracted from step S2), and MPC prediction error.

[0205] Output: The correction value of the MPC control input (range ±0.1 N).

[0206] Training: Use supervised learning, the dataset is 5000 groups of simulated surgical data (force error and displacement error), the optimizer is Adam (learning rate 0.001), and train until the mean square error < 0.01.

[0207] Implementation tool: MPC is implemented in MATLAB / Simulink, the neural network is built using TensorFlow, and integrated into the robot system through the ROS interface.

[0208] Force feedback and tool control

[0209] The real-time contact force is transmitted to the Intuitive Surgical Da Vinci Xi surgical robot through ROS. The hybrid control framework adjusts the suturing tool at a frequency of 30 Hz: Force sensing control: Adjust the force according to the real-time force feedback (error < 0.05 N).

[0210] Torque control: Adjust the direction according to the tool attitude (error < 0.1°).

[0211] Hybrid control: MPC provides the initial control instructions, and the neural network corrects the non-linear errors.

[0212] The above description of the embodiments is to enable those of ordinary skill in the art to understand and use the invention. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative efforts. Therefore, the present invention is not limited to the above embodiments, and all improvements and modifications made by those skilled in the art without departing from the scope of the present invention should be within the protection scope of the present invention.

Claims

1. A vision-based force feedback estimation method for robotic vascular suturing, characterized in that It includes the following steps: S1: Collect high-frequency dynamic images of the blood vessel area and the suture tool, and preprocess the images; S2: Perform time-frequency analysis on the preprocessed images; after converting the preprocessed images to the frequency domain, extract the motion modes of the cardiovascular system, and the motion modes include cardiac pulsation, respiratory movement, blood flow fluctuation, blood vessel elastic deformation, and contact deformation between the suture tool and the blood vessel; S3: Based on the extracted motion modes and the prior biomechanical model of the cardiovascular system, construct a mechanical model considering the periodic motion, local deformation, hemodynamic effects of the blood vessel system, and mechanical interaction between the blood vessel and the suture tool; S4: Through a spatio-temporal information fusion algorithm, combine the extracted motion mode information with the mechanical response of the mechanical model to estimate the contact force when the suture tool contacts the blood vessel.

2. The force feedback estimation method for robot vascular suture based on vision according to claim 1, wherein In step S1, the high-frequency dynamic images are obtained by X-ray and / or ultrasonic imaging, and the high-frequency dynamic images are HDR images; the preprocessing includes one or more of denoising, artifact removal, contrast enhancement, illumination correction, and Gaussian filtering.

3. A method for force feedback estimation of a vision-based robotic vascular suture according to claim 1, characterized in that In step S2, it includes one or more of the following: i) The time-frequency analysis is performed by wavelet transform; ii) The preprocessed images are converted to the frequency domain by fast Fourier transform; iii) Extracting the motion modes of the cardiovascular system is performed by multi-modal analysis, including principal component analysis and independent component analysis, and the parameters in the extraction process are dynamically adjusted by an adaptive modal decomposition method.

4. A visual-based force feedback estimation method for robotic vascular suturing according to claim 1, characterized in that In step S3, it includes one or two of the following: i) Based on the mechanical model, use multi-physics field coupling simulation technology to simulate the three-dimensional deformation and mechanical response of the blood vessel under dynamic conditions, and the multi-physics field coupling simulation technology includes computational fluid dynamics and finite element analysis; ii) By means of inverse problem solving, adjust and optimize the parameters of the mechanical model through experimental data feedback.

5. A vision-based force feedback estimation method for robotic vascular suture according to claim 1, characterized in that, In step S4, the spatio-temporal information fusion algorithm is: a joint model based on a deep convolutional neural network and a temporal recursive neural network; the estimation improves the accuracy of dynamic estimation through adaptive Kalman filtering, and online optimizes the mechanical model by recursive least squares method.

6. The force feedback estimation method for robotic vascular suture based on vision according to claim 1, characterized in that The method further includes step S5: Optimize the parameters of the mechanical model based on the data collected during the operation through an adaptive mechanism and / or autonomous learning; the adaptive mechanism and / or autonomous learning includes deep reinforcement learning and meta-learning.

7. A vision-based force feedback estimation method for robotic vascular suturing according to claim 6, wherein In step S5, through incremental learning technology, optimize the parameters of the mechanical model according to the new data obtained during the operation.

8. A vision-based force feedback estimation method for robotic vascular suturing according to claim 1, characterized in that The method further includes step S6: Dynamically adjust the mechanical model during the operation through multi-modal information fusion; the multi-modal information includes the collected images and force sensor data.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the vision-based robotic vascular suture force feedback estimation method according to any one of claims 1-8.

10. A vision-based force feedback estimation system for robotic vascular suture, characterized in that, It includes: An image acquisition module for collecting high-frequency dynamic images of the blood vessel area and the suture tool; An algorithm analysis module, which is used to estimate the contact force when the suture tool contacts the blood vessel based on the high-frequency dynamic images collected by the image acquisition module and by using the vision-based force feedback estimation method for robotic vascular suture as described in any one of claims 1-8; A robot control module, which is used to feedback the contact force estimated by the algorithm analysis module to the surgical robot.

Images