Depth estimation module, control system and method for robot-assisted heart valve catheterization

Abstract

The invention discloses a depth estimation module, a control system and a method for a robot-assisted heart valve catheter operation, which provide key technical guarantee for accurate navigation and operation of the robot-assisted heart valve catheter operation and are beneficial to improving the operation safety, reducing operation wounds and shortening the operation time. The depth estimation module introduces a pre-trained monocular depth estimation network by using an LoRA adaptation layer to realize rapid adaptation to patient individual differences and surgical environment changes, and realizes real-time and accurate depth estimation. Meanwhile, the problem of lack of annotation data is solved by utilizing self-supervised learning, and the robustness of depth estimation is further improved through multi-modal information fusion and heart periodic motion compensation. A robot control module in the control system generates a three-dimensional point cloud according to an estimated depth map, plans a catheter path in combination with a robot kinematics model, adjusts the position of an end effector of the robot in real time, and ensures accurate operation.

Description

Technical Field

[0001] The present invention relates to the technical field of robot-assisted surgery systems, and more specifically, to a depth estimation module, control system and method for robot-assisted cardiac valve catheterization. Background Art

[0002] As a revolutionary minimally invasive technology, cardiac valve catheter surgery has brought good news to many patients with heart valve diseases. Compared with traditional open-chest surgery, it has significant advantages such as less trauma, faster recovery, and fewer complications, and is gradually becoming the preferred option for treating such diseases. In recent years, with the introduction of robot-assisted technology, the safety and effectiveness of cardiac valve catheter surgery have been further improved. The robot-assisted system can provide the operator with stable instrument control, eliminate hand tremors, and achieve precise positioning, thereby minimizing surgical risks and increasing the success rate of surgery.

[0003] However, at present, robot-assisted systems still face some key challenges in depth perception, which restricts their application and development in cardiac valve catheter surgery. Depth perception is the key for robot-assisted systems to perceive the surgical environment and understand surgical operations, which directly affects the safety and accuracy of the surgery. In cardiac valve catheter surgery, accurate depth perception is crucial because it is directly related to the following aspects:

[0004] Precise positioning of catheters and instruments: In complex cardiac anatomy, catheters and instruments need to be accurately guided to the target location. The accuracy of depth perception directly determines the accuracy of positioning, which in turn affects the safety and effectiveness of surgical operations.

[0005] Avoid accidental damage to surrounding tissue structures: There are many important blood vessels and nerves around the heart, and any accidental damage may have serious consequences. Accurate depth perception can help the robot-assisted system identify and avoid these key structures to ensure the safety of the operation.

[0006] Optimizing the release and deployment of surgical instruments: Heart valve replacement surgery requires the new valve to be delivered to the designated location and accurately released and deployed. The accuracy of depth perception is crucial to ensure the correct installation of the valve.

[0007] Real-time evaluation of surgical results: During surgery, it is necessary to monitor the surgical results in real time, such as the opening and closing of the valve, changes in hemodynamics, etc. Depth perception can provide important reference information for the evaluation of surgical results.

[0008] However, existing depth perception technologies (such as monocular depth estimation networks and binocular depth estimation networks) have the following limitations when applied to heart valve catheter surgery:

[0009] The high deformability and periodic motion of cardiac tissue interfere with depth estimation: The heart is a continuously beating organ, and its shape and position are constantly changing. This dynamic change can interfere with depth perception and lead to inaccurate depth estimation.

[0010] Lack of sufficient annotated data for training high-precision depth estimation models: Deep learning is the mainstream technology in the field of depth perception, and training high-precision deep learning models requires a large amount of annotated data. However, due to the complexity and particularity of heart valve catheter surgery, there is currently a lack of sufficient annotated data for model training.

[0011] Difficulty adapting to anatomical differences between patients and dynamic changes during surgery: The cardiac anatomy of each patient is different, and the shape and position of the heart will also change during surgery. Existing depth perception technology is difficult to fully adapt to such individual differences and dynamic changes, resulting in reduced accuracy and stability in practical applications.

[0012] In summary, depth perception is a key technology for robot-assisted heart valve catheter surgery, but it still faces many challenges at this stage. More advanced depth perception technology needs to be developed to overcome the above limitations, further improve the safety and effectiveness of heart valve catheter surgery, and benefit more patients. Summary of the invention

[0013] In response to the depth perception problem in robot-assisted heart valve catheter surgery, the present invention proposes a depth estimation module, control system and method for robot-assisted heart valve catheter surgery, which achieves real-time and accurate depth estimation by quickly adapting to individual differences in patients and changes in the surgical environment, overcoming the shortcomings of traditional methods in terms of real-time, accuracy and adaptability. At the same time, self-supervised learning is used to solve the problem of lack of labeled data, and the robustness of depth estimation is further improved through multimodal information fusion and cardiac periodic motion compensation. The present invention provides key technical guarantees for the precise navigation and operation of robot-assisted heart valve catheter surgery, which helps to improve surgical safety, reduce surgical trauma and shorten surgical time.

[0014] To achieve the above-mentioned object, in a first aspect, the present invention provides a depth estimation module for robot-assisted cardiac valve catheterization, comprising a pre-trained monocular depth estimation network as a basic network;

[0015] It is characterized by further comprising:

[0016] LoRA adaptation layer, adding low-rank adaptation matrices A and B to the key layers of the base network;

[0017] During the online adaptive training of the depth estimation module, only LoRA parameters A and B are updated, and the basic network parameters are kept unchanged; a self-supervised learning strategy is adopted to construct a self-supervised loss function using the stereo image pairs obtained during the operation; the self-supervised loss function takes into account losses including reprojection error and smoothness.

[0018] The depth estimation module can improve the depth perception accuracy and real-time adaptability during surgery through the LoRA adaptation layer and a unique self-supervised learning strategy. It can also quickly adapt to the anatomical features and surgical environment of specific patients and can achieve real-time updates without the need for a large amount of computing resources, thereby enhancing the safety and effectiveness of robot-assisted heart valve catheter surgery.

[0019] Furthermore, the low-rank adaptation matrix A∈R r×d ,B∈R d×r , where r<<d, d is the dimension of the original weight matrix;

[0020] LoRA modifies the original weight matrix W by the following formula:

[0021] W′=W+BA

[0022] During the online adaptive training process, only the LoRA parameters A and B are updated, and the basic network parameters W are kept unchanged.

[0023] Furthermore, the online adaptive training adopts an optimization algorithm including mini-batch gradient descent to update only LoRA parameters.

[0024] Furthermore, the self-supervised loss function is:

[0025]

[0026] in:

[0027] is the photometric consistency loss, which is used to constrain the difference in pixel values ​​after reprojection of the left and right views; is the smoothness loss, which is used to constrain the smoothness of the depth map;

[0028] It is the temporal consistency loss, which is used to constrain the consistency of depth maps between adjacent frames;

[0029] λ 1 and λ 2 is the weight coefficient. The model is trained by self-supervisory signals of reprojection consistency, smoothness, and temporal consistency, overcoming the problem of scarce labeled data.

[0030] In a second aspect, the present invention provides a control system for robot-assisted heart valve catheterization, characterized in that it includes:

[0031] Image acquisition module, which collects multimodal images including preoperative CT / MRI images, intraoperative ultrasound, and stereo images within the surgical field of view;

[0032] As mentioned above, the depth estimation module uses stereo image pairs acquired during surgery and uses self-supervisory signals including reprojection consistency and smoothness to train the model for estimating depth information from stereo images.

[0033] The robot control module generates a 3D point cloud based on the estimated depth map, combines the robot kinematic model, plans the catheter path, and adjusts the position of the robot end effector in real time to ensure precise operation.

[0034] In the last aspect, the present invention provides a control method for robot-assisted heart valve catheterization, characterized in that the control system of the robot-assisted heart valve catheterization as described above is used to control a heart valve catheterization assisting robot, comprising the following steps:

[0035] Acquire multimodal images including preoperative CT / MRI images, intraoperative ultrasound, and stereo images within the surgical field of view;

[0036] Before the operation begins, the depth estimation module is preheated using the patient's preoperative multimodal images;

[0037] The depth estimation module is trained adaptively online during surgery to output an estimated depth map based on real-time stereo images.

[0038] Depth map to point cloud conversion: Input the depth map D(u,v) obtained from the depth estimation module, where (u,v) represents the pixel coordinates in the image. Use the camera's intrinsic parameter matrix K (including the focal length f x ,f y and the main point c x ,c y ) Convert each pixel in the depth map to a point in three-dimensional space; let P = (X, Y, Z) be a point in three-dimensional space, then:

[0039]

[0040] Output a point cloud P containing the three-dimensional coordinates of the points in the surgical space i =(X i ,Y i ,Z i );

[0041] Conversion of point cloud to robot coordinate system: Input point cloud P i and the pose of the robot base, according to the transformation of the camera relative to the robot end effector and the transformation of the robot end effector relative to the robot base A point P in space i The representation in the robot coordinate system is It can be obtained by the following transformation:

[0042]

[0043] Output point cloud representation in the robot coordinate system

[0044] Path planning and control: Input the point cloud of the target operation position, representing and the current position of the robot's end effector; use a path planning algorithm to find a safe path from the current position to the target position in the robot's operating space; output a path from the current position to the target position;

[0045] Real-time adjustment and execution: Based on the planned path, the robot's joint angles are adjusted through the robot power control system to fine-tune the path in real time, taking into account the periodic motion of the heart and the depth information fed back by the depth estimation module; the output is based on the robot's real-time execution of the adjusted action to precisely manipulate the catheter or other surgical instruments.

[0046] Furthermore, during the online adaptive training of the depth estimation module, the LoRA parameters are updated once per frame.

[0047] Furthermore, the transformation of the camera relative to the robot end effector Obtained through hand-eye calibration.

[0048] Furthermore, the path planning algorithm includes RRT-Connect or PRM.

[0049] Furthermore, the output of the path planning and control is a set of robot joint angles θ i =(θ i1 ,θ i2 ,…,θ in ), where n is the robot degree of freedom.

[0050] Compared with the prior art, the present invention has the following technical effects:

[0051] (1) The present invention uses LoRA technology to achieve rapid adaptation: by introducing a low-rank adaptation layer in the pre-trained model, the model can quickly adapt to the anatomical characteristics and surgical environment of a specific patient, and real-time updates can be achieved without the need for a large amount of computing resources.

[0052] (2) The present invention adopts a self-supervised learning strategy, using stereo image pairs acquired during surgery to train the model through self-supervisory signals such as reprojection consistency and timing consistency, thus overcoming the problem of scarce labeled data.

[0053] (3) The present invention integrates multimodal information and combines multimodal data such as preoperative CT / MRI images and intraoperative ultrasound to improve the robustness and accuracy of depth estimation.

[0054] (4) The present invention adapts to the periodic motion of the heart and designs a specific network structure and loss function to capture and compensate for the impact of the periodic motion of the heart on depth estimation.

[0055] (5) The present invention uses model compression and hardware acceleration technology to ensure that depth estimation can be performed in real time during surgery, meeting the real-time requirements of robot control.

[0056] The present invention provides key technical guarantees for the precise navigation and operation of robot-assisted heart valve catheter surgery, which helps to improve surgical safety, reduce surgical trauma, and shorten operation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an execution flow chart of a depth estimation module in one embodiment of the present invention;

[0058] Figure 2 FIG. 4 is a control system architecture diagram of a robot-assisted heart valve catheterization in one embodiment of the present invention.

[0059] Figure 3 This is an execution flow chart of a robot control module in one embodiment of the present invention; DETAILED DESCRIPTION

[0060] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0061] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that well-known algorithms and models (such as the MonoDepth2 model, the small batch gradient descent optimization algorithm, the path planning algorithm) are not shown in detail to avoid obscuring the subject matter of the present invention.

[0062] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0063] Example 1

[0064] See also Figure 1 In this embodiment, the depth estimation module of the robot-assisted cardiac valve catheterization includes a pre-trained monocular depth estimation network as a basic network, such as MonoDepth2. The network has been trained on a large-scale data set and has a certain depth estimation capability. The depth estimation module also includes a LoRA adaptation layer. The LoRA adaptation layer adds low-rank adaptation matrices A and B to the key layers (such as convolutional layers) of the basic network.

[0065] The execution of the depth estimation module is embedded from the image position to the basic network decoder containing the LoRA adaptation layer, and then the depth decoder outputs the depth map. During the online adaptive training of the depth estimation module, only the LoRA parameters A and B are updated, and the basic network parameters are kept unchanged; a self-supervised learning strategy is adopted to construct a self-supervised loss function using the stereo image pairs obtained during the operation; the self-supervised loss function considers losses including reprojection error and smoothness.

[0066] More specifically, the low-rank adaptation matrix A∈R r×d ,B∈R d×r , where r<<d, d is the dimension of the original weight matrix; LoRA modifies the original weight matrix W by the following formula:

[0067] W′=W+BA

[0068] During the online adaptive training process, only the LoRA parameters A and B are updated, and the basic network parameters W are kept unchanged.

[0069] The online adaptive training uses an optimization algorithm including mini-batch gradient descent to update only the LoRA parameters to achieve rapid adaptation of the model.

[0070] The self-supervised loss function is:

[0071]

[0072] in:

[0073] is the photometric consistency loss, which is used to constrain the difference in pixel values ​​after reprojection of the left and right views;

[0074] is the smoothness loss, which is used to constrain the smoothness of the depth map;

[0075] It is the temporal consistency loss, which is used to constrain the consistency of depth maps between adjacent frames;

[0076] λ 1 and λ 2 is the weight coefficient.

[0077] The model is trained by self-supervisory signals such as reprojection consistency and timing consistency, overcoming the problem of scarce labeled data. In addition, the above-designed specific network structure and loss function can effectively capture and compensate for the impact of cardiac periodic motion on depth estimation.

[0078] The following is a specific embodiment of executing the depth estimation module to help those skilled in the art understand the implementation process of this embodiment.

[0079] Data collection: A binocular camera system is used, fixed at the end of the heart valve interventional catheter robot, to collect stereo images within the surgical field of view. The camera parameters are as follows:

[0080] Resolution: 1280 x 1024 pixels

[0081] Frame rate: 30 frames per second

[0082] Baseline length: 10 cm

[0083] Base network: MonoDepth2

[0084] LoRA adaptation layer: A LoRA adaptation layer with rank r=16 is added to each convolutional layer in the decoder part of MonoDepth2.

[0085] Self-supervised loss function:

[0086] Optimization algorithm: Adam, learning rate $10 -4 , batch size 4

[0087] Before the operation begins, the model is warmed up using the patient's preoperative CT / MRI images, and then online adaptive training is performed during the operation, updating the LoRA parameters once per frame.

[0088] Example 2

[0089] See also Figure 2 This embodiment provides a control system and a control method for robot-assisted heart valve catheterization. The control system includes:

[0090] Image acquisition module, which collects multimodal images including preoperative CT / MRI images, intraoperative ultrasound, and stereo images within the surgical field of view (a binocular camera system or a structured light camera system can be selected to obtain high-quality stereo image pairs);

[0091] The LoRA-based self-supervised depth estimation module as described in Example 1 uses a stereo image pair obtained during surgery to perform model training through self-supervised signals including reprojection consistency and smoothness to estimate depth information from the stereo images;

[0092] The robot control module generates a 3D point cloud based on the estimated depth map, combines the robot kinematic model, plans the catheter path, and adjusts the position of the robot end effector in real time to ensure precise operation.

[0093] The method for controlling a heart valve catheterization assisting robot to perform robot-assisted heart valve catheterization using the control system of the robot-assisted heart valve catheterization comprises the following steps:

[0094] Acquire multimodal images including preoperative CT / MRI images, intraoperative ultrasound, and stereo images within the surgical field of view;

[0095] Before the operation begins, the depth estimation module is preheated using the patient's preoperative multimodal images;

[0096] The depth estimation module is adaptively trained online during surgery and outputs estimated depth maps based on real-time stereo images.

[0097] Next, the control of the surgical robot incorporating the depth map estimated by the self-supervised depth estimation module is crucial to ensure the success of the surgery. Figure 3 ,The following is a detailed description of the robot control module, including specific implementation steps and mathematical formulas:

[0098] 1. Conversion from depth map to point cloud

[0099] Input: Depth map D(u,v) obtained from the depth estimation module, where (u,v) represents the pixel coordinates in the image.

[0100] Process: Use the camera's intrinsic parameter matrix K (including focal length f x ,f y and the main point c x ,c y ) converts each pixel in the depth map into a point in three-dimensional space. Let P = (X, Y, Z) be a point in three-dimensional space, then:

[0101]

[0102] Output: A point cloud P containing the 3D coordinates of the points in the surgical space i =(X i ,Y i ,Z i ).

[0103] 2. Conversion from point cloud to robot coordinate system

[0104] Input: Point cloud P i and the pose (position and orientation) of the robot base.

[0105] The process first needs to know the transformation of the camera relative to the robot end effector (which can be obtained through hand-eye calibration), and the transformation of the robot end effector relative to the robot base Then a point P in space i The representation in the robot coordinate system is It can be obtained by the following transformation:

[0106]

[0107] Output: Point cloud representation in the robot coordinate system

[0108] Hand-eye calibration can accurately determine the relative position and orientation between the camera and the robot's end effector, which is the key to converting the depth information from the camera's perspective into the robot's operating space.

[0109] 3. Path planning and control

[0110] Input: Point cloud representation of the target operation location and the current position of the robot end effector.

[0111] Process: Use a path planning algorithm (such as RRT-Connect or PRM) to find a safe path from the current position to the target position in the robot's operating space. This involves avoiding sensitive areas and blood vessels in the heart tissue.

[0112] Output: A path from the current position to the target position, usually a set of robot joint angles Θ i =(θ i1 ,θ i2 ,…,θ in ), where n is the robot degree of freedom.

[0113] 4. Real-time adjustment and execution

[0114] Process: Based on the planned path, the robot's joint angles are adjusted by the control system. Given the cyclical motion of the heart, the path may need to be fine-tuned in real time. This typically involves a feedback control loop where real-time depth information provided by depth estimation is used to fine-tune the position of the robot's end effector.

[0115] Output: The robot executes the adjusted movements in real time, precisely manipulating the catheter or other surgical instrument.

[0116] The control system of the above-mentioned robot-assisted heart valve catheterization also needs to consider:

[0117] Real-time performance: Considering the dynamic environment of the heart, the robotic control system must have high real-time performance. This is usually achieved by optimizing the computational efficiency of the algorithm, parallel processing, and the use of dedicated hardware (such as FPGA or GPU).

[0118] Safety and fault tolerance: The designed path planning and control algorithms must take into account the complexity of the surgical environment, including avoiding accidental contact with cardiac tissue to ensure the safety of the operation.

[0119] Through this precise robotic control process, combined with real-time depth perception and adaptable path planning, robot-assisted heart valve catheter surgery can be performed more efficiently, thereby reducing surgical risks and improving success rates.

[0120] If the control method of the above-mentioned robot-assisted heart valve catheterization is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the essence of this technical solution or the part that contributes to the prior art or the part of this technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0121] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0122] In summary, the present invention provides a depth estimation module, control system and method for robot-assisted heart valve catheterization, which provides key technical guarantees for the precise navigation and operation of robot-assisted heart valve catheterization surgery, and helps to improve surgical safety, reduce surgical trauma, and shorten surgical time. The depth estimation module uses the LoRA adaptation layer to introduce a pre-trained monocular depth estimation network to quickly adapt to individual differences in patients and changes in the surgical environment, and achieve real-time and accurate depth estimation. At the same time, self-supervised learning is used to solve the problem of lack of labeled data, and the robustness of depth estimation is further improved through multimodal information fusion and cardiac periodic motion compensation. The robot control module in the control system generates a three-dimensional point cloud based on the estimated depth map, combines the robot kinematic model, plans the catheter path, and adjusts the position of the robot end effector in real time to ensure precise operation.

[0123] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0124] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A depth estimation module for robot-assisted heart valve catheterization, comprising a pre-trained monocular depth estimation network as a base network; characterized in that: Also includes: LoRA adaptation layer, adding low-rank adaptation matrices A and B to the key layers of the base network; During the online adaptive training of the depth estimation module, only LoRA parameters A and B are updated, and the basic network parameters are kept unchanged; a self-supervised learning strategy is adopted to construct a self-supervised loss function using the stereo image pairs obtained during the operation; the self-supervised loss function takes into account losses including reprojection error and smoothness.

2. The depth estimation module for robot-assisted heart valve catheterization according to claim 1, characterized in that: The low-rank adaptation matrix A∈R r×d ,B∈R d×r , where r<<d, d is the dimension of the original weight matrix; LoRA modifies the original weight matrix W by the following formula: W′=W+BA During the online adaptive training process, only the LoRA parameters A and B are updated, and the basic network parameters W are kept unchanged.

3. The depth estimation module for robot-assisted heart valve catheterization according to claim 2, characterized in that: The online adaptive training uses an optimization algorithm including mini-batch gradient descent to update only the LoRA parameters.

4. The depth estimation module for robot-assisted heart valve catheterization according to claim 1, characterized in that: The self-supervised loss function is: in: is the photometric consistency loss, which is used to constrain the difference in pixel values ​​after reprojection of the left and right views; is the smoothness loss, which is used to constrain the smoothness of the depth map; It is the temporal consistency loss, which is used to constrain the consistency of depth maps between adjacent frames; λ1 and λ2 are weight coefficients.

5. A control system for robot-assisted heart valve catheterization, characterized in that: include: Image acquisition module, which collects multimodal images including preoperative CT / MRI images, intraoperative ultrasound, and stereo images within the surgical field of view; The depth estimation module according to any one of claims 1 to 4, using a stereo image pair acquired during surgery, and training a model through self-supervisory signals including reprojection consistency and smoothness, for estimating depth information from the stereo image; The robot control module generates a 3D point cloud based on the estimated depth map, combines the robot kinematic model, plans the catheter path, and adjusts the position of the robot end effector in real time to ensure precise operation.

6. A control method for robot-assisted heart valve catheterization, characterized in that: Using the control system of the robot-assisted heart valve catheterization as claimed in claim 5 to control the heart valve catheterization-assisted robot comprises the following steps: Acquire multimodal images including preoperative CT / MRI images, intraoperative ultrasound, and stereo images within the surgical field of view; Before the operation begins, the depth estimation module is preheated using the patient's preoperative multimodal images; The depth estimation module is trained adaptively online during surgery to output an estimated depth map based on real-time stereo images. Depth map to point cloud conversion: Input the depth map D(u,v) obtained from the depth estimation module, where (u,v) represents the pixel coordinates in the image. Use the camera's intrinsic parameter matrix K (including the focal length f x ,f y and the main point c x ,c y ) Convert each pixel in the depth map to a point in three-dimensional space; let P = (X, Y, Z) be a point in three-dimensional space, then: Output a point cloud P containing the three-dimensional coordinates of the points in the surgical space i =(X i ,Y i ,Z i ); Conversion of point cloud to robot coordinate system: Input point cloud P i and the pose of the robot base, according to the transformation of the camera relative to the robot end effector and the transformation of the robot end effector relative to the robot base A point P in space i The representation in the robot coordinate system is It can be obtained by the following transformation: Output point cloud representation in the robot coordinate system Path planning and control: Input the point cloud of the target operation position, representing and the current position of the robot's end effector; use a path planning algorithm to find a safe path from the current position to the target position in the robot's operating space; output a path from the current position to the target position; Real-time adjustment and execution: Based on the planned path, the robot's joint angles are adjusted through the robot power control system to fine-tune the path in real time, taking into account the periodic motion of the heart and the depth information fed back by the depth estimation module; the robot executes the adjusted movements in real time to precisely manipulate the catheter or other surgical instruments.

7. The control method for robot-assisted heart valve catheterization according to claim 6, characterized in that: During the online adaptive training of the depth estimation module, the LoRA parameters are updated once per frame.

8. The control method for robot-assisted heart valve catheterization according to claim 6, characterized in that: Transformation of the camera relative to the robot end effector Obtained through hand-eye calibration.

9. The control method for robot-assisted heart valve catheterization according to claim 6, characterized in that: The path planning algorithm includes RRT-Connect or PRM.

10. The control method for robot-assisted heart valve catheterization according to claim 6, characterized in that: The output of the path planning and control is a set of robot joint angles θ i =(θ i1 ,θ i2 ,…,θ in ), where n is the robot degree of freedom.

Images