Remote interventional robotic system, control method, and device

Abstract

A remote interventional robotic system (10), a control method, and a device. The system (10) comprises: a remote interventional robot (30) determining that a puncturing apparatus (304) does not move to a skin puncturing position of a target organ; on the basis of the puncturing apparatus (304) not moving to the skin puncturing position of the target organ, receiving a first motion control instruction for a robotic arm (303); and sending the first motion control instruction to the remote interventional robot (30), and the remote interventional robot (30) controlling the robotic arm (303) to execute the first motion control instruction such that the puncturing apparatus (304) moves to the skin puncturing position, and controlling a puncture needle to advance to a lesion point of the target organ along a needle entry path. By means of a remote control end (20), the remote interventional robot (30) is accurately controlled, thereby remotely controlling a puncture needle to accurately reach a lesion point.

Description

Remote intervention robot system, control method and equipment Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a remote intervention robot system, control method and equipment. Background Art

[0002] Percutaneous interventional surgery is a typical minimally invasive procedure, in which medical imaging equipment is used to guide microsurgical instruments through tiny incisions on the patient's body surface to the patient's lesion site for surgical operation. Interventional technology is a type of percutaneous interventional surgery. It uses interventional equipment to reach the designated location of the target organ and complete procedures such as puncture biopsy, catheterization, aspiration, drug injection, and ablation therapy. This allows patients to avoid the pain of traditional surgery and achieve or even exceed the therapeutic effects of surgery. The advantages of interventional technology include precision, safety, ease of operation, ultra-minimally invasive, minimal pain, good efficacy, and no or short hospitalization, which is in line with the treatment philosophy of postmodern medicine.

[0003] In related technologies, interventional procedures are typically performed on-site by professional interventional physicians who operate interventional equipment. However, in some special scenarios, such as in remote areas where specialized interventional physicians are unavailable, or in isolation wards in hospitals where specialized interventional physicians are unable to operate interventional equipment, the development of a remote interventional robotic system is a pressing technical challenge.

[0004] Summary of the Invention

[0005] This application proposes a remote intervention robot system, control method and equipment.

[0006] On the one hand, an embodiment of the present application proposes a remote intervention robot system, which includes a remote control end and a remote intervention robot, wherein the remote control end includes a first remote communication module and a first controller, and the remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is provided on the puncture device, wherein: the first controller is used to determine that the puncture device has not moved to the skin needle insertion position of the target organ; based on the puncture device not moving to the skin needle insertion position of the target organ, receive a first motion control instruction for the robotic arm, and send the first motion control instruction to the remote intervention robot through the first remote communication module; the second controller is used to receive the first motion control instruction through the second remote communication module and control the robotic arm to execute the first motion control instruction, so that the puncture device moves to the skin needle insertion position, and controls the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

[0007] On the other hand, an embodiment of the present application proposes a control method for a remote intervention robot, the method is applied in a first controller of a remote control end, the remote control end also includes a first remote communication module, the remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is provided on the puncture device, wherein the method includes: determining that the puncture device has not moved to the skin needle insertion position of the target organ; based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ, receiving a first motion control instruction for the robotic arm; sending the first motion control instruction to the remote intervention robot through the first remote communication module, so that the second controller controls the robotic arm to move the puncture device to the skin needle insertion position according to the first motion control instruction, and controls the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

[0008] On the other hand, an embodiment of the present application proposes a method for controlling a remote intervention robot through a remote control end, wherein the remote control end includes a first remote communication module and a first controller, and the remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is provided on the puncture device, wherein the method is applied in the second controller, and the method includes: receiving a first motion control instruction through the second remote communication module, wherein the first motion control instruction is sent by the first controller through the first remote communication module, wherein the first motion control instruction is received by the first controller based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ when it is determined that the puncture device has not moved to the skin needle insertion position of the target organ; controlling the robotic arm to execute the first motion control instruction so that the puncture device moves to the skin needle insertion position, and controlling the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

[0009] On the other hand, an embodiment of the present application proposes a remote control terminal, which is used to control a remote intervention robot. The remote control terminal includes a first controller and a first remote communication module, wherein the remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is provided on the puncture device, wherein the first controller is used to execute the control method of the remote intervention robot disclosed in the present application.

[0010] On the other hand, an embodiment of the present application proposes a remote intervention robot, which is controlled by a remote control end, and the remote control end includes a first remote communication module and a first controller. The remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is provided on the puncture device, wherein the second controller is used to execute the method disclosed in the present application for controlling the remote intervention robot through the remote control end.

[0011] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the control method of the remote intervention robot of the embodiment of the present application or the method of controlling the remote intervention robot through a remote control terminal is implemented.

[0012] According to the technical solution provided in the embodiments of the present application, when the remote intervention robot determines that the puncture device has not moved to the skin insertion position of the target organ, it receives a first motion control instruction for the robotic arm based on the fact that the puncture device has not moved to the skin insertion position of the target organ, and sends the first motion control instruction to the remote intervention robot. The remote intervention robot controls the robotic arm to execute the first motion control instruction, so that the puncture device moves to the skin insertion position and controls the puncture needle to advance to the lesion point of the target organ along the needle insertion path. In this way, the remote intervention robot is accurately controlled by the remote control terminal, and the remote-controlled puncture needle is accurately positioned at the lesion point.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0015] FIG1 is a schematic structural diagram of a remote intervention robot system according to one embodiment of the present application;

[0016] FIG2 is an example diagram showing a virtual wall;

[0017] FIG3 is a diagram showing an example of the principle of adjusting a robotic arm based on a proportional differential (PD) control method;

[0018] FIG4 is an example diagram of a fused image obtained by fusing a contour image and a three-dimensional model;

[0019] FIG5 is an example diagram of a matching result obtained by matching the content currently displayed in the first display module with the ultrasound image;

[0020] FIG6 is an example diagram of the relationship between the hardware connections and communication connections between the various components of the remote intervention robot;

[0021] FIG7 is a flow chart of a control method of a remote intervention robot according to an embodiment of the present application;

[0022] FIG8 is a flow chart of a method for controlling a remote intervention robot through a remote control terminal according to one embodiment of the present application;

[0023] FIG9 is a diagram illustrating an example of the interaction process between a remote control terminal and a remote intervention robot according to an embodiment of the present application.

[0024] Figure numerals: Remote intervention robot system 10, remote control terminal 20, remote intervention robot 30, first remote communication module 201, first controller 202, robotic arm control module 203, first display module 204, ultrasound control module 205, first voice and video module 206, third display module 207, second remote communication module 301, second controller 302, robotic arm 303, puncture device 304, ultrasound probe 305, six-dimensional force sensor 306, depth camera 307, second voice and video module 308, second display module 309. DETAILED DESCRIPTION

[0025] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0026] FIG1 is a schematic structural diagram of a remote intervention robot system according to an embodiment of the present application.

[0027] As shown in FIG1 , the tele-interventional robot system 10 includes a tele-control terminal 20 and a tele-interventional robot 30, wherein the tele-control terminal 20 includes a first tele-communication module 201 and a first controller 202, wherein the tele-interventional robot 30 includes a second tele-communication module 301, a second controller 302, a robotic arm 303, and a puncture device 304, wherein the puncture device 304 is provided at the end of the robotic arm 303, and the puncture device 304 is provided with a puncture needle (not shown in the figure), wherein:

[0028] The first controller 202 is used to determine that the puncture device 304 has not moved to the skin needle insertion position of the target organ; based on the fact that the puncture device 304 has not moved to the skin needle insertion position of the target organ, it receives a first motion control instruction for the robotic arm 303 and sends the first motion control instruction to the remote intervention robot 30 through the first remote communication module 201.

[0029] The second controller 302 is used to receive the first motion control instruction through the second remote communication module 301 and control the robotic arm 303 to execute the first motion control instruction, so that the puncture device 304 moves to the skin needle insertion position and controls the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

[0030] In some exemplary embodiments, as shown in FIG. 1 , the remote interventional robot 30 further includes an ultrasound probe 305 , which is disposed at the end of a robotic arm 303 .

[0031] The ultrasound probe 305 is used to obtain ultrasound signals obtained by the ultrasound probe 305 scanning the target organ.

[0032] The target organ in this example is the target organ corresponding to the target object.

[0033] The target object is a virtual object created for a real-world medical object, which can be a patient (human) or other animal, such as a cat or dog. The target object can be a patient identifier, which can be a patient account number, patient ID (number or identification card number), or patient name.

[0034] The target organ may be any part of the body corresponding to the target object that needs to be treated through interventional surgery. For example, the target organ may be the lungs, heart, liver, etc. This embodiment does not specifically limit this.

[0035] The second controller 302 is further configured to synthesize an ultrasonic image based on the ultrasonic signal and send the ultrasonic image to the remote control terminal via the second remote communication module 301;

[0036] The first controller 202 is further configured to obtain an ultrasound image through the first remote communication module 201 and determine, based on the ultrasound image and the target organ, that the puncture device has not moved to the skin insertion position of the target organ.

[0037] In some exemplary embodiments, the first controller 202 may determine that the puncture device has not moved to the skin needle insertion position of the target organ based on the ultrasound image and the three-dimensional model corresponding to the target organ.

[0038] In this example, the first remote communication module 201 and the second remote communication module 301 can communicate with each other through a mobile communication network. For example, the first remote communication module 201 and the second remote communication module 301 can communicate through the fifth generation mobile communication technology (5th Generation Mobile Communication Technology, referred to as 5G) network to ensure that the remote control terminal 20 and the remote intervention robot can perform low-latency interactive communication.

[0039] It is understandable that after determining that the puncture needle is advanced to the lesion point, the puncture needle may be controlled to perform a corresponding target operation at the lesion point.

[0040] The target operation is pre-set according to actual needs. For example, the target operation may include at least one of the following: biopsy, catheterization, aspiration, drug injection, ablation treatment, etc.

[0041] In the remote intervention robot system provided in an embodiment of the present application, when the remote intervention robot determines that the puncture device has not moved to the skin insertion position of the target organ, it receives a first motion control instruction for the robotic arm based on the fact that the puncture device has not moved to the skin insertion position of the target organ, and sends the first motion control instruction to the remote intervention robot. The remote intervention robot controls the robotic arm to execute the first motion control instruction, so that the puncture device moves to the skin insertion position and controls the puncture needle to advance to the lesion point of the target organ along the needle insertion path. In this way, the remote intervention robot is accurately controlled by the remote control terminal, and the remotely controlled puncture needle is accurately positioned at the lesion point.

[0042] Based on the above embodiment, in order to facilitate professional interventional doctors to remotely control the robotic arm 303, as shown in Figure 1, the remote control terminal 20 of this example also includes a robotic arm control module 203. The first controller 202 is used to receive a first motion control instruction for the robotic arm 303 through the robotic arm control module 203.

[0043] Among them, the robotic arm control module 203 in this example can include a posture sensor, a position sensor and a force sensor, which are used to obtain the 6-dimensional posture data of the professional interventional doctor's posture input, plane coordinate input and vertical position coordinate input, so as to achieve accurate acquisition of the doctor's operation information.

[0044] In some examples, a professional interventional physician may input gestures through an operating handle in the remote control terminal 20. Correspondingly, in order to ensure that the physician's operation information on the remote control terminal 20 can be accurately reproduced by the robotic arm 303, after the second controller 302 receives the first motion control instruction input by the professional interventional physician through the robotic arm control module 203, the second controller 302 may convert the position vector and the gesture rotation matrix in the first motion control instruction based on the coordinate conversion relationship between the operating handle coordinate system and the robotic arm coordinate system to obtain the converted position vector and gesture rotation matrix, respectively. Correspondingly, the robotic arm 303 may be motion-controlled based on the converted position vector and gesture rotation matrix.

[0045] As an example, suppose the position vector and attitude rotation matrix sent to the remote intervention robot are p robot and R robot , then: p robot =R handle_wrt_robot p handle

[0046] Regarding the attitude rotation matrix, since the ultrasonic probe 305 is installed at the zero position of the manipulator 303, the zero position attitude of the ultrasonic probe 305 is consistent with the zero position attitude of the operating handle. This is equivalent to pre-rotating a fixed value in the manipulator coordinate system. This rotation is R handle_wrt_robot , so: R robot R handle_wrt_robot =R handle_wrt_robot R handle

[0047] That is: R robot =R handle_wrt_robot R handle R robot_wrt_handle

[0048] Among them, p in the formula handle and R handle Represent the position and posture of the operating handle output, R handle_wrt_robot Represents the rotation matrix of the operating handle coordinate system 1 relative to the robot arm coordinate system 2, R handle_wrt_robot =R z (-90°)R x (180°); R robot_wrt_handle Represents the rotation matrix of the manipulator coordinate system 2 relative to the operating handle coordinate system 1, R robot_wrt_handle =R x (180°)R z (90°). After the above conversion, the position vector and posture rotation vector of the operating handle can be transferred to the manipulator coordinate system. It should be noted that the manipulator coordinate system in this example can also be called the robot coordinate system.

[0049] In one embodiment of the present application, the remote control terminal 20 further includes a first display module 204, wherein:

[0050] The first controller 202 is specifically configured to process the ultrasound image and the three-dimensional model corresponding to the target organ to include section position information corresponding to the ultrasound image in the three-dimensional model, and perform needle insertion planning based on the section position information to obtain a current needle insertion position of the puncture device 304. If the current needle insertion position does not match the skin needle insertion position, it is determined that the puncture device 304 has not moved to the skin needle insertion position;

[0051] The first display module 204 is used to display a three-dimensional model including the position information of the section corresponding to the ultrasound image. Thus, the first display module 204 visually displays information to the professional interventional physician, facilitating accurate remote control based on the displayed information.

[0052] In some exemplary embodiments, in order to visualize the ultrasound image to professional interventional physicians, the ultrasound image may also be displayed via the first display module 204 .

[0053] In addition, it is understandable that the three-dimensional model including the section position information corresponding to the ultrasound image is visually displayed by the first display module 204, which can make professional interventional doctors feel as if they are actually there.

[0054] In one embodiment of the present application, in order to facilitate professional interventional doctors to conveniently remotely control the ultrasound probe 305, as shown in FIG1 , the remote control terminal 20 in this example further includes an ultrasound control module 205, wherein:

[0055] The ultrasound control module 205 is configured to receive ultrasound control instructions for the ultrasound probe 305 and send the ultrasound control instructions to the first controller 202;

[0056] The first controller 202 is further configured to send ultrasonic control instructions to the second controller 302;

[0057] The second controller 302 is further configured to control the ultrasound probe 305 according to ultrasound control instructions.

[0058] Specifically, the first controller 202 can send the ultrasound control instruction to the second controller 302 through the first remote communication module 201. Correspondingly, the second controller 302 can receive the ultrasound control instruction through the second remote communication module 301 and control the ultrasound probe 305 based on the ultrasound control instruction.

[0059] Based on the above embodiment, during an interventional procedure performed by a remote interventional robot 30, after reaching the skin needle insertion location, the remote control terminal 20 can control the robotic arm 303 to be fixed. The robotic arm 303 can maintain a constant position, posture, and pressure according to commands. In this case, the physician does not need to apply force to the ultrasound probe 305, which can reduce physician fatigue. During the puncture process, if excessive gas in the body causes reduced image clarity, or if the needle insertion path needs to be adjusted, the posture and pressure information of the ultrasound probe 305 need to be readjusted. In this example, the remote control terminal 20 can also include a fine-tuning control module for fine-tuning the robotic arm. Correspondingly, after the professional interventional physician activates the fine-tuning control module, the remote control terminal 20 can send instructions to the remote interventional robot for fine-tuning the remote interventional robot. Correspondingly, upon receiving the instructions, the second controller 302 in the remote interventional robot 30 can control the robotic arm 303 to move within a small space bounded by a pre-set virtual wall, thereby achieving fine-tuning of the position and posture of the ultrasound probe 305. Among them, an example diagram of a virtual wall is marked, as shown in Figure 2. After the ultrasonic control module 205 in the remote control terminal 20 is turned on, the ultrasonic probe 305 can only move within the range of the virtual wall and can move downward along the normal direction to increase the real-time pressure.

[0060] In one embodiment of the present application, in order to enable a professional interventional physician to accurately obtain the current contact force between the ultrasound probe 305 and the target organ during scanning, and to remotely control the robotic arm 303 based on the current contact force, as shown in FIG1 , the remote interventional robot 30 in this example further includes a six-dimensional force sensor 306 , which is fixed to the end of the robotic arm 303 , and the ultrasound probe 305 is fixed to the six-dimensional force sensor 306 via a clamp;

[0061] a six-dimensional force sensor 306 for acquiring the current contact force between the ultrasound probe 305 and the target organ during scanning, and sending the current contact force to the second controller 302;

[0062] The second controller 302 is further configured to send the current contact force to the first controller 202 .

[0063] Specifically, the second controller 302 may send the current contact force to the remote control terminal 20 via the second remote communication module 301 . Correspondingly, the first controller 202 in the remote control terminal 20 may receive the current contact force via the first remote communication module 201 .

[0064] In one embodiment of the present application, the first controller 202 is further configured to, when determining that the current contact force does not reach the preset contact force, determine a second motion control instruction for the robot arm 303 based on the current contact force and the preset contact force, and send the second motion control instruction to the second controller 302;

[0065] The second controller 302 is further configured to control the robotic arm 303 to move accordingly according to the second motion control instruction, so that the contact force between the ultrasound probe 305 and the target organ reaches a preset contact force.

[0066] The preset contact force is a critical value of the contact force preset in the remote control terminal. The value of the preset contact force can be set according to actual needs, and this embodiment does not impose any specific limitation on this.

[0067] In some examples, when the first controller 202 determines that the current contact force is greater than the preset contact force, the first controller 202 may generate a second motion control instruction for controlling the downward movement of the robotic arm 303 based on the difference between the current contact force and the preset contact force, and send the second motion control instruction to the first controller 202. Accordingly, the first controller 202 controls the downward movement of the robotic arm 303 based on the second motion control instruction. In other examples, when the first controller 202 determines that the current contact force is less than the preset contact force, the first controller 202 may generate a second motion control instruction for controlling the upward movement of the robotic arm 303 based on the difference between the preset contact force and the current contact force, and send the second motion control instruction to the first controller 202. Accordingly, the first controller 202 controls the upward movement of the robotic arm 303 based on the second motion control instruction, thereby ensuring that a constant contact force is maintained between the ultrasound probe 305 and the skin of the target organ.

[0068] In other exemplary embodiments, the second controller 302 may also receive the preset contact force sent by the first controller 201 and adjust the robotic arm 303 based on the preset contact force and the current contact force to ensure a constant contact force between the ultrasound probe 305 and the skin of the target organ. Thus, the remote interventional robot can automatically adjust the robotic arm 303 based on the preset contact force and the current contact force to ensure a constant contact force between the ultrasound probe 305 and the skin of the target organ.

[0069] As an example, when the second controller 302 determines that the current contact force is greater than the preset contact force, a second motion control instruction for controlling the downward movement of the robotic arm 303 may be generated based on the difference between the current contact force and the preset contact force, and the robotic arm 303 may be controlled to move downward according to the second motion control instruction. In other examples, when the second controller 302 determines that the current contact force is less than the preset contact force, a second motion control instruction for controlling the upward movement of the robotic arm 303 may be generated based on the difference between the preset contact force and the current contact force, and the robotic arm 303 may be controlled to move upward based on the second motion control instruction, thereby ensuring that a constant contact force is maintained between the ultrasound probe 305 and the skin of the target organ.

[0070] In some exemplary embodiments, the second controller 302 in the ultrasound interventional robot may use a proportional differential (PD) control method to adjust the robotic arm 303 to ensure a constant contact force between the ultrasound probe 305 and the skin of the target organ to ensure the comfort of the patient.

[0071] FIG3 shows an exemplary diagram of the principle of adjusting the robotic arm 303 using a proportional differential (PD) control method in the second controller 302. The following describes an exemplary process of adjusting the robotic arm 303 to ensure a constant contact force between the ultrasound probe 305 and the skin of the target organ in conjunction with FIG3 . The specific process is as follows:

[0072] The preset contact force (or the preset contact force can also be called the target contact force) sent by the remote control terminal 20 is F desire , at the current contact force F z Less than F desire When the robot arm 303 moves forward along the attitude direction, when F z Greater than F desire When the robot arm 303 moves backward in the opposite direction of the posture, it is assumed that the control output of each posture direction position is P control , then: P control =K p E f +K d ΔE f

[0073] Among them, K p and k d are proportional gain and differential gain respectively, E f is the error, ΔE f is the error differential. E f =F desire -F true , ΔE f =E f -E f_previous

[0074] In the contact direction with the patient corresponding to the target object, the ultrasonic interventional robot can also detect the collision force received in real time based on the pressing force corresponding to the professional interventional doctor sent by the remote control terminal 20, and based on the pressing force control and the contact force with the patient. In the non-contact direction, the ultrasonic interventional robot will also detect the collision force received in real time. When the collision force exceeds the set protection force, it will generate an evasive action according to the protection algorithm.

[0075] In one embodiment of the present application, the second controller 302 is also used to obtain preoperative imaging images of the target organ, perform three-dimensional modeling on the preoperative imaging images to obtain a three-dimensional model corresponding to the target organ, and determine the skin needle insertion position and needle insertion path based on the three-dimensional model, and send the skin needle insertion position and needle insertion path to the first controller 202.

[0076] In one embodiment of the present application, the remote intervention robot 30 further includes a depth camera 307, wherein:

[0077] The depth camera 307 is used to obtain a contour image of the target object to which the target organ belongs, and send the contour image to the second controller 302;

[0078] The second controller 302 is further configured to send the contour image to the first controller 202;

[0079] The first controller 202 is further configured to fuse the contour image and the three-dimensional model to obtain a fused image, and display the fused image through the first display module 204 .

[0080] The contour image refers to an image of the patient's body surface corresponding to the target object.

[0081] In some exemplary embodiments, the first controller 202 may fuse the contour image and the three-dimensional model based on the coordinate system based on the depth camera 307 corresponding to the contour image and the organ coordinate system based on the reconstructed three-dimensional model to obtain a fused image, and display the fused image on the first display module 204 in the remote control terminal 20 to visualize the entire surgical process.

[0082] For example, the contour image captured by the depth camera 307 is shown in Figure 4 a, and the three-dimensional model corresponding to the target organ is shown in Figure 4 b. Correspondingly, the fused image obtained by fusing the contour image and the three-dimensional model is shown in Figure 4 c.

[0083] In one embodiment of the present application, during the interventional surgery, the ultrasound image collected by the remote ultrasound machine can also be displayed on the first display module 204, and the section position information corresponding to the ultrasound image can be marked in the three-dimensional model currently displayed in the first display module 204 to facilitate professional interventional doctors to observe the lesion information in real time.

[0084] For example, the content currently displayed in the first display module 204 is shown in Figure a in Figure 5. Correspondingly, assuming that the ultrasound image currently sent by the remote intervention robot 30 is shown in Figure b in Figure 5, the ultrasound image can be displayed in the first display module 204, and an example diagram of the section position information of the ultrasound image can be marked in the three-dimensional model, as shown in Figure c in Figure 5. It should be noted that mark A in Figure c in Figure 5 represents the section position information of the ultrasound image.

[0085] In some exemplary embodiments, in order to enhance the surgeon's real experience, virtual reality (VR) glasses can be used to reconstruct the data information transmitted from the remote intervention robot 30. The surgeon wears the glasses to perform remote operations, thereby improving the authenticity of the surgery.

[0086] In one embodiment of the present application, the remote control terminal 20 also includes a first voice and video module 206, and the remote intervention robot 30 also includes a second voice and video module 308 and a second display module 309, wherein the first voice and video module 206 and the second voice and video module 308 are used for voice and video communication between the remote control terminal 20 and the remote intervention robot 30.

[0087] The second display module 309 is used to display the video data in the audio and video data sent by the second audio and video module 308 in real time.

[0088] It should be noted that both the remote ultrasound intervention machine and the remote control terminal 20 in this example have speakers for playing the audio data in the corresponding audio and video data in real time.

[0089] In one embodiment of the present application, the remote control terminal 20 further includes a third display module, which is configured to display the video data in the audio and video data sent by the first audio and video module 206 in real time.

[0090] In some exemplary embodiments, the remote intervention robot 30 in this example may further include an ultrasound host to synthesize the ultrasound signals output by the ultrasound probe 305 through the ultrasound host and send the synthesized ultrasound image to the second controller 302 .

[0091] Correspondingly, when the remote intervention robot includes a second voice and video module, a second display module, a second controller, an ultrasound host, an ultrasound probe, a robotic arm, a six-dimensional force sensor, and a puncture device, an example diagram of the relationship between the hardware connection and the communication connection between the various components in the second controller and the ultrasound host in the remote intervention robot is shown in Figure 6, where it should be noted that the solid lines in Figure 6 represent hardware connections and the dotted lines represent communication connections.

[0092] The robotic arm in this example may be a six-degree-of-freedom robotic arm.

[0093] The following takes the remote intervention robot system shown in FIG1 as an example to exemplarily describe the control method of the remote intervention robot.

[0094] Figure 7 is a flow chart of a control method for a remote intervention robot according to an embodiment of the present application. It should be noted that the control method for a remote intervention robot provided in this embodiment is applied to the first controller of the remote control terminal. In other words, the control method for the remote intervention robot in this example is executed by the first controller of the remote control terminal.

[0095] As shown in FIG7 , the control method of the remote intervention robot may include:

[0096] Step 701: Determine that the puncture device has not moved to the skin insertion position of the target organ.

[0097] In some examples, the current position of the puncture device may be obtained, and if it is determined that the current position is inconsistent with the skin needle insertion position, it is determined that the puncture device has not moved to the skin needle insertion position of the target organ.

[0098] Step 702: Based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ, a first motion control instruction for the robotic arm is received.

[0099] In step 703, the first motion control instruction is sent to the remote intervention robot through the first remote communication module, so that the second controller controls the robotic arm to move the puncture device to the skin needle insertion position according to the first motion control instruction, and controls the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

[0100] It should be noted that the aforementioned explanation of the embodiment of the remote intervention robot system is also applicable to the explanation of the control method of the remote intervention robot in this embodiment, and this embodiment does not specifically limit this.

[0101] The control method of the remote intervention robot provided in an embodiment of the present application is as follows: when the remote intervention robot determines that the puncture device has not moved to the skin needle insertion position of the target organ, it receives a first motion control instruction for the robotic arm based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ, and sends the first motion control instruction to the remote intervention robot. The remote intervention robot controls the robotic arm to execute the first motion control instruction, so that the puncture device moves to the skin needle insertion position and controls the puncture needle to advance to the lesion point of the target organ along the needle insertion path. In this way, the remote intervention robot is accurately controlled by the remote control terminal, and the remote control puncture needle is accurately reached to the lesion point, which facilitates the subsequent target operation performed on the lesion at the lesion point by the puncture needle.

[0102] In some exemplary embodiments, one possible implementation method for determining that the puncture device has not moved to the skin needle insertion position of the target organ is: obtaining an ultrasound image of the target organ, and determining that the puncture device has not moved to the skin needle insertion position of the target organ based on the ultrasound image and the target organ.

[0103] In some examples, one possible implementation method for determining that the puncture device has not moved to the skin needle insertion position of the target organ based on the ultrasound image and the target organ is: processing the three-dimensional model corresponding to the ultrasound image and the target organ to include the section position information corresponding to the ultrasound image in the three-dimensional model; performing needle insertion planning based on the section position information to obtain the current needle insertion position of the puncture device; and determining that the puncture device has not moved to the skin needle insertion position when the current needle insertion position does not match the skin needle insertion position.

[0104] In some exemplary embodiments, the remote intervention robot also includes an ultrasound probe, which is arranged at the end of the robotic arm to obtain an ultrasound image of the target organ, including: obtaining an ultrasound image through a first remote communication module, wherein the ultrasound image is obtained by synthesizing the ultrasound signal obtained by scanning the target organ with the ultrasound probe, and is sent by the second controller through the second remote communication module.

[0105] In some exemplary embodiments, the remote control terminal may further include a first display module. To visually display information to the interventionalist and facilitate accurate remote control based on the displayed information, the method of this embodiment may further include controlling the first display module to display a three-dimensional model containing the slice position information corresponding to the ultrasound image. This facilitates visual viewing of the three-dimensional model containing the slice position information corresponding to the ultrasound image through the display module, facilitating subsequent accurate control of the remote ultrasound robot by the physician based on the displayed information.

[0106] In some exemplary embodiments, in order to facilitate professional interventional doctors to remotely control the robotic arm, when the remote control end may also include a robotic arm control module, the method may further include: receiving a first motion control instruction for the robotic arm through the robotic arm control module.

[0107] Based on the above embodiment, to facilitate professional interventional physicians to control the ultrasound probe, if the remote control terminal also includes an ultrasound control module, the method may further include: receiving, via the ultrasound control module, ultrasound control instructions for the ultrasound probe, and sending the ultrasound control instructions to the second controller. This facilitates professional interventional physicians to remotely control the ultrasound probe via the ultrasound control module.

[0108] In some exemplary embodiments, in order to facilitate professional interventional physicians to obtain the current contact force between the ultrasound probe and the target organ during scanning, when the remote interventional robot further includes a six-dimensional force sensor, the six-dimensional force sensor is fixed to the end of the robotic arm, and the ultrasound probe is fixed to the six-dimensional force sensor via a clamp, the method may further include: receiving the current contact force sent by the second controller, wherein the current contact force is obtained by the six-dimensional force sensor detecting the contact force between the ultrasound probe and the target organ during scanning and sending it to the second controller. In this way, the remote control end can obtain the current contact force between the ultrasound probe and the target organ during scanning through the information sent by the remote interventional robot.

[0109] Based on the above embodiments, in order to remotely adjust the contact force between the ultrasound probe and the target organ during scanning through the remote control end so that the current contact force between the ultrasound probe and the target organ during scanning reaches the preset contact force, the method may also include: when it is determined that the current contact force has not reached the preset contact force, determining a second motion control instruction for the robotic arm based on the current contact force and the preset contact force, and sending the second motion control instruction to the second controller, wherein the second motion control instruction is used to instruct the robotic arm to move accordingly so that the contact force between the ultrasound probe and the target organ reaches the preset contact force.

[0110] In some exemplary embodiments, the method may further include: receiving a skin needle insertion position and a needle insertion path sent by a second controller, wherein the skin needle insertion position and the needle insertion path are determined by the second controller based on a three-dimensional model corresponding to the target organ, wherein the three-dimensional model is obtained by three-dimensional modeling of a preoperative image of the target organ.

[0111] In order to visualize the entire surgical process and facilitate professional interventional doctors to understand the situation visually, the method of this embodiment may also include: receiving a contour image of the target object sent by the second controller, wherein the contour image is obtained by capturing an image of the target object by a depth camera; fusing the contour image and the three-dimensional model to obtain a fused image, and displaying the fused image through the first display module.

[0112] Based on the above embodiments, in order to enable audio and video calls with the remote intervention robot and facilitate real-time understanding of the situation at the surgical site, the remote control end in this example also includes a first voice and video module, and the remote intervention robot also includes a second voice and video module and a second display module. The method also includes: conducting voice and video communication with the second voice and video module through the first voice and video module.

[0113] The second display module is used to display the video data in the audio and video data sent by the first audio and video module in real time.

[0114] In some exemplary embodiments, the remote control terminal in this example further includes a third display module, and the method further includes: displaying, in real time, the video data in the audio and video data transmitted by the first audio and video module via the third display module. Thus, by displaying the video data in real time via the third display module, the professional interventional physician can promptly understand the situation at the surgical site and remotely control the remote interventional robot accordingly.

[0115] The following takes the remote intervention robot system shown in FIG1 as an example to exemplarily describe a method for controlling the remote intervention robot through a remote control terminal.

[0116] FIG8 is a flow chart illustrating a method for controlling a remote interventional robot via a remote control terminal according to one embodiment of the present application. It should be noted that the method for controlling a remote interventional robot via a remote control terminal provided in this embodiment is applied to the second controller of the remote interventional robot. In other words, the execution entity of the method for controlling a remote interventional robot via a remote control terminal in this example is the second controller of the remote interventional robot.

[0117] As shown in FIG8 , the method for controlling the remote intervention robot through the remote control terminal may include:

[0118] Step 801: Receive a first motion control instruction through a second remote communication module.

[0119] Among them, the first motion control instruction is sent by the first controller through the first remote communication module, wherein the first motion control instruction is the first motion control instruction for the robotic arm received by the first controller based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ when the first controller determines that the puncture device has not moved to the skin needle insertion position of the target organ.

[0120] Step 802 , controlling the robotic arm to execute a first motion control instruction, so that the puncture device moves to the skin needle insertion position, and controls the puncture needle to advance to the lesion point of the target organ along the needle insertion path.

[0121] It should be noted that the aforementioned explanation of the remote intervention robot system is also applicable to the embodiment of the method for controlling the remote intervention robot through a remote control terminal in this embodiment, and this embodiment does not specifically limit this.

[0122] The present application provides a method for controlling a remote intervention robot through a remote control terminal. When the remote intervention robot determines that the puncture device has not moved to the skin needle insertion position of the target organ, the remote intervention robot receives a first motion control instruction for the robotic arm based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ, and sends the first motion control instruction to the remote intervention robot. The remote intervention robot controls the robotic arm to execute the first motion control instruction, so that the puncture device moves to the skin needle insertion position and controls the puncture needle to advance to the lesion point of the target organ along the needle insertion path. In this way, the remote intervention robot is accurately controlled by the remote control terminal, and the remote control puncture needle is accurately reached to the lesion point, which facilitates the subsequent execution of the target operation on the lesion at the lesion point by the puncture needle.

[0123] Based on the above embodiment, in order to facilitate professional doctors to obtain ultrasound images of the target organ in a timely manner, the remote intervention robot also includes an ultrasound probe, which is arranged at the end of the robotic arm. The method may also include: synthesizing an ultrasound image based on an ultrasound signal, wherein the ultrasound signal is obtained by scanning the target organ with the ultrasound probe; and sending the ultrasound image to the first controller via a second remote communication module. As a result, the remote intervention device also has an ultrasound detection function and can promptly send the ultrasound image corresponding to the target organ to the remote control terminal, so that professional intervention doctors can further accurately remotely control the remote intervention device based on the ultrasound image sent by the remote intervention device, further improving the accuracy of the interventional surgery.

[0124] Based on the above embodiments, when the remote control end also includes an ultrasonic control module, the method also includes: receiving an ultrasonic control instruction sent by the first controller, wherein the ultrasonic control instruction is an ultrasonic control instruction for the ultrasonic probe received by the first controller through the ultrasonic control module; and controlling the ultrasonic probe according to the ultrasonic control instruction.

[0125] In one embodiment of the present application, the remote intervention robot also includes a force sensor, which can be a six-dimensional force sensor. The six-dimensional force sensor is fixed at the end of the robotic arm, and the ultrasound probe is fixed on the six-dimensional force sensor through a clamp. The method also includes: obtaining the current contact force between the ultrasound probe and the target organ during scanning through the six-dimensional force sensor, and sending the current contact force to the first controller.

[0126] In one embodiment of the present application, the method further includes: receiving a second motion control instruction for the robotic arm sent by the first controller, wherein the second motion control instruction is determined based on the current contact force and the preset contact force when it is determined that the current contact force has not reached the preset contact force; and controlling the robotic arm to move accordingly based on the second motion control instruction so that the contact force between the ultrasound probe and the target organ reaches the preset contact force. Thus, based on the motion control instruction from the remote control terminal, the contact force between the ultrasound probe and the target organ is timely adjusted so that the contact force between the ultrasound probe and the target organ reaches the preset contact force.

[0127] In other exemplary embodiments, the method may also include: receiving a preset contact force sent by the first controller, and when it is determined that the current contact force has not reached the preset contact force, controlling the robotic arm to move accordingly based on the current contact force and the preset contact force so that the contact force between the ultrasound probe and the target organ reaches the preset contact force.

[0128] In one embodiment of the present application, the method also includes: acquiring a preoperative imaging image of the target organ; performing three-dimensional modeling on the preoperative imaging image to obtain a three-dimensional model corresponding to the target organ; determining the skin needle insertion position and needle insertion path based on the three-dimensional model; and sending the skin needle insertion position and needle insertion path to the first controller.

[0129] In one embodiment of the present application, the remote intervention robot further includes a depth camera, wherein: a contour image of a target object to which the target organ belongs is acquired by the depth camera, and the contour image is sent to the first controller.

[0130] In one embodiment of the present application, the remote control terminal further includes a first voice and video module, the remote intervention robot further includes a second voice and video module and a third display module, and the method further includes: performing voice and video communication with the first voice recognition module through the second voice and video module;

[0131] The video data in the audio and video data sent by the second audio and video module is displayed in real time through the third display module.

[0132] In order to clearly understand the present application, the interaction process between the remote control terminal and the remote intervention robot is exemplarily described below with reference to FIG9 .

[0133] FIG9 is a diagram illustrating an example of the interaction process between a remote control terminal and a remote intervention robot according to an embodiment of the present application.

[0134] As shown in Figure 9, it may include:

[0135] In step 901 , a remote interventional robot obtains a preoperative image of a target organ of a target subject.

[0136] The preoperative imaging image in this example may be a computed tomography (CT) image and / or a magnetic resonance imaging (MRI) image of the target organ.

[0137] In step 902 , the remote interventional robot performs three-dimensional reconstruction based on the preoperative imaging images to obtain a three-dimensional model of the target organ.

[0138] In some exemplary embodiments, when the preoperative images are CT or MRI, segmentation of the preoperative images using a 3D convolutional neural network (V-Net or 3D UNet) can fully leverage information from all dimensions to accurately identify regions of interest (ROI). For example, in liver tumor surgery, ROIs include the tumor, liver, skin, and vital structures such as the heart, lungs, ribs, and blood vessels, creating a challenging multi-label segmentation task.

[0139] Because 3D volume data consumes a large amount of memory, implementing the aforementioned multi-label segmentation requires extremely high computing power. Therefore, in this example, a step-by-step segmentation approach, from coarse to fine, from global to local, is employed to segment each region of interest one by one. First, based on the volume data size and available computing power, the 3D volume data is significantly downsampled until the computing power supports segmentation of the entire volume data using a 3D convolutional neural network. This step determines the approximate location of each region of interest within the 3D volume data. Based on this, the smallest cuboid containing each region of interest is determined, and voxels in this region are extracted for the next step of fine-grained segmentation. Second, since each extracted cuboid corresponds to a specific region of interest, the next step of fine-grained segmentation can be simplified to a binary segmentation problem. The fine-grained segmentation step segments the volume data at its original scale, eliminating any missing data. This allows for more accurate identification of the edges of the region of interest, achieving accurate segmentation, within the constraints of available computing power. After the fine-grained segmentation is complete, the corresponding cuboid mask is restored to its corresponding position, ultimately obtaining an accurate segmentation mask that is the same size as the 3D volume data.

[0140] After completing the multi-label segmentation task, 3D reconstruction of liver CT image data was achieved using the VTK library. This project employed surface rendering techniques and the Marching Cubes algorithm to extract isosurfaces for accurate segmentation masks. Furthermore, pseudo-color parameters for each region of interest were appropriately set to produce realistic 3D reconstruction results with realistic morphology and color.

[0141] In step 903 , the remote interventional robot performs needle insertion analysis based on the three-dimensional model to obtain the number and positions of needles.

[0142] In some examples, during the needle insertion analysis using a three-dimensional model, the lesion area of ​​the target organ can be determined based on the three-dimensional model, and needle insertion analysis can be performed based on the lesion area to obtain the number and location of needle insertions.

[0143] As an example, in the case of ablation surgery performed by a remote interventional robot, the thermal field generated by the lesion area and the puncture needle can be analyzed. Specifically, the thermal field generated by the puncture needle can be simulated by a cylinder. This can be achieved by solving the following optimization problem:

[0144] The following constraints must be met

[0145] in,

[0146] I, J, and K are the needle target position set, potential needle insertion position set, and lesion area point set, respectively;

[0147] i, j, k are the indices of sets I, J, K respectively;

[0148] NK is the cardinality of set K;

[0149] M is a large constant;

[0150] is a binary parameter whose value is 1 if K(k) is in the cylindrical ablation zone. The axis of the cylinder is along the path determined by the target position I(i) and the needle position J(j), and its radius is about 1 mm smaller than the radius of a single ablation to ensure full coverage of the lesion area.

[0151] Is a binary parameter that takes the value 1 if K(k) is in the cylindrical ablation zone. The axis of the cylinder is along the path determined by the target position I(i) and the needle position J(j), and its radius is l c (4mm-6mm, about 2 to 3 times the diameter of the needle).

[0152] is a binary parameter, which takes the value 1 if K(k) is in the ellipsoidal ablation zone. The center of the ellipsoid is I(i) and the circular symmetry axis is along the path determined by the target position I(i) and the needle position J(j);

[0153] P ij An integer score determined by the target position I(i) and the needle insertion position J(j). If the path does not meet the conditions, it is 0. A higher score indicates a higher fitness.

[0154] L i is the distance from I(i) to the centroid of set K.

[0155] sgn is the sign function.

[0156] The above optimization problem can obtain the minimum number of needles and their paths so that the generated thermal field can cover the entire lesion area.

[0157] In some exemplary embodiments, in order to determine the optimal ablation position and number of ablations for ablation in the lesion area, the following optimization problem may be solved:

[0158] The following constraints must be met

[0159] Not parallel to any path in T

[0160] T is the path identified in the first step. Therefore, the potential optimal ablation locations are points generated equidistantly along these paths.

[0161] In step 904 , the remote interventional robot sends the three-dimensional model, the number of needles, and the needle insertion positions to the remote control terminal.

[0162] Step 905 : When the patient corresponding to the target object is lying on a bed, the remote intervention robot obtains a contour image of the patient's body surface through a depth camera.

[0163] Step 906: The remote intervention robot sends the contour image to the remote control terminal.

[0164] In step 907, the remote control terminal matches the three-dimensional model with the contour image, and displays the matching result on the display module, and displays the needle insertion position and needle insertion posture on the display module.

[0165] Step 908: The remote control terminal receives a first motion control instruction input by the doctor for the robotic arm of the remote intervention robot.

[0166] Step 909: The remote intervention robot controls the robotic arm to move accordingly according to the first motion control instruction.

[0167] It should be understood that the end of the robotic arm in this example is equipped with an ultrasonic probe, and accordingly, the movement of the robotic arm can drive the movement of the ultrasonic probe.

[0168] In step 910 , the remote intervention robot obtains an ultrasound image of the patient's target organ and sends the ultrasound image to the remote control terminal.

[0169] Specifically, the ultrasound host of the remote intervention robot obtains the ultrasound signal output by the ultrasound probe when scanning the target organ, synthesizes the ultrasound signal to obtain an ultrasound image of the target organ, and sends the ultrasound image to the remote control end.

[0170] Step 911: The remote control terminal obtains the ultrasound image sent by the remote intervention robot.

[0171] In step 912 , the remote control terminal displays the section position information of the ultrasound image on the three-dimensional model of the display module in real time, and displays the ultrasound image on the display module.

[0172] In step 913 , the remote control terminal performs needle insertion planning on the ultrasound image to determine the current needle insertion position of the puncture device of the remote intervention robot.

[0173] Specifically, the artificial intelligence on the remote control end analyzes the ultrasound image to obtain information about blood vessels and lesions in the ultrasound image, and performs needle insertion planning based on the information about blood vessels and lesions to determine the current needle insertion position of the puncture device of the remote interventional robot.

[0174] In step 914 , the remote control end determines whether the current needle insertion position has reached the corresponding needle insertion position. If so, a fixed instruction is sent to the remote intervention robot, wherein the fixed instruction is used to instruct the remote intervention robot to control the robotic arm to be stationary. If not, jump to step 908 .

[0175] In step 915 , the remote intervention robot controls the robotic arm to be still according to fixed instructions, and maintains a constant posture and pressure to ensure a clear ultrasound image.

[0176] Step 916: The remote intervention robot controls the puncture device to insert the needle.

[0177] Step 917: During the needle insertion process, the remote interventional robot uses artificial intelligence algorithms to intelligently avoid blood vessels and other tissues.

[0178] In some exemplary embodiments, during ultrasound interventional ablation surgery performed by a remote interventional robot, in order to achieve real-time identification of sensitive organs such as blood vessels and areas of interest during the surgery and perform precise intraoperative navigation, in this example, the remote interventional robot can construct an automatic segmentation artificial intelligence model of blood vessels and lesion areas based on ultrasound images and a real-time tracking algorithm for ablation needle during the surgery, thereby achieving automatic identification and judgment of organs such as blood vessels and lesion areas in the intervention path during the scanning process of the remote interventional robot, and ultimately achieving an intelligent early warning function for the entire puncture intervention surgical process.

[0179] In some examples, to accurately segment blood vessels and lesions, a remote interventional robot can employ a bilaterally symmetrical U-shaped deep network structure to perform end-to-end supervised learning on the blood vessel and lesion labels provided by medical experts, resulting in a model capable of making dense predictions for unseen input images. Furthermore, the network employs feature fusion, aggregating semantic representations at different levels through a hierarchical connection structure to achieve better segmentation results.

[0180] In some examples, the puncture needle can also be tracked in real time based on the ultrasound image. The real-time tracking problem of the puncture needle can be decomposed into three steps: 1. Automatic identification of the puncture needle in the ultrasound image; 2. Axial estimation of the puncture needle (including direction and insertion point); 3. Positioning the needle tip of the puncture needle. In some examples, in order to accurately track the puncture needle in real time during surgery, a deep convolutional neural network framework based on automatic recognition tasks can be used. The framework consists of a modified version of Faster-RCNN (Region Convolutional Neural Networks) with a lightweight model as the backbone network and a region proposal network. The ultrasound image as the network input is passed through the lightweight backbone network to generate a feature map, and then through the region proposal network to generate a candidate region. Finally, the bounding box of the puncture needle in the image is output through classification and regression.

[0181] In step 918 , the remote control terminal determines whether the ultrasound probe needs to be fine-tuned. If so, the remote control terminal inputs a second motion control instruction for the robotic arm through the fine-tuning control module and sends the second motion control instruction to the remote intervention robot.

[0182] In step 919 , the remote intervention robot controls the robotic arm to make minute movements based on the second motion control instruction, obtains a clear ultrasound image, and sends the ultrasound image to the remote control terminal.

[0183] In step 920 , after determining that the ultrasound image is clear, the remote control terminal turns off the fine-tuning control module and sends an instruction to the remote intervention robot for controlling the robotic arm to be still.

[0184] In step 921 , the remote ultrasound robot controls the robotic arm to be stationary based on the instruction, maintains a constant position, posture and pressure, and ensures a clear ultrasound image.

[0185] In step 922 , the remote control terminal determines whether the puncture device has reached the lesion location. If so, step 923 is executed.

[0186] In step 923 , the remote control terminal controls the remote intervention robot to perform ablation.

[0187] In step 924 , the remote interventional robot determines whether all lesions have been ablated. If so, the operation is completed.

[0188] It should be noted that if the lesion is not completely ablated, the puncture needle in the puncture device is brought to the corresponding lesion position by the robotic arm according to the motion control instructions of the remote control end for the robotic arm, and the puncture needle is controlled to continue ablation treatment at the corresponding lesion position.

[0189] In this example, before an interventional procedure, a 3D reconstruction of the patient's target organ, obtained from a computed tomography (CT) or magnetic resonance imaging (MRI) image, is performed to create a 3D model of the target organ, facilitating observation by the physician. Using an artificial intelligence (AI) algorithm, the 3D model of the target organ is used to intelligently plan the number, location, and path of needle insertion, ultimately determining the optimal surgical plan.

[0190] Furthermore, the remote interventional robot acquires real-time patient information and, using augmented reality technology, overlays the displayed scene with a 3D reconstructed model, enhancing the visualization of the surgical scene. During the procedure, the remote doctor uses the system to control the robot's robotic arm, performing real-time ultrasound scanning to locate the lesion and controlling the arm to maintain a stable posture and output pressure, facilitating the operation.

[0191] If minor adjustments are needed during surgery, the remote control unit features a fine-tuning module that allows the surgeon to adjust the position, posture, and pressure of the device, ensuring clear ultrasound image information. During surgery, an AI-powered algorithm that integrates ultrasound images identifies lesion and blood vessel locations in real time, providing real-time guidance on needle tip position and enhancing surgical safety.

[0192] According to an embodiment of the present application, the present application further provides a remote control terminal, which is used to control a remote intervention robot. The remote control terminal includes a first controller and a first remote communication module, wherein the remote intervention robot includes a second remote communication module, a second controller, a robotic arm, an ultrasound probe, and a puncture device, wherein the puncture device and the ultrasound probe are arranged at the end of the robotic arm, and the puncture device is provided with a puncture needle, wherein:

[0193] In some examples, the remote control terminal may further include ROM (Read-Only Memory) and RAM (Random Access Memory). The first controller, the first remote communication module, the ROM, and the RAM are connected to each other via a bus. An I / O (Input / Output) interface is also connected to the bus.

[0194] Multiple components in the remote control terminal are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a magnetic disk, an optical disk, etc.; and a first remote communication module, such as a network card, a modem, a wireless communication transceiver, etc. The first remote communication module allows the remote control terminal to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0195] In some examples, the first controller can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the first controller include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphic Processing Units), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, DSPs (Digital Signal Processors), etc. The first controller performs the various methods and processes described above, such as the control method for the remote interventional robot. For example, in some embodiments, the control method for the remote interventional robot can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device via the remote control terminal's ROM (Read-Only Memory) and / or communication unit. When the computer program is loaded into the remote control terminal's RAM (Random Access Memory) and executed by the first controller, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the first controller may be configured to execute the aforementioned method in any other appropriate manner (for example, by means of firmware).

[0196] It should be noted that, for the relevant description of the remote control terminal, please refer to the relevant description of the embodiments of the present application, which will not be repeated here.

[0197] According to an embodiment of the present application, the present application also provides a remote intervention robot, which is controlled by a remote control end, and the remote control end includes a first remote communication module and a first controller. The remote intervention robot includes a second remote communication module, a second controller, a robotic arm, an ultrasound probe and a puncture device, wherein the puncture device and the ultrasound probe are arranged at the end of the robotic arm, and a puncture needle is provided on the puncture device, wherein the second controller is used to implement a method of controlling the remote intervention robot through the remote control end.

[0198] In some examples, the tele-interventional robot may further include a ROM (Read-Only Memory) and a RAM (Random Access Memory). The second controller, the second tele-communication module, the ROM, and the RAM are connected to one another via a bus. An I / O (Input / Output) interface is also connected to the bus.

[0199] Multiple components in the tele-interventional robot are connected to the I / O interface, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as magnetic disks, optical disks, etc.; and a second tele-communication module, such as a network card, a modem, a wireless communication transceiver, etc. The second tele-communication module allows the tele-interventional robot to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0200] In some examples, the second controller can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the second controller include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphic Processing Units), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, DSPs (Digital Signal Processors), etc. The second controller performs the various methods and processes described above, such as controlling the remote interventional robot via a remote control terminal. For example, in some embodiments, the control method of the remote interventional robot can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the remote interventional robot via the ROM (Read-Only Memory) and / or communication unit of the remote interventional robot. When the computer program is loaded into the RAM (Random Access Memory) of the remote interventional robot and executed by the second controller, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the second controller may be configured to execute the aforementioned method in any other appropriate manner (for example, by means of firmware).

[0201] Among them, it should be noted that for the relevant description of the remote intervention robot, please refer to the relevant description of the embodiments of this application, which will not be repeated here.

[0202] On the other hand, an embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the control method of the remote intervention robot disclosed in the embodiment of the present application and the method of controlling the remote intervention robot through a remote control terminal.

[0203] It should be noted that the various embodiments herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which may be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0204] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0205] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0206] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0207] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0208] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A remote intervention robot system, characterized in that: The remote intervention robot system comprises a remote control terminal and a remote intervention robot, wherein the remote control terminal comprises a first remote communication module and a first controller, and the remote intervention robot comprises a second remote communication module, a second controller, a mechanical arm and a puncture device, wherein the puncture device is arranged at the end of the mechanical arm, and a puncture needle is arranged on the puncture device, wherein: The first controller is used to determine that the puncture device has not moved to the skin needle insertion position of the target organ; Based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ, receiving a first motion control instruction for the robotic arm, and sending the first motion control instruction to the remote intervention robot through the first remote communication module; The second controller is used to receive the first motion control instruction through the second remote communication module and control the robotic arm to execute the first motion control instruction, so that the puncture device moves to the skin needle insertion position and controls the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

2. The system according to claim 1, characterized in that The remote intervention robot further comprises an ultrasound probe, which is arranged at the end of the mechanical arm, wherein: The ultrasound probe is used to obtain an ultrasound signal obtained by scanning the target organ by the ultrasound probe; The second controller is further used to synthesize an ultrasonic image according to the ultrasonic signal, and send the ultrasonic image to the remote control terminal through the second remote communication module; The first controller is further used to obtain the ultrasound image through the first remote communication module, and determine, based on the ultrasound image and the target organ, that the puncture device has not moved to the skin needle insertion position of the target organ.

3. The system according to claim 2, characterized in that The remote control terminal also includes a first display module, wherein: The first controller is specifically used to process the ultrasound image and the three-dimensional model corresponding to the target organ, so as to include the section position information corresponding to the ultrasound image in the three-dimensional model, and perform needle insertion planning according to the section position information to obtain the current needle insertion position of the puncture device, and determine that the puncture device has not moved to the skin needle insertion position when the current needle insertion position does not match the skin needle insertion position; The first display module is used to display a three-dimensional model including the section position information corresponding to the ultrasound image.

4. The system according to claim 2, characterized in that The remote intervention robot further comprises a six-dimensional force sensor, which is fixed at the end of the mechanical arm, and the ultrasound probe is fixed on the six-dimensional force sensor through a clamp; The six-dimensional force sensor is used to obtain the current contact force between the ultrasound probe and the target organ during scanning, and send the current contact force to the second controller; The second controller is further configured to send the current contact force to the first controller.

5. The system according to claim 4, characterized in that in, The first controller is further configured to determine, when it is determined that the current contact force does not reach the preset contact force, a second motion control instruction for the robot arm according to the current contact force and the preset contact force, and send the second motion control instruction to the second controller; The second controller is further configured to control the robot arm to move accordingly according to the second motion control instruction, so that the robot arm The contact force between the ultrasound probe and the target organ reaches the preset contact force.

6. The system according to claim 1, wherein: in, The second controller is also used to obtain preoperative imaging images of the target organ, perform three-dimensional modeling on the preoperative imaging images to obtain a three-dimensional model corresponding to the target organ, and determine the skin needle insertion position and the needle insertion path based on the three-dimensional model, and send the skin needle insertion position and the needle insertion path to the first controller.

7. The system according to claim 3, characterized in that The remote intervention robot also includes a depth camera, wherein: The depth camera is used to obtain a contour image of a target object to which the target organ belongs, and send the contour image to the second controller; The second controller is further used to send the contour image to the first controller; The first controller is further used to fuse the contour image and the three-dimensional model to obtain a fused image, and display the fused image through the first display module.

8. The system of claim 1, wherein: The remote control terminal further includes a first voice and video module, and the remote intervention robot further includes a second voice and video module and a second display module, wherein the first voice and video module and the second voice and video module are used for voice and video communication between the remote control terminal and the remote intervention robot; The second display module is used to display the video data in the audio and video data sent by the second audio and video module in real time.

9. The system according to claim 8, characterized in that The remote control terminal also includes a third display module, and the third display module is used to display the video data in the audio and video data sent by the first audio and video module in real time.

10. A control method for a remote intervention robot, characterized in that: The method is applied in a first controller of a remote control terminal, the remote control terminal further comprises a first remote communication module, the remote intervention robot comprises a second remote communication module, a second controller, a mechanical arm and a puncture device, wherein the puncture device is arranged at the end of the mechanical arm, and a puncture needle is arranged on the puncture device, wherein the method comprises: Determining that the puncture device has not moved to the skin insertion position of the target organ; Based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ, receiving a first motion control instruction for the robotic arm; The first motion control instruction is sent to the remote intervention robot through the first remote communication module, so that the second controller controls the robotic arm to move the puncture device to the skin needle insertion position according to the first motion control instruction, and controls the puncture needle to advance to the lesion point position of the target organ along the needle insertion path.

11. The method according to claim 10, characterized in that The remote intervention robot further includes an ultrasound probe, which is disposed at the end of the robotic arm. The step of determining that the puncture device has not moved to the skin needle insertion position of the target organ includes: Acquiring an ultrasound image through the first remote communication module, wherein the ultrasound image is obtained by synthesizing ultrasound signals obtained by scanning a target organ with the ultrasound probe, and is sent by the second controller through the second remote communication module; According to the ultrasound image and the target organ, it is determined that the puncture device has not moved to the skin needle insertion position of the target organ.

12. The method according to claim 11, characterized in that The remote control terminal further includes a first display module, and the step of determining that the puncture device has not moved to the skin needle insertion position of the target organ according to the ultrasound image and the target organ includes: Processing the ultrasound image and the three-dimensional model corresponding to the target organ to include the section position information corresponding to the ultrasound image in the three-dimensional model; Performing needle insertion planning according to the section position information to obtain the current needle insertion position of the puncture device; In the case where the current needle insertion position does not match the skin needle insertion position, determining that the puncture device has not moved to the skin needle insertion position; The method further comprises: The first display module is controlled to display a three-dimensional model including section position information corresponding to the ultrasound image.

13. The method according to claim 11, characterized in that The remote intervention robot further includes a six-dimensional force sensor, the six-dimensional force sensor is fixed to the end of the mechanical arm, the ultrasound probe is fixed to the six-dimensional force sensor by a clamp, and the method further includes: The current contact force sent by the second controller is received, wherein the current contact force is obtained by the six-dimensional force sensor detecting the contact force between the ultrasound probe and the target organ during scanning, and is sent to the second controller.

14. The method according to claim 11, characterized in that The method further comprises: Receive the skin needle insertion position and the needle insertion path sent by the second controller, wherein the skin needle insertion position and the needle insertion path are determined by the second controller based on a three-dimensional model corresponding to the target organ, wherein the three-dimensional model is obtained by three-dimensional modeling of a preoperative image of the target organ.

15. A method for controlling a remote intervention robot via a remote control terminal, characterized in that: The remote control end includes a first remote communication module and a first controller, the remote intervention robot includes a second remote communication module, a second controller, a mechanical arm and a puncture device, wherein the puncture device is arranged at the end of the mechanical arm, and a puncture needle is arranged on the puncture device, wherein the method is applied in the second controller, and the method includes: Receiving a first motion control instruction through the second remote communication module, wherein the first motion control instruction is sent by the first controller through the first remote communication module, wherein the first motion control instruction is received by the first controller for the robot arm based on the fact that the puncture device has not moved to the skin needle insertion position of the target organ when it is determined that the puncture device has not moved to the skin needle insertion position of the target organ; The robot arm is controlled to execute the first motion control instruction so that the puncture device moves to the skin needle insertion position, and the puncture needle is controlled to advance to the lesion point position of the target organ according to the needle insertion path.

16. The method according to claim 15, characterized in that The remote intervention robot further includes an ultrasound probe, which is disposed at the end of the mechanical arm. The method further includes: synthesizing an ultrasound image according to an ultrasound signal, wherein the ultrasound signal is obtained by scanning the target organ with the ultrasound probe; The ultrasound image is transmitted to the first controller through the second remote communication module.

17. The method according to claim 15, characterized in that in, The method further comprises: Acquiring a preoperative image of the target organ; Performing three-dimensional modeling on the preoperative image to obtain a three-dimensional model corresponding to the target organ; Based on the three-dimensional model, determining the skin needle insertion position and the needle insertion path; The skin needle insertion position and the needle insertion path are sent to the first controller.

18. A remote control terminal, characterized in that: The remote control end is used to control the remote intervention robot, and the remote control end includes a first controller and a first remote communication module, wherein the remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is arranged on the puncture device, wherein the first controller is used to execute the method as described in any one of claims 10-14.

19. A remote intervention robot, characterized in that: The remote intervention robot is controlled by a remote control end, which includes a first remote communication module and a first controller. The remote intervention robot includes a second remote communication module, a second controller, a robotic arm and a puncture device, wherein the puncture device is arranged at the end of the robotic arm, and a puncture needle is arranged on the puncture device, wherein the second controller is used to execute the method as described in any one of claims 15-17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 10 to 14 or the method according to any one of claims 15 to 17 is implemented.

Images