Surgical robot and method and control device for guiding movement of surgical arm thereof

Abstract

The present invention relates to a surgical robot and a method and control device for guiding the movement of its surgical arms. The surgical robot has multiple operating arms, each comprising a camera arm with an imaging end-device and a surgical arm with an operating end-device. The method comprises: obtaining an initial position of the operating end-device; obtaining a target position that the operating end-device is expected to reach; generating a guide path extending from the initial position to the target position based on the field of view of the imaging end-device; and adjusting the movement of the operating end-device along the guide path from the initial position to the target position. The surgical robot of the present invention can ensure the safety and reliability of surgery.

Description

[0001] This application is a divisional application filed with the China Patent Office on January 6, 2021, with application number CN202110011216.8 and application name “Surgical robot and method and control device for guiding the movement of surgical arms thereof”. The full text of the case is incorporated into this application by reference. Technical Field

[0002] The present invention relates to the field of medical devices, and in particular to a surgical robot and a method and a control device for guiding the movement of a surgical arm thereof. Background Art

[0003] Minimally invasive surgery refers to a procedure performed inside the human body using modern medical devices such as laparoscopes and thoracoscopes. Compared to traditional surgical methods, minimally invasive surgery offers advantages such as less trauma, less pain, and faster recovery.

[0004] With technological advancements, minimally invasive surgical robotics have matured and are now widely used. A surgical robot consists of a master console and slave operating devices. The slave operating devices include multiple manipulators, including a camera arm with an imaging end-use instrument and a surgical arm with an operating end-use instrument. The master console includes a display and a handle. The surgeon manipulates the handle to control the movement of the camera arm or surgical arm, while viewing the field of view provided by the camera arm on the display.

[0005] Generally, it is recognized that the movement of the camera arm itself and the surgical arm within the camera arm's field of view are safe. However, during some operations, the surgical arm may inevitably move outside the camera arm's field of view, such as when inserting or withdrawing the surgical arm from the patient's abdominal cavity. For example, when inserting or withdrawing the surgical arm, the surgeon typically inserts or withdraws it blindly based on their experience. However, due to varying surgeons' experience and patients' physical conditions, such a blind insertion can easily lead to unexpected situations and is therefore unsafe. It is desirable to minimize or even avoid movements of the surgical arm outside the camera arm's field of view. Summary of the Invention

[0006] Based on this, it is necessary to provide a surgical robot and a method and control device for guiding the movement of the surgical arm that can reduce or even avoid the problem of the operating end instrument of the surgical arm moving outside the field of view provided by the image end instrument of the camera arm, thereby ensuring the safety of the operation.

[0007] On the one hand, the present invention provides a method for guiding the movement of a surgical arm in a surgical robot, wherein the distal end of the surgical robot has multiple operating arms, and the operating arms include a camera arm with an image end instrument and a surgical arm with an operating end instrument. The method includes the following steps: obtaining an initial position of the operating end instrument; obtaining a target position that the operating end instrument expects to reach; generating a guiding path extending from the initial position to the target position based on the field of view of the image end instrument; and adjusting the operating end instrument to move from the initial position to the target position along the guiding path.

[0008] Among them, the step of obtaining the initial position of the operating end instrument includes: obtaining the joint variables of each joint component in at least the first part of the surgical arm; and determining the initial position using forward kinematics in combination with the kinematic model of the first part and each joint variable.

[0009] Among them, the step of obtaining the target position that the operating end instrument is expected to reach includes: obtaining an input operation mode, the operation mode includes a first operation mode and a second operation mode, the first operation mode is used to guide the operating end instrument to be inserted to the target position, and the second operation mode is used to guide the operating end instrument to be withdrawn to the target position; determining the target position that the operating end instrument is expected to reach based on the obtained operation mode.

[0010] Among them, when the acquired operation mode is the first operation mode, the step of determining the target position that the operation end device is expected to reach according to the acquired operation mode includes: acquiring the target field of view of the image end device; and determining the target position that the operation end device is expected to reach according to the target field of view.

[0011] The two or more operation end instruments configured to execute the first operation mode have different target positions.

[0012] The two or more operation end instruments configured to execute the first operation mode have different target positions, and there is a safe distance between the different target positions.

[0013] In which, the surgical robot includes a puncture device, the proximal end of the puncture device is connected to the distal end of the surgical robot, and the distal end is used to be inserted into and fixed at the incision. The puncture device is used to guide the surgical arm to insert into the human body through the incision. When the acquired operation mode is the second operation mode, the step of determining the target position that the operating end instrument is expected to reach according to the acquired operation mode is: obtaining the position of the target point associated with the puncture device as the target position.

[0014] The target point associated with the trocar as the target position is located on the trocar, or is located on the extension line of the axis of the trocar and is located on the distal end side of the trocar.

[0015] There is a safe distance between the image terminal device and the target position.

[0016] Among them, the step of generating a guidance path extending from the initial position to the target position according to the field of view of the image terminal device includes: obtaining an input guidance mode, the guidance mode includes a first guidance mode and a second guidance mode, the first guidance mode refers to a mode in which the field of view of the image terminal device is automatically adjusted, and the second guidance mode refers to a mode in which the field of view of the image terminal device is manually adjusted; adjusting the field of view of the image terminal device according to the obtained guidance mode to generate a guidance path extending from the initial position to the target position.

[0017] Among them, the image terminal device is an image terminal device with stereoscopic vision. When the acquired guidance mode is the first guidance mode, the step of adjusting the field of view of the image terminal device according to the acquired guidance mode to generate a guidance path extending from the initial position to the target position includes: adjusting the field of view of the image terminal device to scan the entire environment and constructing a three-dimensional environment map using a parallax method; generating a guidance path covering the initial position and the target position according to the three-dimensional environment map.

[0018] Among them, the image terminal device is an image terminal device with stereoscopic vision. When the acquired guidance mode is the first guidance mode, the step of adjusting the field of view of the image terminal device according to the acquired guidance mode to generate a guidance path extending from the initial position to the target position includes: acquiring a reachable range of the field of view of the image terminal device; adjusting the field of view of the image terminal device to scan the environment of the reachable range and constructing a three-dimensional environment map using a parallax method; and generating a guidance path covering the initial position and the target position according to the three-dimensional environment map.

[0019] Among them, before the step of adjusting the field of view of the image terminal device to scan the environment of the reachable range and constructing a three-dimensional environment map using the parallax method, the method also includes: judging whether the initial position and the target position are both located within the reachable range; when the initial position and the target position are both located within the reachable range, entering the step of adjusting the field of view of the image terminal device to scan the environment of the reachable range and constructing a three-dimensional environment map using the parallax method.

[0020] In which, the image terminal device is an image terminal device with stereoscopic vision. When the acquired guidance mode is the first guidance mode, the step of adjusting the field of view of the image terminal device according to the acquired guidance mode to generate a guidance path extending from the initial position to the target position includes: acquiring the reachable range of the operating terminal device; adjusting the field of view of the image terminal device to scan at least the local environment containing the initial position and the target position in the reachable range and constructing a three-dimensional environment map using a parallax method; and generating a guidance path covering the initial position and the target position according to the three-dimensional environment map.

[0021] Among them, the step of adjusting the field of view of the image terminal device to scan at least the local environment containing the initial position and the target position in the reachable range and constructing a three-dimensional environment map using the parallax method is specifically: adjusting the field of view of the image terminal device to scan the environment of the reachable range and constructing a three-dimensional environment map using the parallax method.

[0022] Among them, before the step of generating a guidance path covering the initial position and the target position according to the three-dimensional environment map, the method also includes: judging whether the initial position and the target position are both located within the reachable range of the operating end device; when the initial position and the target position are both located within the reachable range, entering the step of generating a guidance path covering the initial position and the target position according to the three-dimensional environment map.

[0023] The starting point of the guiding path is the initial position, and the end point is the target position.

[0024] Wherein, the surgical robot includes a display, and the method further includes: displaying the three-dimensional environment map on the display; and displaying the position of the operating end instrument in real time in the three-dimensional environment map.

[0025] The position of the operating end device is displayed in the form of an icon in the three-dimensional environment map.

[0026] The generating of the guidance path covering the initial position and the target position according to the 3D environment map specifically includes: performing anti-collision path planning according to the 3D environment map to generate the guidance path covering the initial position and the target position.

[0027] The step of constructing a three-dimensional environment map includes: identifying obstacles within the field of view of the image terminal device; obtaining graded labels for the obstacles; and constructing the three-dimensional environment map containing the graded labels corresponding to each obstacle.

[0028] The step of obtaining the graded mark of the obstacle specifically includes: obtaining the graded mark of the obstacle from a preset relationship table according to the identified type of the obstacle.

[0029] The step of obtaining the graded mark for the obstacle specifically includes: receiving the graded mark corresponding to the obstacle that is input.

[0030] Among them, the step of generating a guidance path covering the initial position and the target position based on the three-dimensional environment map is specifically: performing anti-collision path planning based on the three-dimensional environment map and the hierarchical markings of the obstacles contained therein to generate a guidance path covering the initial position and the target position.

[0031] The method further includes: when the guide path and the obstacle do not have a contact relationship, controlling the operating end device to move at a first speed; and when the guide path and the obstacle have a contact relationship, controlling the operating end device to move at a second speed lower than the first speed.

[0032] Wherein, the method further includes: when the guide path and the obstacle are in contact, adjusting the field of view of the image terminal device so that the operation terminal device and / or the corresponding obstacle fall within the field of view of the image terminal device.

[0033] The method further includes: obtaining a touch interval from the guidance path, wherein the touch interval is an interval in which the operating terminal device and the corresponding obstacle have a touch relationship; adjusting the field of view of the image terminal device to move from the starting point to the end point of the touch interval, and always ensuring that the operating terminal device and / or the corresponding obstacle can fall within the field of view of the image terminal device.

[0034] The graded mark includes two levels: a first graded mark and a second graded mark. The obstacles corresponding to the first graded mark are allowed to be touched, while the obstacles corresponding to the second graded mark are not allowed to be touched.

[0035] The graded marks include three levels: a first graded mark, a second graded mark, and a third graded mark. The obstacles corresponding to the first graded mark are allowed to be passed through and touched, the obstacles corresponding to the second graded mark are only allowed to be touched, and the obstacles corresponding to the third graded mark are not allowed to be touched.

[0036] Each obstacle is also marked with attribute information, and the attribute information includes destructible first attribute information and indestructible second attribute information.

[0037] The method further includes: when a contact relationship exists between the guide path and the obstacle, controlling the operation end device to pass through the obstacle in a corresponding passing manner according to the classification mark and attribute information corresponding to the obstacle and the type of the operation end device.

[0038] Among them, before the step of controlling the operating end device to pass through the obstacle in a corresponding passing manner according to the grading mark, attribute information and type of the operating end device corresponding to the obstacle, the method also includes: obtaining the grading mark of the obstacle; obtaining the type of the operating end device; and determining the passing manner of the operating end device through the obstacle according to the grading mark, attribute information and type of the operating end device.

[0039] Among them, in the step of determining the passage mode of the operating end device through the obstacle based on the grading mark of the obstacle and the type of the operating end device, when the obstacle has the first grading mark and the first attribute information, and the operating end device is of a type with a destructible function, it is determined that the passage mode of passing through the obstacle is a passage mode that can destroy the obstacle to pass through the obstacle.

[0040] Among them, in the step of controlling the operating end device to pass through the obstacle in a corresponding passing manner according to the grade mark corresponding to the obstacle and the type of the operating end device, the destructible function of the operating end device is turned on when the operating end device reaches the obstacle, and the destructible function of the operating end device is turned off when the operating end device leaves the obstacle.

[0041] Among them, in the step of determining the passage mode of the operating end device through the obstacle based on the grading mark of the obstacle and the type of the operating end device, when the obstacle has the first grading mark, the second grading mark and / or has the second attribute information, and / or the operating end device is a type that does not have a destructible function, it is determined that the passage mode of passing through the obstacle is a passage mode of pushing against the obstacle to pass through the obstacle.

[0042] Among them, when the guidance mode obtained is the second guidance mode, before the step of generating a guidance path extending from the initial position to the target position according to the field of view of the image terminal device, the method also includes: judging whether the operation terminal device is located within the field of view of the image terminal device; when the operation terminal device is not located within the field of view of the image terminal device, adjusting the field of view of the image terminal device so that the operation terminal device is located within the field of view of the image terminal device; when the operation terminal device is located within the field of view of the image terminal device, entering the guidance path extending from the initial position to the target position generated according to the field of view of the image terminal device.

[0043] Among them, the step of determining whether the operation terminal device is within the field of view of the image terminal device includes: acquiring an operation image within the field of view of the image terminal device; and determining whether the operation terminal device is within the field of view of the image terminal device by image recognition of whether the operation terminal device is within the operation image.

[0044] Among them, the step of determining whether the operating end device is within the field of view of the image end device includes: obtaining the current position of the operating end device; converting the field of view of the image end device into a position range; and determining whether the operating end device is within the field of view of the image end device by determining whether the current position is within the position range.

[0045] Among them, the step of adjusting the field of view of the image terminal device so that the operation terminal device is located within the field of view of the image terminal device includes: obtaining the current position of the operation terminal device; adjusting the field of view of the image terminal device by changing the camera parameters of the image terminal device according to the current position of the operation terminal device so that the operation terminal device is located within the field of view of the image terminal device, and the camera parameters include field of view angle and / or depth of field.

[0046] Among them, the step of adjusting the field of view of the image terminal device so that the operation terminal device is located within the field of view of the image terminal device includes: obtaining the current position of the operation terminal device; adjusting the field of view of the image terminal device by changing the posture of the image terminal device according to the current position of the operation terminal device so that the operation terminal device is located within the field of view of the image terminal device, and the posture includes position and / or posture.

[0047] Among them, the step of generating a guidance path extending from the initial position to the target position according to the field of view of the image terminal device includes: acquiring the current position of the operating terminal device in real time and initializing the current position as the initial position; generating a guidance path extending from the current position to the target position according to the field of view of the image terminal device, and the guidance path is within the field of view of the image terminal device.

[0048] Among them, the step of generating a guidance path extending from the current position to the target position according to the field of view of the image terminal device is specifically: performing anti-collision path planning according to the field of view of the image terminal device to generate a guidance path extending from the current position to the target position.

[0049] The method further comprises: constraining the adjustment of the field of view of the image terminal device to always be performed under the condition that the operating terminal device is located within the field of view of the image terminal device.

[0050] The fields of view of the image terminal device at each adjacent moment are respectively a first field of view and a second field of view, and there is an overlapping area between the first field of view and the second field of view, which limits the operation terminal device to move toward the target position through the overlapping area.

[0051] The method further includes: constraining the field of view of the image terminal device to move only in the direction of the target position.

[0052] The method further includes: prohibiting the operation end device from moving when the operation end device is not within the field of view of the image end device.

[0053] The method further includes: detecting whether a start instruction is obtained; and when the start instruction is obtained, determining whether the operation terminal device is located within the field of view of the image terminal device.

[0054] The surgical robot includes a power mechanism for installing and driving the operating arm, and the start instruction is triggered when the surgical arm is installed on the power mechanism.

[0055] In which, the method also includes: when the operating end device basically moves from the initial position to the target position, judging whether the current field of view of the image end device is the initial field of view, the initial field of view refers to the field of view at the moment before the image end device is first adjusted; when the current field of view of the image end device is not the initial field of view, adjusting the current field of view of the image end device to restore it to the initial field of view.

[0056] Among them, before the step of adjusting the operating end device to move from the initial position to the target position along the guide path, the method also includes: obtaining and recording the camera parameters and posture corresponding to the initial field of view; the step of adjusting the current field of view of the image end device to restore to the initial field of view includes: directly restoring the field of view of the image end device to the initial field of view according to the recorded camera parameters and posture corresponding to the initial field of view.

[0057] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to be loaded and executed by a processor to implement the steps of the control method as described in any of the above embodiments.

[0058] On the other hand, the present invention provides a control device for a surgical robot, comprising: a memory for storing a computer program; and a processor for loading and executing the computer program; wherein the computer program is configured to be loaded and executed by the processor to implement the steps of the control method described in any of the above embodiments.

[0059] On the other hand, the present invention provides a surgical robot comprising: an operating arm, the operating arm comprising a camera arm and a surgical arm; and a controller, the controller being coupled to the operating arm and configured to execute the steps of the control method described in any one of the above embodiments.

[0060] The surgical robot and the method and control device for guiding the movement of the surgical arm thereof of the present invention have the following beneficial effects:

[0061] By utilizing the field of view provided by the image end instrument in the camera arm, a guide path extending from the initial position of the operating end instrument to the target position is generated, and then the guide path is used to guide the operating end instrument to automatically move from the initial position to the target position. Since the guide path is generated based on the field of view, the safety and reliability of the operation can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic structural diagram of an embodiment of a surgical robot according to the present invention;

[0063] Figure 2 for Figure 1 A partial schematic diagram of an embodiment of a surgical robot is shown;

[0064] Figure 3 This is a flow chart of an embodiment of a control method for a surgical robot;

[0065] Figure 4 This is a schematic diagram of the structure of the operating arm and power unit of the surgical robot;

[0066] Figures 5 to 14 They are respectively flow charts of an embodiment of a method for guiding movement of a surgical arm in a surgical robot;

[0067] Figures 15 and 16 Three-dimensional environment Figure 1 Schematic diagram of an embodiment;

[0068] Figures 17 to 21 They are respectively flow charts of an embodiment of a method for guiding movement of a surgical arm in a surgical robot;

[0069] Figures 22 to 27 They are schematic diagrams of an embodiment of a guiding state of an operating end instrument in a surgical arm;

[0070] Figures 28 and 29 They are respectively flow charts of an embodiment of a method for guiding movement of a surgical arm in a surgical robot;

[0071] Figure 30 2 is a schematic structural diagram of a control device for a surgical robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0073] It should be noted that when an element is referred to as being "disposed on" another element, it may be directly on the other element or there may also be an element centered. When an element is considered to be "connected" to another element, it may be directly connected to the other element or there may be an element centered at the same time. When an element is considered to be "coupled" to another element, it may be directly coupled to the other element or there may be an element centered at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in the present invention are for illustrative purposes only and do not represent the only implementation method. The terms "distal end" and "proximal end" used in the present invention are used as directional words, which are commonly used terms in the field of interventional medical devices, where "distal end" refers to the end away from the operator during surgery, and "proximal end" refers to the end close to the operator during surgery. The terms "first / second" and the like used in the present invention represent a component and two or more components of a type having common characteristics.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this invention pertains. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. The term "each" as used herein includes one or more than one.

[0075] like Figures 1 to 2 As shown in FIG, they are respectively a structural schematic diagram of an embodiment of a surgical robot of the present invention and a partial schematic diagram thereof.

[0076] The surgical robot includes a master console 2 and a slave operating device 3 controlled by the master console 2. The master console 2 has a motion input device 21 and a display 22. The surgeon operates the motion input device 21 to send control commands to the slave operating device 3, causing the slave operating device 3 to perform corresponding operations based on the surgeon's control commands. The surgeon also observes the surgical area through the display 22. The slave operating device 3 includes a drive arm, which includes a robotic arm 30 and one or more manipulator arms 31 detachably mounted at the distal end of the robotic arm 30. The robotic arm 30 includes a base and a connection assembly, each of which has multiple joint assemblies. The manipulator arm 31 includes a connecting rod 32, a connection assembly 33, and an end-use instrument 34, each of which has multiple joint assemblies. The end-use instrument 34 is adjusted by adjusting the joint assemblies of the manipulator arm 31. The end-use instrument 34 includes an imaging end-use instrument 34A and a manipulation end-use instrument 34B. The imaging end-use instrument 34A is used to capture images within the field of view, and the display 22 is used to display these images. The operating end instrument 34B is used to perform surgical operations such as cutting and suturing. In this paper, the operating arm with the image end instrument 34A is referred to as the camera arm 31A, and the operating arm with the operating end instrument 34B is referred to as the surgical arm 31B.

[0077] Figure 1The surgical robot on display is a single-port surgical robot, in which each operating arm 31 is inserted into the patient's body through the same puncture device 4 installed at the distal end of the robotic arm 30. In a single-port surgical robot, the doctor generally only controls the operating arm 31 to complete basic surgical operations. At this time, the operating arm 31 of the single-port surgical robot should have both positional freedom (i.e., positioning freedom) and posture freedom (i.e., orientation freedom) to achieve changes in posture within a certain range. For example, the operating arm 31 has horizontal movement freedom x, vertical movement freedom y, rotation freedom α, pitch freedom β, and yaw freedom γ. The operating arm 31 can also achieve forward and backward movement freedom z (i.e., feed freedom) under the drive of the distal joint assembly of the robotic arm 30, i.e., the power mechanism 301. In addition, in some embodiments, redundant degrees of freedom can be set for the operating arm 31 to achieve the possibility of more functions. For example, under the premise that the above six degrees of freedom can be achieved, one, two, or even more degrees of freedom can be additionally set. For example, the power mechanism 301 has a guide rail and a power unit slidingly arranged on the guide rail, and the operating arm 31 is detachably mounted on the power unit. On the one hand, the sliding of the power unit on the guide rail provides the operating arm 31 with a degree of freedom z for forward and backward movement. On the other hand, the power unit provides power to the joint assembly of the operating arm 31 to realize the remaining five degrees of freedom (i.e., [x, y, α, β, γ]).

[0078] The surgical robot also includes a controller. This controller can be integrated into the master console 2 or the slave operating device 3. Of course, the controller can also be independent of the master console 2 and the slave operating device 3. For example, the controller can be deployed locally or in the cloud. The controller can be composed of more than one processor.

[0079] The surgical robot also includes an input unit. This input unit can be integrated into the main console 2 or the slave operating device 3. Of course, the input unit can also be independent of the main console 2 and the slave operating device 3. This input unit can be, for example, a mouse, keyboard, voice input device, or touch screen. In one embodiment, a touch screen is used as the input unit, and the touch screen can be installed, for example, on the armrest of the main console 2.

[0080] The operating arm 31 also includes sensors for sensing joint variables of the joint assembly. These sensors include angle sensors for sensing rotational motion of the joint assembly and displacement sensors for sensing linear motion of the joint assembly. Specifically, appropriate sensors can be configured according to the type of joint assembly.

[0081] The controller is coupled to the sensors, and to the input and display 22 .

[0082] For example, Figure 3As shown, the abutment surface of the driving box 310 of the manipulator arm 31 abutting the power unit 302 of the power mechanism 301 is equipped with a storage unit 311. Correspondingly, the abutment surface of the power unit 302 abutting the driving box 310 is equipped with a reading unit 303 that is compatible with the storage unit 311. The reading unit 303 is coupled to the controller. When the manipulator arm 31 is installed on the power unit 302, the reading unit 303 communicates with the storage unit 311 and reads relevant information from the storage unit 311. The storage unit 311 is, for example, a memory or an electronic tag. The storage unit stores, for example, the type of manipulator arm, the target location of the manipulator arm that can be configured, the kinematic model of the manipulator arm, etc. For example, the storage unit 311 of the camera arm 31A also stores camera parameters.

[0083] like Figure 4 As shown, it is a schematic structural diagram of an embodiment of the surgical robot of the present invention. More specifically, Figure 4 Shown is a schematic structural diagram of an embodiment of a multi-hole surgical robot. Figure 4 The multi-port surgical robot shown is Figure 1 The difference between the single-port surgical robots shown mainly lies in the difference between their slave operating devices. Figure 4 The driving arm of the slave operating device in the multi-hole surgical robot shown has a robotic arm 110, an adjustment arm 120, a manipulator 130, and an operating arm 150 connected in sequence. The number of adjustment arms 120, manipulators 130, and operating arms 150 is the same and there are two or more of them, for example, four. The distal end of the robotic arm 110 has an orientation platform, the proximal end of the adjustment arm 120 is connected to the orientation platform, and the proximal end of the manipulator 130 is connected to the distal end of the adjustment arm 120. The manipulator 130 is used to detachably connect to the operating arm 150, and the manipulator 130 has multiple joint assemblies. Each manipulator 130 has a power mechanism, and the operating arm 150 is mounted on the power mechanism and further driven by the power mechanism. In a multi-port surgical robot, different operating arms 150 are inserted into the patient's body through different puncture devices. The operating arms 150 of the multi-port surgical robot generally have fewer degrees of freedom compared to the operating arms 31 of the single-port surgical robot. Usually, the operating arm 150 only has posture freedom (i.e., orientation freedom). Of course, changes in its posture generally also affect the position, but because the impact is small, it can be ignored in certain scenarios. The change in the position of the operating arm 150 can usually be achieved with the assistance of the manipulator 130. Since the manipulator 130 and the operating arm 150 are linked to achieve posture changes, the two can be considered as manipulator components, which are equivalent to the operating arm 31 in the single-port surgical robot.

[0084] Depending on the configuration, the motion input device 21 can input posture commands, including position commands and posture commands, to control the posture changes of the distal end of the first portion of the actuator arm. The distal end of the first portion typically refers to the end instrument. Alternatively, the distal end of the first portion may refer to a joint assembly connected to the end instrument. Changes in the posture of the end instrument typically coincide with changes in the posture of the joint assembly.

[0085] exist Figure 1 In the surgical robot shown, the driving arm includes a robotic arm and a manipulator arm. The proximal end of the manipulator arm is mounted on the distal end of the robotic arm, and the end instrument is mounted on the distal end of the manipulator arm. Depending on the configuration, the first part can be configured as the manipulator arm; alternatively, the first part can be configured as a combination of the robotic arm and the manipulator arm.

[0086] And correspondingly Figure 4 In the surgical robot shown, the driving arm includes a robotic arm, an adjustment arm, a manipulator, and an operating arm. The proximal end of the adjustment arm is mounted to the distal end of the robotic arm, the proximal end of the manipulator is mounted to the distal end of the adjustment arm, the proximal end of the operating arm is mounted to the distal end of the manipulator, and the end instrument is mounted to the distal end of the operating arm. Depending on the configuration, the first part can be configured as the operating arm; or the first part can be configured as a combination of the manipulator and the operating arm; or the first part can be configured as a combination of the robotic arm, adjustment arm, manipulator, and operating arm.

[0087] Understandable, whether Figure 1 The single-port surgical robot shown is still Figure 4 In the multi-aperture surgical robot shown, the robotic arm is typically used to adjust the position of the end instrument over a wide range, while the manipulator arm is used to fine-tune the position of the end instrument. For example, the robotic arm is used to position the end instrument before surgery, and the manipulator arm is primarily used to control the surgery during surgery. Of course, in some embodiments, the robotic arm and the manipulator arm, as well as other corresponding arm structures, can also be combined to coordinate motion to achieve specific functions. Depending on the configuration, one or more of the end instruments can be configured as a controlled end instrument to receive control from a motion input device.

[0088] In one embodiment, the present invention provides a method for guiding the movement of a surgical arm in a surgical robot. The method can be executed by a controller and is applicable to various types of surgical robots. Figure 5 As shown, the method includes the following steps:

[0089] Step S11: Acquire the initial position of the operating end instrument.

[0090] For example, the joint variables of each joint assembly in at least the first portion of the surgical arm can be obtained first, and then the initial position can be determined using forward kinematics in combination with the kinematic model of at least the first portion of the surgical arm and each joint variable. For example, in the conventional control mode of a single-port surgical robot, the initial position of the operating end instrument can be determined only by the joint variables of each joint assembly in the surgical arm and its kinematic model. For example, in the conventional control mode of a single-port surgical robot, the initial position of the operating end instrument can be determined by the joint variables of each joint assembly in the surgical arm and the manipulator and its kinematic model.

[0091] Step S12: obtaining the target position that the end-of-operation instrument is expected to reach.

[0092] The positions and postures of various objects described in the present invention, including but not limited to the imaging end-use instrument of the camera arm and the manipulation end-use instrument of the surgical arm, are all described based on the same reference coordinate system. This reference coordinate system includes but is not limited to the surgical robot's base coordinate system. For example, it may also be another coordinate system converted from this base coordinate system that can be used as a reference, such as the coordinate system of the main operating console.

[0093] The target position may be a position currently located within the field of view of the image end instrument, or a position currently not located within the field of view of the image end instrument, and may be specifically determined according to the needs during the operation.

[0094] Step S13: generating a guidance path extending from the initial position to the target position according to the field of view of the end device of the image.

[0095] According to the coverage of the visual field of the imaging end device, the guidance path can be all or part of a complete path from the initial position to the target position, wherein the initial position and the target position are two spaced points on the guidance path.

[0096] Step S14: adjusting the operating end instrument to move from the initial position to the target position along the guide path.

[0097] In the above steps S11 to S14, the relevant position of the operating end instrument of interest may be, for example, the relevant position of a part of the operating end instrument, such as one or more points on its tip, middle or tail end, or the relevant position of the entire operating end instrument.

[0098] When the guidance path is the entirety of a complete path from the initial position to the target position, the end-user device can be automatically guided directly from the initial position to the target position. When the guidance path is a portion of a complete path from the initial position to the target position, the end-user device can be automatically guided from the initial position toward the target position to approach the target position. The principle is roughly as follows:

[0099] The guidance path is discretized into multiple sub-target positions, and then the joint variables of each joint component in the first part of the surgical arm to achieve the corresponding sub-target position are solved through inverse kinematics. Then, the joint components in the first part of the surgical arm are controlled to move to the corresponding joint variables to guide the operating end instrument to reach the corresponding sub-target position.

[0100] According to the above steps S11 to S14, a guide path extending from the initial position of the operating end instrument to the target position is generated by utilizing the field of view provided by the image end instrument in the camera arm, and then the guide path is used to guide the operating end instrument to automatically move from the initial position to the target position. Since the guide path is generated based on the field of view, the safety and reliability of the operation can be guaranteed.

[0101] Generally, different operation requirements correspond to different target positions of the operation end instrument. Figure 6 As shown, the above step S12, i.e., the step of obtaining the target position that the end-of-operation instrument is expected to reach, includes:

[0102] Step S121: Acquire the input operation mode.

[0103] Among them, the operation mode includes but is not limited to a first operation mode and a second operation mode. For example, the first operation mode is used to guide the operation end instrument to be inserted into the target position, and its applicable scenarios are not limited to the scenario of inserting the surgical arm from outside the patient's body into the body before surgery; the second operation mode is used to guide the operation end instrument to be withdrawn to the target position, and its applicable scenarios are not limited to the scenario of replacing the surgical arm during surgery and retracting the surgical arm at the end of surgery.

[0104] Step S122: determining a target position to be reached by the desired operating end device according to the acquired operating mode.

[0105] In one embodiment, when the acquired operation mode is the first operation mode, Figure 7 As shown, the above step S122, i.e., the step of determining the target position to which the end device is expected to be reached according to the acquired operation mode, includes:

[0106] Step S1221: Acquire the target field of view of the end device of the image.

[0107] The target field of view can be, for example, based on a doctor's confirmation of the field of view corresponding to a specific moment in time, and the surgery is typically performed within this target field of view. For example, before surgery, the doctor typically inserts a camera arm into the patient's body and uses the camera arm's image end device to observe and determine the field of view suitable for the surgery. Upon receiving a confirmation command triggered by the doctor, the field of view corresponding to the moment the confirmation command was generated is used as the target field of view.

[0108] Step S1222: Determine the target position to which the end instrument is expected to be reached based on the target field of view.

[0109] The target position is a point having a specific positional relationship with the target visual field, and can be, for example, the center of the target visual field, or a point deviated from the center of the target visual field and intersecting with the extension direction of the connecting rod of the surgical arm.

[0110] In one embodiment, two or more end-use devices configured to perform the first operating mode have different target locations. Furthermore, the different target locations are typically spaced a safe distance apart to prevent collision between the end-use devices. For example, one of the target locations may be the center of the target field of view, while the remaining target locations may be specific points outside the center of the target field of view.

[0111] In one embodiment, when there are more than two operating end devices that need to be guided to the target position, they can usually be guided one by one, that is, after guiding one operating end device to the target position, another operating end device is guided to the target position until all the operating end devices are guided to the target position.

[0112] The surgical robot includes a trocar, the proximal end of which is detachably connected to the distal end of the surgical robot, and the distal end is inserted into and fixed at the incision. The trocar is used to guide the surgical arm through the incision and into the human body. In one embodiment, when the acquired operating mode is the second operating mode, step S1222, i.e., the step of determining the target position to be reached by the desired operating end instrument based on the acquired operating mode, may include: acquiring the position of a certain point associated with the trocar as the target position. The point associated with the trocar serving as the target position may be located on the trocar, or may be located on an extension of the trocar's axis and on the distal side of the trocar. The trocar typically has a cylindrical insertion portion, and the axis here typically refers to the central axis of the insertion portion.

[0113] In the above embodiment, a safe distance is maintained between the imaging end device and the target position to prevent collision between the imaging end device and the operating end device.

[0114] Typically, the guide path is generated differently depending on how the camera arm is adjusted. Figure 8 As shown, the above step S13, i.e., the step of generating a guidance path extending from the initial position to the target position according to the field of view of the image terminal device, includes:

[0115] Step S131: Acquire the input guidance mode.

[0116] The guidance mode includes a first guidance mode and a second guidance mode. The first guidance mode refers to a mode in which the field of view of the image terminal device is automatically adjusted, and the second guidance mode refers to a mode in which the field of view of the image terminal device is manually adjusted.

[0117] Step S132: adjusting the field of view of the image terminal device according to the acquired guidance mode to generate a guidance path extending from the initial position to the target position.

[0118] When the guidance mode obtained is the first guidance mode, the above-mentioned step S132 can also be implemented by a variety of implementation methods. Among them, the guidance path can be generated with the help of the field of view obtained by the image terminal device of stereoscopic vision, wherein the change of the field of view of the image terminal device can be automatic or manual. In addition, the guidance path can also be generated with the help of information sensed by ultrasonic sensors, light sensors, etc. Of course, they can also be used in combination. The generation of the guidance path by the field of view obtained by the image terminal device of stereoscopic vision is used as an example for illustrative explanation. When the guidance mode obtained is the first guidance mode, the above-mentioned step S132 can be implemented, for example, in the following three ways.

[0119] <Method 1>

[0120] like Figure 9 As shown, the above step S132 may include:

[0121] Step S13211: Adjust the field of view of the image terminal device to scan the entire environment and construct a three-dimensional environment map using the parallax method.

[0122] That is, a global scan of the patient's internal environment is performed, wherein the camera arm may not be restricted from moving around the telecentric fixed point to achieve the global scan.

[0123] Step S13212: Generate a guidance path covering the initial position and the target position based on the three-dimensional environment map.

[0124] Here, the guidance route covering the initial position and the target position means that the points constituting the guidance route include the initial position and the target position.

[0125] <Method 2>

[0126] like Figure 10 As shown, the above step S132 may include:

[0127] Step S13221: Obtain the reachable range of the field of view of the image terminal device.

[0128] The reachable range of the field of view of the image terminal device refers to the spatial set of its entire field of view.

[0129] Step S13222: Adjust the field of view of the image terminal device to scan the environment within the reachable area and construct a three-dimensional environment map using the parallax method.

[0130] The reachable range of the field of view refers to, for example, the reachable range of the field of view when the camera arm moves around the telecentric fixed point.

[0131] Step S13223: Generate a guidance path covering the initial position and the target position based on the three-dimensional environment map.

[0132] Preferably, before step S13222, i.e., before adjusting the field of view of the imaging terminal device to scan the environment within the reachable range and constructing a 3D environmental map using the parallax method, the following step is performed: determining whether both the initial position and the target position are within the reachable range. If both the initial position and the target position are within the reachable range, the process proceeds to step S13222; otherwise, the physician may be prompted to adjust the imaging terminal device and / or the operating terminal device to meet the activation condition.

[0133] <Method 3>

[0134] like Figure 11 As shown, the above step S132 may include:

[0135] Step S13231: Obtain the reachable range of the operating end device.

[0136] The reachable range of the end-operation device refers to the reachable range of the end-operation device, for example, the reachable range when the manipulator moves around a telecentric fixed point.

[0137] Step S13232: Adjust the field of view of the image terminal device to scan at least the local environment including the initial position and the target position in the reachable range and construct a three-dimensional environment map using the parallax method.

[0138] For example, the field of view of the imaging end device can be adjusted to scan only a portion of the reachable range of the operating end device, including the initial and target positions, and a 3D environment map of this portion can be constructed using the parallax method. Another example is that the field of view of the imaging end device can be adjusted to scan the entire reachable range of the operating end device, and a 3D environment map of the entire reachable range can be constructed using the parallax method.

[0139] Step S13233: Generate a guidance path covering the initial position and the target position based on the three-dimensional environment map.

[0140] For example, in the example of adjusting the field of view of the image terminal instrument to scan the entire reachable range of the operation terminal instrument and constructing a 3D environment map of the entire reachable range using the parallax method, the method of the present invention may preferably further include: determining whether the reachable range of the operation terminal instrument is within the reachable range of the field of view of the image terminal instrument. If the reachable range of the operation terminal instrument is within the reachable range of the field of view of the image terminal instrument, the step of adjusting the field of view of the image terminal instrument to scan the environment of the entire reachable range and constructing a 3D environment map using the parallax method is initiated; otherwise, the doctor may be prompted to adjust the image terminal instrument and / or the operation terminal instrument to meet the activation condition.

[0141] In the three embodiments described above, preferably, before the step of generating a guidance path covering the initial position and the target position based on the 3D environment map, the following step may be further included: determining whether the initial position and the target position are both within the reachable range of the operation terminal instrument. If both the initial position and the target position are within the reachable range of the operation terminal instrument, the step of constructing the 3D environment map or the step of generating a guidance path covering the initial position and the target position based on the 3D environment map is entered. In other words, this determination step may be performed before the step of constructing the 3D environment or before the step of generating a guidance path covering the initial position and the target position based on the 3D environment map. Otherwise, the doctor may be prompted to adjust the image terminal instrument and / or the operation terminal instrument to meet the start condition. The purpose of performing this determination is to determine whether the operation terminal instrument can reach the target position from the initial position. If it cannot reach the target position, there is no need to continue the execution if it is necessary to achieve the target position. However, in fact, even if the operation terminal instrument cannot reach the target position, a guidance path extending from the initial position to the target position can be generated based on the constructed 3D environment map to adjust the operation terminal instrument to move closer to the target position as much as possible.

[0142] For example, in the above steps S13212, S13223 and S13233, i.e., the steps of generating a guidance path covering the initial position and the target position based on the three-dimensional environment map, the starting point of the planned guidance path can be the initial position and the end point can be the target position.

[0143] In one embodiment, if Figure 12 As shown, the method of the present invention also includes:

[0144] Step S141: displaying a three-dimensional environment map on a display.

[0145] Step S142: Display the position of the operating terminal device in real time in the three-dimensional environment map.

[0146] Exemplarily, the position of the operation end device is displayed in the 3D environment map in the form of an icon, for example, the icon is an icon of the operation end device, or for another example, the icon is a light spot.

[0147] Through steps S141 to S142 , the doctor can be assisted in learning the positional relationship between the operating end instrument and the environment, for example, the doctor can learn the positional relationship between the operating end instrument and an obstacle.

[0148] In the above-mentioned steps S13212, S13223 and S13233, there can be multiple planning methods for generating a guidance path covering the initial position and the target position based on the three-dimensional environment map. For example, the planned guidance path is the shortest path. For another example, the planned guidance path is the smoothest path. In one embodiment, this step can specifically generate a guidance path covering the initial position and the target position by performing anti-collision path planning based on the three-dimensional environment map. The methods that can be used for anti-collision path planning include but are not limited to graph search method, RRT algorithm, artificial potential field method, etc. Preferably, when there are multiple guidance paths that meet the non-collision requirement, the optimization selection can be performed in combination with the constraint conditions, for example, the best guidance path can be selected by combining the shortest path and / or the smoothest path.

[0149] Due to the complexity of the human body's internal environment, if obstacles within the environment are completely untouchable and need to be avoided, this is often not feasible even for manual surgery by doctors. Therefore, it is possible to consider grading obstacles based on actual conditions or experience to clearly identify which obstacles are actually touchable. This will also help to plan the guidance path more flexibly.

[0150] In one embodiment, if Figure 13 As shown, the steps of constructing a 3D environment map in the above steps S13212, S13223 and S13233 may further include:

[0151] Step S151: Identify obstacles within the field of view of the image terminal device.

[0152] Obstacles within the field of view can be identified using a neural network, such as a convolutional neural network. In the present invention, these obstacles may be tissues and organs of the human body, or objects such as a surgical arm inserted into an animal's body.

[0153] Step S152: Obtain a graded label for the obstacle.

[0154] For example, the obstacle classification label can be automatically obtained from a preset relationship table based on the identified obstacle type. This is an automated process. Another example is receiving a classification label corresponding to an obstacle input by a doctor, which is a manual process. Of course, in many scenarios, both methods can be combined to obtain the classification label, facilitating rapid classification and mutual verification.

[0155] Exemplarily, these graded marks can be divided into two levels. For example, the graded marks include a first graded mark and a second graded mark. Among them, the obstacles corresponding to the first graded mark can be touched by the operating end instrument, for example, they can change their position when touched by the operating end instrument; the obstacles corresponding to the second graded mark cannot be touched by the operating end instrument and must be strictly avoided. For example, the human body's fat tissue can be set as the first graded mark; the human body's important organs including but not limited to the liver, kidneys, spleen, stomach, blood vessels, heart, intestines, gallbladder and other surgical arms can be set as the second graded mark.

[0156] Exemplarily, these grading marks can also be divided into three levels. For example, the grading marks include a first grading mark, a second grading mark, and a third grading mark. Among them, the obstacle corresponding to the first grading mark can be allowed to be directly passed through by the operating end instrument, the obstacle corresponding to the second grading mark can be allowed to be touched by the operating end instrument, and the obstacle corresponding to the third grading mark cannot be touched by the operating end instrument. Among them, "directly pass through" includes the meaning of "touch", that is, it can be directly passed through and can also be touched; "touch" does not include other meanings. That is, the first grading mark to the third grading mark are divided according to the level of influence of the obstacle on the path from light to heavy. For example, the fat tissue of the human body can be set as the first grading mark; the important organs of the human body including but not limited to the liver, kidney, spleen, stomach, and intestines can be set as the second grading mark; the important organs of the human body including but not limited to blood vessels, heart, gallbladder and other surgical arms can be set as the third grading mark. The above-mentioned grading marks are only exemplary and can actually be defined as needed.

[0157] Of course, these hierarchical markings can also be set to more levels of division to facilitate flexible planning of anti-collision guidance paths.

[0158] Step S153: construct a 3D environment map containing hierarchical labels corresponding to each obstacle.

[0159] For example, the respective hierarchical labels of the corresponding obstacles are marked in the image attributes of the three-dimensional environment map.

[0160] Therefore, the aforementioned 3D environment map containing the hierarchical markings corresponding to each obstacle can be used to plan a more reasonable guidance path. In one embodiment, specifically in the aforementioned steps S13212, S13223, and S13233, i.e., the steps of generating a guidance path covering the initial position and the target position based on the 3D environment map: collision avoidance path planning can be performed based on the 3D environment map and the hierarchical markings of obstacles contained therein to generate a guidance path covering the initial position and the target position.

[0161] Compared to the situation where obstacles are not divided into graded markings, which results in the inability to avoid obstacles and plan an effective guidance path when planning a collision avoidance guidance path, under the conditions of the same 3D environment map, obstacles are divided into two levels with first and second grade markings. Since obstacles corresponding to the first grade marking can be touched, an effective guidance path may exist. Furthermore, obstacles are divided into three levels with first, second, and third grade markings. Since obstacles corresponding to the first grade marking can be directly passed through and obstacles corresponding to the second grade marking can be touched, the probability of an effective guidance path existing will be higher. Therefore, the higher the level of reasonable grade markings, the higher the probability of being able to plan an effective collision avoidance guidance path in most cases.

[0162] In some embodiments, the same obstacle may be divided into only one level. For example, it is recommended to plan the anti-collision guidance path without distinguishing the graded markings of the obstacles. If such a guidance path exists, it will be the most reliable and safe for the guidance of the terminal device. If the anti-collision guidance path is planned without distinguishing the graded markings of the obstacles, and such a guidance path does not exist, then consider distinguishing the graded markings of the obstacles to plan the anti-collision guidance path. For example, when there are only two levels of division as mentioned above, the obstacles can be divided into two levels of the first graded marking (touchable) and the second graded marking (untouchable) to plan the anti-collision guidance path. On the one hand, it increases the probability of planning a reasonable guidance path, and on the other hand, it can meet the requirements of reliability and safety for the guidance of the terminal device. For another example, when there are only three levels of division as mentioned above, obstacles can be divided into three levels: first-level markings (passable), second-level markings (touchable) and third-level markings (untouchable), and then anti-collision guidance paths can be planned. This can further improve the probability of planning a reasonable guidance path, and can also meet the requirements for reliable and safe guidance of operating end devices.

[0163] In some embodiments, the same obstacle can be divided into multiple levels. For example, it is still recommended to first plan the anti-collision guidance path without distinguishing the hierarchical markings of obstacles. If such a guidance path exists, it will be the most reliable and safe for guiding the end-user instrument. In this case, when the guidance path exists, the end-user instrument can be controlled to move at the first speed. If the anti-collision guidance path is planned without distinguishing the hierarchical markings of obstacles, and such a guidance path does not exist, consider planning the anti-collision guidance path after considering distinguishing the hierarchical markings of obstacles. For example, when both the above-mentioned two-level division and the above-mentioned three-level division are present, obstacles may be first divided into two levels, namely, a first grade mark (touchable) and a second grade mark (untouchable), and then a collision-avoidance guidance path may be planned. If a reasonable guidance path can be planned at such a level, the guidance path is used to guide the movement of the end instrument. In this case, when the guidance path exists, the end instrument may be controlled to move at a second speed lower than the first speed. If a reasonable guidance path cannot be planned at such a level, obstacles may be further divided into three levels, namely, a first grade mark (traversable), a second grade mark (touchable), and a third grade mark (untouchable), and then a collision-avoidance guidance path may be planned. If a reasonable guidance path can be planned at such a level, the guidance path is used to guide the movement of the end instrument. In this case, when the guidance path exists, the end instrument may be controlled to move at a third speed lower than the first or second speed. If a reasonable guidance path cannot be planned at such a level, the doctor may be advised or prompted to perform manual operation to guide the movement of the end instrument.

[0164] In other words, under the condition of multi-layer hierarchical marking, priority can be given to path planning in which no collision occurs with obstacles, then to path planning that is collidable but not passable, and finally to path planning that is collidable and passable.

[0165] In some embodiments, if the guidance path planned for collision avoidance inevitably involves collision (including crossing), the field of view of the image terminal instrument can be automatically adjusted so that the operating terminal instrument and / or the corresponding obstacle can fall within the field of view, so that the doctor can observe the collision between the operating terminal instrument and the surrounding environment. For example, the field of view can be adjusted by adjusting the camera parameters of the image terminal instrument, or by adjusting the position and / or posture of the image terminal instrument. Of course, the field of view can also be adjusted in combination with the camera parameters, position and / or posture of the image terminal instrument. Furthermore, a touch interval can be obtained from the guidance path, and the touch interval is a section in which the operating terminal instrument and the corresponding obstacle have a touch relationship. Then, within the touch interval, the field of view of the image terminal instrument can be adjusted to move from the starting point to the end point of the touch interval, and it is always ensured that the operating terminal instrument and / or the corresponding obstacle can fall within the field of view.

[0166] In some embodiments, when there is contact (including passage) between the planned guidance path and an obstacle, the operating end device can be controlled to pass through the obstacle in a corresponding manner based on the grade mark, attribute information and type of the operating end device corresponding to the obstacle.

[0167] In one embodiment, if Figure 14 As shown, before the step of controlling the operating end device to pass through the obstacle in a corresponding passing manner according to the grade mark, attribute information and type of the operating end device corresponding to the obstacle, the method includes:

[0168] Step S161: Obtain the classification mark and attribute information of the obstacle.

[0169] Obstacle attribute information includes destructible first attribute information and indestructible second attribute information, both of which can be stored in the 3D environment map associated with the obstacle. Obstacle attribute information can also be automatically set or marked using image recognition combined with a correspondence table. Alternatively, obstacle attribute information can be manually marked by a doctor, for example, by obtaining the attribute information marked by the doctor and automatically adding it to the corresponding obstacle. This attribute information can also be stored in the constructed 3D environment map associated with the corresponding obstacle.

[0170] Step S162: Obtain the type of the operating end instrument.

[0171] The type of the operation end instrument can be obtained through image recognition or read from a storage unit of the operation arm. For example, the type of the operation end instrument includes a destructible type and a non-destructible type.

[0172] Step S163 : determining a passing mode of the operation end device through the obstacle according to the grade mark, attribute information and type of the operation end device of the obstacle.

[0173] Exemplarily, if the obstacle has the first classification mark and the first attribute information (i.e., penetrable and destructible), and the end-user instrument is a type of end-user instrument with a destructible function, such as a water knife, ion knife, ultrasonic knife, condensation knife, electric knife, etc., the method of passing through the obstacle can be a method of passing through the obstacle by destroying the obstacle. This method of passing mainly relies on burning through the obstacle and then passing through the burned-through channel to overcome the obstruction of the obstacle. When controlling the end-user instrument to pass through the obstacle in this way, the destructible function can be turned on when the end-user instrument reaches the obstacle, and the destructible function can be turned off when the end-user instrument leaves the obstacle.

[0174] Exemplarily, when the obstacle has the first grading mark, the second grading mark and / or has the second attribute information (i.e., penetrable, touchable and / or indestructible), and / or the operating end device is of a type that does not have a destructible function, the method of passing through the obstacle can be a method of pushing against the obstacle to pass through the obstacle. This method of passing mainly relies on deforming or shifting the obstacle to overcome the obstruction of the obstacle.

[0175] For example, Figure 15 As shown, it is assumed that there are obstacles 1 to 4 in the three-dimensional environment map, and it is assumed that obstacles 1 to 4 are fat tissue, liver, kidney, and stomach respectively. If obstacles 1 to 4 are all untouchable, it is impossible to plan a collision-free guidance path. Therefore, it is possible to consider marking obstacles 1 to 4 in a graded manner. Assuming that it is divided into two levels, for example, obstacle 1 is divided into the first grade mark (touchable), and obstacles 2 to 4 are all divided into the second grade mark (untouchable), it may be found that even if obstacle 1 is touched, an effective guidance path cannot be planned. In Figure 14 In the embodiment shown, considering that the end-user device is a destructible end-user device, it is possible to reclassify these obstacles 1 to 4, for example, classifying obstacle 1 as the first classification mark (passable) and classifying obstacles 2 to 4 as the second classification mark (untouchable). In this way, a reasonable guidance path from the initial position to the target position can be planned, which passes through obstacle 1 and touches obstacle 2 along the way, while avoiding obstacles 3 and 4. Figure 16 As shown. Then, the electric hook type operating end instrument is along Figure 16 The guidance path shown turns on the destructible function when reaching obstacle 1, turns off the destructible function after passing through obstacle 1, then hits obstacle 2 and passes by the side of obstacle 2, and finally reaches the target position.

[0176] In the first guidance mode, the guidance paths for multiple end-devices are typically different. Depending on the configuration, the end-devices can be guided sequentially to their respective target locations along their corresponding guidance paths, or multiple end-devices can be guided simultaneously along their corresponding guidance paths to their respective target locations. Of course, when planning the guidance paths for each end-device, these end-devices can be considered as obstacles in the 3D environment map, thereby rationally planning the guidance paths.

[0177] In one embodiment, when the acquired guidance mode is the second guidance mode, Figure 17 As shown, before the above step S13, that is, before the step of generating a guidance path extending from the initial position to the target position according to the field of view of the end device of the image, the method further includes:

[0178] Step S171 , determining whether the operating end device is within the field of view of the image end device.

[0179] When the operating end device is not within the field of view of the imaging end device, the process proceeds to step S172 ; and when the operating end device is within the field of view of the imaging end device, the process proceeds to step S13 .

[0180] Step S172: adjusting the field of view of the imaging terminal device so that the operating terminal device is located within the field of view of the imaging terminal device.

[0181] There are multiple methods for the above-mentioned step S171, ie, determining whether the operating end device is located within the field of view of the image end device. The present invention exemplifies two methods to implement the step S171.

[0182] In one embodiment, if Figure 18 As shown, step S171 may include:

[0183] Step S1711: Acquire an operation image within the field of view of the imaging terminal device.

[0184] Step S1712 : determining whether the operation terminal device is located within the operation image by image recognition to determine whether the operation terminal device is located within the field of view of the image terminal device.

[0185] In step S1712, if the operation end device is recognized to exist in the operation image, it is determined that the operation end device is located in the field of view of the image end device; and if the operation end device is recognized not to exist in the operation image, it is determined that the operation end device is not located in the field of view of the image end device.

[0186] In order to better perform image recognition, a neural network can be trained to perform image recognition. For example, the trained neural network can be a convolutional neural network.

[0187] In another embodiment, if Figure 19 As shown, step S171 may further include:

[0188] Step S1711 ′: obtaining the current position of the operating end instrument.

[0189] The current position of the operating end instrument can be obtained, for example, by combining the kinematic model of the surgical arm and the joint variables of each joint assembly in the surgical arm using forward kinematics calculations. These joint variables can be detected by sensors at the corresponding joint assemblies. In other embodiments, the current position of the operating end instrument can also be scanned and identified by adjusting the stereoscopic image end instrument, for example, by identifying the position of the operating end instrument relative to the image end instrument, and then the current position of the operating end instrument in the reference coordinate system can be determined by coordinate system transformation.

[0190] Step S1712': convert the field of view of the end device of the image into a position range.

[0191] The field of view is a region that actually has boundaries and can therefore be converted into, for example, a positional range of a reference coordinate system.

[0192] Step S1713 ′: determining whether the current position is within the position range to determine whether the operation end device is within the field of view of the image end device.

[0193] In step S1713', if the current position of the operating end device is within the position range corresponding to the field of view of the image end device, it is determined that the operating end device is within the field of view of the image end device; and if the current position of the operating end device is not within the position range corresponding to the field of view of the image end device, it is determined that the operating end device is not within the field of view of the image end device.

[0194] In some embodiments, these two methods can be combined to mutually verify whether the operating end device is within the field of view of the imaging end device. For example, if the results determined by image recognition and position detection are inconsistent, for safety reasons, adjustment of the imaging end device's field of view can be stopped first and then resumed after receiving the doctor's confirmation. This process can also be used to calibrate the image recognition neural network to improve its judgment accuracy.

[0195] The above-mentioned step S172, ie, the step of adjusting the field of view of the imaging terminal device so that the operating terminal device is located within the field of view of the imaging terminal device, can also be implemented in various ways.

[0196] In one embodiment, Figure 20 As shown, step S172 may include:

[0197] Step S1721: Acquire the current position of the operating end instrument.

[0198] Step S1722 : According to the current position of the operating end device, the field of view of the imaging end device is adjusted by changing the camera parameters of the imaging end device so that the operating end device is located within the field of view of the imaging end device.

[0199] The camera parameters include the field of view angle and / or depth of field. If it is pre-calculated that only adjusting the camera parameters can cover both the current position and the target position of the end device, this method can be used to maintain the posture of the end device in the image.

[0200] In another embodiment, if Figure 21 As shown, step S172 may further include:

[0201] Step S1721', obtaining the current position of the operating end instrument.

[0202] Step S1722 ′: adjusting the field of view of the image terminal instrument by changing the posture of the image terminal instrument according to the current position of the operation terminal instrument so that the operation terminal instrument is located within the field of view of the image terminal instrument.

[0203] The pose includes position and / or attitude. If it is pre-calculated that only adjusting the pose of the imaging end device can cover the area between the current position of the operating end device and the target position, this method can be used to maintain the camera parameters of the imaging end device.

[0204] For example, see Multiport Surgical Robot. Figures 22 to 24 Assume that Figure 22 As shown, the current position of the imaging end instrument of the camera arm is B0, the current position of the operating end instrument of the surgical arm is A0, the field of view of B0 is the target field of view, and A0 is outside the target field of view. In one embodiment, it can be as follows Figure 23 As shown, the current position B0 of the image terminal device is kept unchanged, and the field of view is adjusted by adjusting the camera parameters of the image terminal device, such as the field of view angle, so that A0 falls within the adjusted field of view. In one embodiment, it can be as follows Figure 24 As shown, the camera parameters of the image end device are kept unchanged, and the position of the image end device is adjusted to position B1 to adjust the field of view so that A0 falls within the adjusted field of view.

[0205] Of course, this is also applicable to single-port surgical robots, see Figures 25 to 27 Assume that Figure 25As shown, the current position of the imaging end instrument of the camera arm is B0, the current position of the operating end instrument of the surgical arm is A0, the field of view of B0 is the target field of view, and A0 is outside the target field of view. In one embodiment, as Figure 26 As shown, the camera parameters of the image end device are kept unchanged, and the position of the image end device is adjusted to position B1 to adjust the field of view so that A0 falls within the adjusted field of view; Figure 27 As shown, the camera parameters of the end-image device remain unchanged, and the position of the end-image device is adjusted to B1 to adjust the field of view so that A0 falls within the adjusted field of view. Of course, in some cases, the current position B0 of the end-image device can also be kept unchanged, and the camera parameters of the end-image device, such as the field of view angle, can be adjusted to adjust the field of view so that A0 falls within the adjusted field of view, which is not shown in the figure.

[0206] In some embodiments, the two methods can also be combined with each other to jointly adjust the field of view of the image terminal instrument to better move toward the current position and / or target position of the operating terminal instrument so that the current position and / or target position can fall within the field of view of the image terminal instrument. For example, the posture of the image terminal instrument can be adjusted first; for another example, the camera parameters of the image terminal instrument can be adjusted first. The object of priority adjustment (i.e., the posture and camera parameters of the image terminal instrument) can be configured according to the instructions input by the doctor. For example, when the priority adjustment object is the posture of the image terminal instrument, the posture of the image terminal instrument will be adjusted as much as possible to move its field of view toward the current position and / or target position of the operating terminal instrument. If the field of view of the image terminal instrument does not cover the current position and / or target position of the operating terminal instrument when the movement reaches the limit, the camera parameters of the image terminal instrument will be adjusted to make the field of view of the image terminal instrument cover the current position and / or target position of the operating terminal instrument. When the priority adjustment object is the camera parameters of the image terminal device, the camera parameters of the image terminal device will be adjusted as much as possible to move its field of view toward the current position and / or target position of the operating terminal device. If the field of view of the image terminal device does not cover the current position and / or target position of the operating terminal device when the movement reaches the limit, the posture of the image terminal device will be adjusted to make the field of view of the image terminal device cover the current position and / or target position of the operating terminal device.

[0207] In some embodiments, even when adjusting only the posture of the imaging terminal device, a priority can be set. For example, the posture can be prioritized, or the position can be prioritized. Similarly, even when adjusting only the camera parameters of the imaging terminal device, a priority can be set. For example, the field of view angle can be prioritized, or the depth of field can be prioritized. The priority adjustment objects (i.e., the posture and position in the posture, and / or the field of view angle and depth of field in the camera parameters) can also be configured based on the instructions input by the doctor.

[0208] In some embodiments, multiple levels of priority can be configured to adjust the field of view of the image terminal device, and the corresponding parameters of the image terminal device can be adjusted step by step according to the configured priorities to achieve the field of view adjustment until the field of view can cover the current position and / or target position of the operating terminal device. For example, the first level of priority is to adjust the posture of the image terminal device, the second level of priority is to adjust the posture of the posture of the image terminal device, and the third level of priority is to adjust the field of view angle of the camera parameters of the image terminal device. Assuming that the current position and / or target position of the operating terminal device can be achieved by combining the adjustment of the posture and camera parameters of the image terminal device, the entire working process is roughly as follows:

[0209] First, adjust the posture of the end-device of the image to reach the limit of the field of view;

[0210] If the field of view can cover the current position and / or target position of the end-device at this time, the adjustment is completed; if the field of view cannot cover the current position and / or target position of the end-device at this time, adjust the position of the end-device of the image to reach the limit of the field of view;

[0211] If the field of view can cover the current position and / or target position of the end-user instrument at this time, the adjustment is completed; if it still cannot cover the current position and / or target position of the end-user instrument, further adjust the field of view angle of the image end-user instrument to reach the limit of the field of view;

[0212] If the field of view can cover the current position and / or target position of the operating end device at this time, the adjustment is completed; if the field of view cannot cover the current position and / or target position of the operating end device at this time, adjust the depth of field of the image end device to reach the limit of the field of view, and the field of view should be able to cover the current position and / or target position of the operating end device.

[0213] In the second boot mode, if Figure 28 As shown, the above step S13, i.e., the step of generating a guidance path extending from the initial position to the target position according to the field of view of the image terminal device, includes:

[0214] Step S181: Acquire the current position of the operating end device in real time and initialize the current position as the initial position.

[0215] In step S182 , a guidance path extending from the current position to the target position is generated according to the field of view of the image terminal device, and the guidance path is located within the field of view of the image terminal device.

[0216] Preferably, in step S182, collision avoidance path planning can also be performed based on the field of view of the end-user device to generate a guidance path extending from the current position to the target position. This involves collision avoidance path planning within a local field of view, and the aforementioned collision avoidance path planning method can be referenced and will not be further elaborated here.

[0217] In one embodiment, in the second guidance mode, the method further includes: constraining adjustment of the imaging end device's field of view to always be performed while the operating end device is within the imaging end device's field of view. For example, when the operating end device exits the imaging end device's field of view, consideration may be given to generating resistance that impedes movement in the corresponding degree of freedom.

[0218] In one embodiment, in the second guidance mode, the fields of view of the image terminal device at each adjacent moment are respectively the first field of view and the second field of view, and there is an overlapping area between the first field of view and the second field of view, and the operating terminal device is limited to move to the target position through the overlapping area.

[0219] In one embodiment, in the second guidance mode, the method further includes: constraining the field of view of the imaging end-use instrument to move only in a direction toward a desired target location of the operating end-use instrument. Such a constraint can avoid or prevent ineffective adjustments of the field of view of the operating end-use instrument that are unrelated to the purpose of moving toward the target location.

[0220] In one embodiment, in the second guidance mode, the method further includes: prohibiting the operation end instrument from moving when the operation end instrument is not within the field of view of the imaging end instrument. This can prevent the operation end instrument from leaving the imaging end instrument, thereby further ensuring safety.

[0221] In one embodiment, in the second guidance mode, the method further includes: detecting whether a start instruction is obtained; and when the start instruction is obtained, determining whether the operating end device is located within the field of view of the image end device.

[0222] The start command includes, but is not limited to, being triggered when the surgical arm is mounted on the power mechanism, and being triggered by a confirmation command input by a physician after the surgical arm is mounted on the power mechanism. For example, the power mechanism may be equipped with a sensor, such as a distance sensor. When the surgical arm is mounted on the power mechanism, the sensor detects the distance between the surgical arm and the power mechanism. When the controller determines that the distance is less than or equal to a preset value, the start command is triggered.

[0223] In some embodiments, such as Figure 29 As shown, the method further includes:

[0224] Step S191 , when the operating end device is basically moved from the initial position to the target position, it is determined whether the current field of view of the image end device is the initial field of view.

[0225] The initial field of view refers to the field of view immediately before the image end device is adjusted for the first time.

[0226] Step S192: When the current field of view of the image terminal device is not the initial field of view, the current field of view of the image terminal device is adjusted to restore to the initial field of view.

[0227] In one embodiment, prior to step S192, i.e., prior to adjusting the operation end device to move from the initial position to the target position along the guide path, the method further includes: obtaining and recording the camera parameters and pose corresponding to the initial field of view determined by the imaging end device. Furthermore, in step S192, i.e., the step of adjusting the imaging end device's current field of view to restore it to the initial field of view, the imaging end device's field of view is directly restored to the initial field of view based on the recorded camera parameters and pose corresponding to the initial field of view determined by the imaging end device.

[0228] In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and the computer program is configured to be loaded and executed by a processor to implement the following steps: obtaining an initial position of an operating end device; obtaining a target position that the operating end device is expected to reach; generating a guide path extending from the initial position to the target position based on the field of view of the image end device; and adjusting the operating end device to move from the initial position to the target position along the guide path.

[0229] In one embodiment, a control device for a surgical robot is provided. Figure 30 As shown, the control device may include: a processor (processor) 501 , a communication interface (Communications Interface) 502 , a memory (memory) 503 , and a communication bus 504 .

[0230] The processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0231] The communication interface 502 is used to communicate with other devices such as various sensors, motors, solenoid valves, or network elements of other clients or servers.

[0232] The processor 501 is configured to execute a program 505 , and specifically may execute the relevant steps in the above method embodiment.

[0233] Specifically, the program 505 may include program codes, which include computer operation instructions.

[0234] The processor 505 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), one or more integrated circuits configured to implement an embodiment of the present invention, or a graphics processing unit (GPU). The one or more processors included in the control device may be processors of the same type, such as one or more CPUs or one or more GPUs; or they may be processors of different types, such as one or more CPUs and one or more GPUs.

[0235] The memory 503 is used to store the program 505. The memory 503 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0236] Program 505 can specifically be used to enable the processor 501 to perform the following operations: obtain the initial position of the operating end device; obtain the target position that the operating end device expects to reach; generate a guidance path extending from the initial position to the target position based on the field of view of the image end device; and adjust the operating end device to move from the initial position to the target position along the guidance path.

[0237] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0238] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A surgical robot, characterized in that: The distal end of the surgical robot has a plurality of operating arms, each of which includes a camera arm having an image end instrument and a surgical arm having an operating end instrument. The image end instrument has stereoscopic vision. The surgical robot also includes a controller coupled to the operating arms, and the controller is configured to: Identifying obstacles within the field of view of the imaging terminal device; Obtaining a classification mark of the obstacle; A three-dimensional environment map containing the hierarchical marks corresponding to the obstacles is constructed using a parallax method; the hierarchical marks include a first hierarchical mark and a second hierarchical mark, the obstacles include a first obstacle associated with the first hierarchical mark and a second obstacle associated with the second hierarchical mark, the first obstacle is allowed to be touched by the operating end device, and the second obstacle is prohibited from being touched by the operating end device.

2. The surgical robot according to claim 1, characterized in that: The controller is further configured to: obtain an initial position of the operating end instrument; Obtaining the target position that the operation end instrument is expected to reach; If a first guide path can be generated according to the three-dimensional environment map, adjusting the operating end device to move from the initial position to the target position along the first guide path; If the first guide path cannot be generated according to the 3D environment map, generating a second guide path according to the 3D environment map, and adjusting the operating end device to move from the initial position to the target position along the second guide path; The first guide path includes a guide path extending from the initial position to the target position, and the operating end instrument is expected to reach the guide path without contacting the obstacle; The second guide path includes a guide path extending from the initial position to the target position, in which the operation end instrument contacts the first obstacle but does not contact the second obstacle.

3. The surgical robot according to claim 2, characterized in that: The adjusting the operating end instrument to move from the initial position to the target position along the first guide path includes: adjusting the operating end instrument to move from the initial position to the target position along the first guide path at a first speed; The adjusting the operating end instrument to move from the initial position to the target position along the second guide path includes adjusting the operating end instrument to move from the initial position to the target position along the second guide path at a second speed, wherein the second speed is less than the first speed.

4. A surgical robot, characterized in that: The distal end of the surgical robot has a plurality of operating arms, each of which includes a camera arm having an image end instrument and a surgical arm having an operating end instrument. The image end instrument has stereoscopic vision. The surgical robot also includes a controller coupled to the operating arms, and the controller is configured to: Identifying obstacles within the field of view of the imaging terminal device; Obtaining a classification mark of the obstacle; A three-dimensional environment map containing the hierarchical marks corresponding to the obstacles is constructed using a parallax method, wherein the hierarchical marks include a first hierarchical mark, a second hierarchical mark, and a third hierarchical mark. The obstacles include a first obstacle associated with the first hierarchical mark, a second obstacle associated with the second hierarchical mark, and a third obstacle associated with the third hierarchical mark. The first obstacle is allowed to be passed through by the operating end device, the second obstacle is allowed to be touched by the operating end device, and the third obstacle is prohibited from being touched by the operating end device.

5. The surgical robot according to claim 4, characterized in that: The controller is further configured to: Obtaining the initial position of the operating end instrument; Obtaining the target position that the operation end instrument is expected to reach; If a first guide path can be generated according to the three-dimensional environment map, adjusting the operating end device to move from the initial position to the target position along the first guide path; If the first guide path cannot be generated according to the 3D environment map, generating a second guide path according to the 3D environment map, and adjusting the operating end device to move from the initial position to the target position along the second guide path; If the first guide path and the second guide path cannot be generated according to the 3D environment map, generating a third guide path according to the 3D environment map, and adjusting the operating end device to move from the initial position to the target position along the third guide path; The first guide path includes a guide path extending from the initial position to the target position, and the operating end instrument does not come into contact with the obstacle; The second guide path includes a guide path extending from the initial position to the target position, in which the operating end instrument contacts the first obstacle and / or the second obstacle and does not contact the third obstacle; The third guide path includes a guide path extending from the initial position to the target position, in which the operating end instrument passes through the first obstacle, contacts the first obstacle and / or the second obstacle, and does not contact the third obstacle.

6. The surgical robot according to claim 5, characterized in that: The adjusting the operating end instrument to move from the initial position to the target position along the first guide path includes: adjusting the operating end instrument to move from the initial position to the target position along the first guide path at a first speed; The adjusting the operating end instrument to move from the initial position to the target position along the second guide path includes: adjusting the operating end instrument to move from the initial position to the target position along the second guide path at a second speed, wherein the second speed is less than the first speed; Adjusting the operating end instrument to move from the initial position to the target position along the third guide path includes: adjusting the operating end instrument to move from the initial position to the target position along the third guide path at a third speed, wherein the third speed is less than the first speed or the second speed.

7. The surgical robot according to claim 2 or 5, characterized in that: The surgical robot further includes a display, the controller is coupled to the display, and the controller is further configured to: Displaying the three-dimensional environment map on the display; The position of the operation end instrument is displayed in real time in the three-dimensional environment map.

8. The surgical robot according to claim 2 or 5, characterized in that: The step of obtaining the target position that the operation end instrument is expected to reach comprises: Obtaining an input operation mode, the operation mode including a first operation mode and a second operation mode, the first operation mode being used to guide the operation end instrument to be inserted into a target position, and the second operation mode being used to guide the operation end instrument to be withdrawn to the target position; The target position that the operating end instrument is expected to reach is determined according to the acquired operating mode.

9. A computer-readable storage medium, characterized in that Applicable to a surgical robot, wherein the distal end of the surgical robot has multiple operating arms, the operating arms including a camera arm having an imaging end instrument and a surgical arm having an operating end instrument, the imaging end instrument having stereoscopic vision, the computer-readable storage medium storing a computer program, the computer program being configured to be loaded and executed by a processor to implement the following steps: Identifying obstacles within the field of view of the imaging terminal device; Obtaining a classification mark of the obstacle; A 3D environment map containing the hierarchical markers corresponding to the obstacles is constructed using a parallax method; the hierarchical markers include a first hierarchical marker and a second hierarchical marker, the obstacles include a first obstacle associated with the first hierarchical marker and a second obstacle associated with the second hierarchical marker, the first obstacle is allowed to be touched by the end-user device, and the second obstacle is prohibited from being touched by the end-user device; Alternatively, the grading mark includes a first grading mark, a second grading mark and a third grading mark, and the obstacle includes a first obstacle associated with the first grading mark, a second obstacle associated with the second grading mark, and a third obstacle associated with the third grading mark, the first obstacle is allowed to be passed through by the operating end instrument, the second obstacle is allowed to be touched by the operating end instrument, and the third obstacle is prohibited from being touched by the operating end instrument.

10. A control device for a surgical robot, characterized in that: The distal end of the surgical robot has a plurality of operating arms, each of which includes a camera arm having an image terminal instrument and a surgical arm having an operating terminal instrument. The image terminal instrument has stereoscopic vision. The control device includes: memory for storing computer programs; and a processor for loading and executing the computer program; The computer program is configured to be loaded and executed by the processor to implement the following steps: Identifying obstacles within the field of view of the imaging terminal device; Obtaining a classification mark of the obstacle; A 3D environment map containing the hierarchical markers corresponding to the obstacles is constructed using a parallax method; the hierarchical markers include a first hierarchical marker and a second hierarchical marker, the obstacles include a first obstacle associated with the first hierarchical marker and a second obstacle associated with the second hierarchical marker, the first obstacle is allowed to be touched by the end-user device, and the second obstacle is prohibited from being touched by the end-user device; Alternatively, the grading mark includes a first grading mark, a second grading mark and a third grading mark, and the obstacle includes a first obstacle associated with the first grading mark, a second obstacle associated with the second grading mark, and a third obstacle associated with the third grading mark, the first obstacle is allowed to be passed through by the operating end instrument, the second obstacle is allowed to be touched by the operating end instrument, and the third obstacle is prohibited from being touched by the operating end instrument.

Images