Surgical robot, graphical control device, and graphical display method thereof

Abstract

The present invention relates to a surgical robot and its graphical control device and graphical display method. The surgical robot includes: an input unit; a display; an operating arm having a feature point sequence composed of orderly arranged feature points, wherein the feature points represent joints; and a controller coupled to the input unit, the display, and a sensor, and configured to: obtain the feature point sequence of the operating arm and its corresponding kinematic model; obtain joint variables sensed by the sensor and obtain a virtual camera selected by the input unit; determine the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera based on the kinematic model and the joint variables; orderly fit and connect each projection point to generate a projection image of the operating arm; and display the projection image on the display. The surgical robot of the present invention facilitates a doctor to observe the motion state of the operating arm in all directions.

Description

[0001] This application is a divisional application submitted to the China Patent Office on October 8, 2020, with application number CN202011068091.4 and application name "Surgical robot and its graphical control device, graphical display method". The full text of the case is incorporated into this application by reference. Technical Field

[0002] The present invention relates to the field of medical devices, and in particular to a surgical robot and a graphical control device and a graphical display method thereof. Background Art

[0003] Minimally invasive surgery refers to a procedure performed inside the human body using modern medical devices such as laparoscopes and thoracoscopes. Compared to traditional surgical methods, minimally invasive surgery offers advantages such as less trauma, less pain, and faster recovery.

[0004] With technological advancements, minimally invasive surgical robotics have matured and are now widely used. A surgical robot consists of a master console and slave operating devices. The slave operating devices include multiple manipulators, including a camera arm with an imaging end-use instrument and a surgical arm with an operating end-use instrument. The master console includes a display and a handle. The surgeon manipulates the handle to control the movement of the camera arm or surgical arm, while viewing the field of view provided by the camera arm on the display.

[0005] However, in most scenarios, such as Figure 1 As shown, the field of view provided by camera arm 34A' often only allows for a partial view of surgical arm 34B'. The visible area is the visible region, while a large area remains invisible. The surgeon cannot observe the status of camera arm 34A' within the visible region, nor can they detect any collisions, whether already occurring or potentially occurring, between surgical arms 34B' or between surgical arm 34B' and camera arm 34A' within the invisible region. This situation can easily lead to surgical safety issues. Summary of the Invention

[0006] Based on this, it is necessary to provide a surgical robot and its graphical control device and graphical display method that facilitate doctors to observe the movement state of the operating arm in all directions.

[0007] On the one hand, the present invention provides a surgical robot, comprising: an input part; a display; an operating arm, comprising multiple joints and sensors for sensing joint variables of the joints, the operating arm having a feature point sequence composed of multiple feature points arranged in an orderly manner for associating the corresponding joints; and a controller, the controller being coupled to the input part, the display and the sensor, and being configured to: obtain the feature point sequence of the operating arm and its corresponding kinematic model; obtain the joint variables sensed by the sensor, and obtain a virtual camera selected by the input part; determine the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to the kinematic model and the joint variables; orderly fit and connect each of the projection points to generate a projection image of the operating arm; and display the projection image on the display.

[0008] Wherein, in the step of determining the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to the kinematic model and the joint variables, the controller is configured to: obtain the first position of each feature point in the feature point sequence in the reference coordinate system according to the kinematic model and the joint variables; convert each first position into a second position in the virtual camera coordinate system; obtain the virtual focal length of the virtual camera and determine the projection plane of the virtual camera according to the virtual focal length; obtain the projection point of each second position on the projection plane according to the virtual focal length.

[0009] Wherein, in the step of determining the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to the kinematic model and the joint variables, the controller is configured to: obtain the first position of each feature point in the feature point sequence in the reference coordinate system according to the kinematic model and the joint variables; convert each first position into a second position in the virtual camera coordinate system; obtain the contour information of the joint corresponding to each feature point; and obtain the projection point of each second position on the projection plane in combination with the virtual focal length and the contour information.

[0010] Wherein, in the step of orderly fitting and connecting the projection points to generate the projection image of the operating arm, the controller is configured to: orderly fit and connect the projection points in combination with the contour information to generate the projection image of the operating arm.

[0011] Among them, in the step of orderly fitting and connecting each of the projection points to generate the projection image of the operating arm, the controller is configured to: orderly connect each of the projection points according to the order of the feature points corresponding to each of the projection points in the feature point sequence to generate the projection image of the operating arm.

[0012] Among them, in the step of orderly fitting and connecting each of the projection points to generate the projection image of the operating arm, the controller is configured to: obtain the icon of the end device of the operating arm; determine the position of the end device in the projection plane of the virtual camera according to the joint variables and the kinematic model; rotate and / or scale the icon according to the position of the end device in the projection plane of the virtual camera; splice the processed icon at the projection point at the far end to generate the projection image.

[0013] Wherein, in the step of obtaining the icon of the end instrument of the operating arm, the controller is configured to: obtain the type of the operating arm, and match the icon of the end instrument of the operating arm according to the type.

[0014] In which, the virtual camera has a selectable virtual focal length and / or virtual aperture, and the controller is configured to: obtain the virtual focal length and / or virtual aperture of the virtual camera selected by the input unit, and determine the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera in combination with the virtual focal length and / or virtual aperture, the kinematic model and the joint variables in the step of determining the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera.

[0015] Wherein, the controller is configured to execute, before the step of displaying the projection image on the display, the following steps: detecting whether the projection image is distorted; when it is detected that the projection image is distorted, increasing the virtual focal length of the virtual camera and re-entering the step of determining the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera by combining the virtual focal length and / or virtual aperture, the kinematic model and the joint variables; and when it is detected that the projection image is not distorted, entering the step of displaying the projection image on the display.

[0016] Wherein, the controller is configured to: obtain the position of each of the projection points in the reference coordinate system; obtain the number of first projection points among the projection points that fall into the edge area of ​​the projection plane or the edge area of ​​the display window in the display for displaying the projection image; calculate the ratio of the number of the first projection points to the total number of the projection points, and when the ratio reaches a threshold value, determine that the projection image is distorted.

[0017] In which, the operating arm includes a camera arm with an image end device; the controller is further configured to: obtain camera parameters of the image end device of the camera arm, and calculate the visible area of ​​the image end device based on the camera parameters, the camera parameters including focal length and aperture; determine the posture of the image end device in a reference coordinate system based on the joint variables and kinematic model of the camera arm; convert the visible area of ​​the image end device into the visible area of ​​the virtual camera based on the conversion relationship between the posture of the image end device and the posture of the virtual camera in the reference coordinate system; calculate the boundary line of the visible area of ​​the virtual camera on the projection plane, and display the boundary line in the projection image displayed on the display.

[0018] Wherein, the operating arm includes a camera arm with an image end instrument and a surgical arm with an operating end instrument; the controller is also configured to perform the following steps in the step of orderly fitting and connecting each of the projection points to generate a projection image of the operating arm: obtaining an operating image of the surgical area captured by the image end instrument of the camera arm; identifying the characteristic parts of the surgical arm from the operating image; matching the associated first feature point from the feature point sequence based on the identified characteristic parts; orderly fitting and connecting each of the projection points and marking the first projection point associated with the first feature point in the projection points and the line segment connected to the first projection point to generate a projection image of the operating arm.

[0019] Wherein, the feature point sequence also includes an unmatched second feature point. After the step of matching the associated first feature point from multiple feature point sequences based on the identified feature parts, the controller is configured to: obtain the unmatched second feature point; generate an image model of the corresponding feature part in combination with the contour information, joint variables and kinematic model of the feature part corresponding to the second feature point; convert the image model into a supplementary image in the coordinate system of the image end instrument; splice the supplementary image to the image of the feature part corresponding to the first feature point according to the sequential relationship between the second feature point and the first feature point in the feature point sequence to form a complete sub-image of the operating arm in the operating image; and display the operating image with the complete sub-image of the operating arm.

[0020] In which, the controller is also configured to: obtain the maximum motion range of the operating arm in a first direction; calculate the motion amount of the operating arm in the first direction based on the joint variables and kinematic model of the operating arm; generate an icon based on the maximum motion range and the motion amount in the first direction; and display the icon on the display.

[0021] Wherein, the first direction is the front-back feeding direction.

[0022] Wherein, the icon is a progress bar or a pie chart.

[0023] The controller is configured to darken or lighten the color of the variable-length bar accordingly when the amount of movement increases or decreases.

[0024] The controller is configured to: detect a first operating arm currently being controlled from the operating arms, and identify the first operating arm in the projection image.

[0025] The multiple virtual cameras selectable by the input unit have different postures in the reference coordinate system.

[0026] The position and posture of the virtual camera in the reference coordinate system is determined based on the reachable workspace of the manipulator in the reference coordinate system.

[0027] The position and posture of the virtual camera in the reference coordinate system is determined based on the union space of the reachable workspace of the operating arm in the reference coordinate system.

[0028] The position of the virtual camera in the reference coordinate system is always outside the union space, and the posture of the virtual camera in the reference coordinate system is always facing the union space.

[0029] The virtual camera has an optional virtual focal length, and the position of the virtual camera is outside a first area, and the first area is an area determined by the union space that can be just visible at the shortest virtual focal length.

[0030] The virtual camera has a selectable virtual focal length, and the position of the virtual camera is located within a second area, which is an area determined by the longest virtual focal length that can just see the union space.

[0031] The posture of the virtual camera is always oriented toward the center of the union space.

[0032] The controller is configured to display the projection image in a first display window of the display and generate a plurality of selectable icons of the virtual cameras in the first display window.

[0033] The relative position of the icon and the projected image is fixed and changes with the change of the viewpoint of the projected image.

[0034] There are six icons, which respectively correspond to the virtual imaging of the operating arm from the left side, right side, top side, bottom side, front side and back side to generate the projection image under the corresponding viewpoint.

[0035] The icon is displayed as an arrow pattern or a camera pattern, and any one of the icons selected by rotation corresponds to a virtual camera.

[0036] The icon is displayed as a rotatable sphere, and any position to which the icon is rotated corresponds to one of the virtual cameras.

[0037] Wherein, the controller is configured to execute, in the step of obtaining the virtual camera selected by the input unit, the following: obtaining the virtual camera selected by the input unit and at least two target positions of the virtual camera input by the input unit; determining the target projection point of the projection plane of each feature point in the feature point sequence under each target position of the virtual camera according to the preset movement speed of the virtual camera and based on the kinematic model and the joint variables; orderly fitting and connecting the target projection points under each target position to generate a target projection image of the operating arm; generating an animation based on each target projection image; and playing the animation on the display at a preset frequency.

[0038] Wherein, the controller is configured to execute, in the step of obtaining the virtual camera selected by the input unit, the following: obtaining the motion trajectory of the virtual camera input by the input unit; discretizing the motion trajectory to obtain each discrete position of the virtual camera as a target position; determining the target projection point of the projection plane of each feature point in the feature point sequence under each target position of the virtual camera according to the preset motion speed of the virtual camera and based on the kinematic model and the joint variables; orderly fitting and connecting each target projection point under each target position to generate a target projection image of the operating arm; generating an animation based on each target projection image; and playing the animation on the display at a preset frequency.

[0039] Wherein, the operating arm includes a camera arm with an image terminal device; the controller is configured to: obtain an operating image of the surgical area captured by the image terminal device; display the operating image on the display; and display the projection image suspended in the operating image.

[0040] Wherein, the controller is configured to perform, in the step of displaying the projection image suspended in the operation image: obtaining the overlapping area between the operation image and the projection image, and obtaining the first image attribute of the part of the operation image in the overlapping area; and adjusting the second image attribute of the part of the projection image in the overlapping area according to the first image attribute.

[0041] The controller is configured to: when a first operating arm among the operating arms reaches a threshold of an event, mark at least a portion of the first operating arm in the projection image and display the mark on the display.

[0042] The threshold is a warning threshold, and the event is a situation to be avoided.

[0043] The warning threshold is based on a range of motion of at least one joint in the first operating arm, and the situation to be avoided is a limitation of the range of motion of at least one joint in the first operating arm.

[0044] The warning threshold is based on a distance between the first operating arm and a second operating arm in the manipulator, and the situation to be avoided is a collision between the first operating arm and the second operating arm.

[0045] Wherein, the controller is configured to: obtain the minimum distance between the first operating arm and the second operating arm and determine the relationship between the minimum distance and the warning threshold; when the minimum distance reaches the threshold corresponding to the situation to be avoided but the warning threshold has not yet reached, form a first identifier to identify the minimum distance point on the sub-image of the first operating arm and the second operating arm.

[0046] The controller is configured to: when the minimum distance reaches a situation to be avoided, form a second mark for marking a minimum distance point on the model of the first operating arm and the second operating arm.

[0047] Wherein, the controller is configured to execute, in the steps of obtaining the minimum distance between the first operating arm and the second operating arm and judging the relationship between the minimum distance and the warning threshold, the following steps: constructing the corresponding geometric models of the first operating arm and the second operating arm based on the kinematic models and structural features of the first operating arm and the second operating arm; discretizing the geometric models of the first operating arm and the second operating arm to obtain a set of external information points of the first operating arm and the second operating arm in a reference coordinate system; determining the minimum distance between the first operating arm and the second operating arm based on the set of external information points of the first operating arm and the second operating arm; and identifying the minimum distance point on the sub-image of the first operating arm and the second operating arm, including: determining the minimum distance point corresponding to the minimum distance, and identifying the minimum distance point on the model of the first operating arm and the second operating arm.

[0048] Wherein, the controller is configured to: when the minimum distance reaches the warning threshold, determine the collision direction according to the position of the minimum distance point on the sub-image of the first operating arm and the second operating arm in the reference coordinate system; and identify the collision direction between the first operating arm and the second operating arm.

[0049] The surgical robot includes a mechanical handle coupled to the controller and used to control the movement of the operating arm, and the controller is configured to generate resistance that hinders the movement of the mechanical handle in the associated direction according to the collision direction.

[0050] Wherein, the mechanical handle has multiple joint components and drive motors for driving the movement of each joint component, each drive motor is coupled to the controller, and the controller is configured to: generate a reverse torque for the drive motor in the associated direction according to the resistance.

[0051] The controller is configured such that when the minimum distance is between the warning threshold and a threshold corresponding to a situation to be avoided, the magnitude of the reverse torque is negatively correlated with the magnitude of the minimum distance.

[0052] On the other hand, the present invention provides a graphical display method for a surgical robot, which includes: an input part; a display; an operating arm, including multiple joints and sensors for sensing joint variables of the joints, the multiple joints constitute positioning degrees of freedom and / or orientation degrees of freedom, and the operating arm has a feature point sequence composed of orderly arranged feature points, and the feature points represent the joints; the control method includes the following steps: obtaining the feature point sequence of the operating arm and its corresponding kinematic model; obtaining the joint variables sensed by the sensor, and obtaining a virtual camera selected by the input part; determining the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to the kinematic model and the joint variables; orderly fitting and connecting each of the projection points to generate a projection image of the operating arm; and displaying the projection image on the display.

[0053] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to be loaded and executed by a processor to implement the steps of the graphical display method as described in any of the above embodiments.

[0054] On the other hand, the present invention provides a graphical control device for a surgical robot, comprising: a memory for storing a computer program; and a processor for loading and executing the computer program; wherein the computer program is configured to be loaded and executed by the processor to implement the steps of the graphical display method described in any of the above embodiments.

[0055] The surgical robot, its graphical control device, and its graphical display method of the present invention have the following beneficial effects:

[0056] By configuring a virtual camera to simulate a real camera to image the manipulator, it is possible to observe all manipulators and each manipulator as a whole, which is beneficial for doctors to observe the movement status of the manipulator in all directions, and thus contributes to the reliability and continuity of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a partial schematic diagram of a conventional surgical robot in a surgical state;

[0058] Figure 2 This is a schematic structural diagram of an embodiment of a surgical robot according to the present invention;

[0059] Figure 3 for Figure 1 A partial schematic diagram of an embodiment of a surgical robot is shown;

[0060] Figure 4 This is a flowchart of an embodiment of a surgical robot graphical display method;

[0061] Figure 5 This is a schematic diagram of the structure of the operating arm and power unit of the surgical robot;

[0062] Figure 6 for Figure 1 A schematic diagram of a virtual camera layout of an embodiment of a surgical robot is shown;

[0063] Figure 7 for Figure 1 A schematic diagram of a configuration interface of a virtual camera of an embodiment of a surgical robot is shown;

[0064] Figure 8 for Figure 4 The schematic diagram of projection imaging of an embodiment of the graphical display method is shown;

[0065] Figures 9 and 10 They are respectively schematic diagrams of display interfaces of an embodiment of a graphical display method;

[0066] Figure 11 This is a flowchart of an embodiment of a surgical robot graphical display method;

[0067] Figures 12 and 13 They are respectively schematic diagrams of display interfaces of an embodiment of a graphical display method;

[0068] Figure 14 This is a flowchart of an embodiment of a surgical robot graphical display method;

[0069] Figure 15 A schematic diagram of a display interface of an embodiment of a graphical display method;

[0070] Figures 16 and 17They are respectively flow charts of an embodiment of a graphical display method for a surgical robot;

[0071] Figure 18 A schematic diagram of a display interface of an embodiment of a graphical display method;

[0072] Figure 19 This is a flowchart of an embodiment of a surgical robot graphical display method;

[0073] Figures 20 and 21 They are Figure 1 A schematic diagram of a configuration interface of a virtual camera of an embodiment of a surgical robot is shown;

[0074] Figures 22 and 23 They are respectively flow charts of an embodiment of a graphical display method for a surgical robot;

[0075] Figure 24 for Figure 1 A schematic diagram of a configuration interface of a virtual camera of an embodiment of a surgical robot is shown;

[0076] Figures 25 to 27 They are respectively flow charts of an embodiment of a graphical display method for a surgical robot;

[0077] Figure 28 Schematic diagram of observing the manipulator with a large field of view;

[0078] Figure 29 To adopt Figure 28 Schematic diagram of the display interface generated by the large field of view shown;

[0079] Figure 30 To adjust Figure 28 Schematic diagram of the display interface generated after the wide field of view angle is shown;

[0080] Figure 31 This is a flowchart of an embodiment of a surgical robot graphical display method;

[0081] Figure 32 for Figure 1 A partial schematic diagram of an embodiment of a surgical robot is shown;

[0082] Figures 33 and 34 They are respectively schematic diagrams of display interfaces of an embodiment of a graphical display method;

[0083] Figures 35 to 38 They are respectively flow charts of an embodiment of a graphical display method for a surgical robot;

[0084] Figure 39 This is a schematic structural diagram of another embodiment of the surgical robot of the present invention;

[0085] Figure 40Schematic diagram of the structure of a graphical control device for a surgical robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0086] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0087] It should be noted that when an element is referred to as being "disposed on" another element, it may be directly on the other element or there may also be an element centered. When an element is considered to be "connected" to another element, it may be directly connected to the other element or there may be an element centered at the same time. When an element is considered to be "coupled" to another element, it may be directly coupled to the other element or there may be an element centered at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in the present invention are for illustrative purposes only and do not represent the only implementation method. The terms "distal end" and "proximal end" used in the present invention are used as directional words, which are commonly used terms in the field of interventional medical devices, where "distal end" refers to the end away from the operator during surgery, and "proximal end" refers to the end close to the operator during surgery. The terms "first / second" and the like used in the present invention represent a component and two or more components of a type having common characteristics.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this invention pertains. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0089] like Figures 2 to 3 As shown in FIG, they are respectively a structural schematic diagram of an embodiment of a surgical robot of the present invention and a partial schematic diagram thereof.

[0090] The surgical robot includes a master console 2 and a slave operating device 3 controlled by the master console 2. The master console 2 has a motion input device 21 and a display 22. The surgeon operates the motion input device 21 to send control commands to the slave operating device 3, causing the slave operating device 3 to perform corresponding operations based on the surgeon's control commands. The surgeon also observes the surgical area through the display 22. The slave operating device 3 includes an arm mechanism comprising a robotic arm 30 and a manipulator arm 31 detachably mounted at the distal end of the robotic arm 30. The robotic arm 30 comprises a base and a connection assembly, each of which has multiple joint assemblies. The manipulator arm 31 comprises a connecting rod 32, a connection assembly 33, and an end-use instrument 34, each of which has multiple joint assemblies. The end-use instrument 34 is adjusted by adjusting the joint assemblies of the manipulator arm 31. The end-use instrument 34 includes an imaging end-use instrument 34A and an operating end-use instrument 34B. The imaging end-use instrument 34A is used to capture images within the field of view, and the display 22 is used to display these images. The operating end instrument 34B is used to perform surgical operations such as cutting and suturing.

[0091] Figure 1 The surgical robot on display is a single-port surgical robot, in which each operating arm 31 is inserted into the patient's body through the same puncture device 4 installed at the distal end of the robotic arm 30. In a single-port surgical robot, the doctor generally only controls the operating arm 31 to complete basic surgical operations. At this time, the operating arm 31 of the single-port surgical robot should have both positional freedom (i.e., positioning freedom) and posture freedom (i.e., orientation freedom) to achieve changes in posture within a certain range. For example, the operating arm 31 has horizontal movement freedom x, vertical movement freedom y, rotation freedom α, pitch freedom β, and yaw freedom γ. The operating arm 31 can also achieve forward and backward movement freedom (i.e., feed freedom) z under the drive of the distal joint of the robotic arm 30, i.e., the power mechanism 301. For example, the power mechanism 301 has a guide rail and a power unit slidingly arranged on the guide rail, and the operating arm is detachably mounted on the power unit. On the one hand, the sliding of the power unit on the guide rail provides the operating arm 31 with a degree of freedom z for moving forward and backward. On the other hand, the power unit provides power to the joints of the operating arm 31 to realize the remaining five degrees of freedom (i.e., [x, y, α, β, γ]).

[0092] The surgical robot also includes a controller. This controller can be integrated into the master console 2 or the slave operating device 3. Of course, the controller can also be independent of the master console 2 and the slave operating device 3. For example, the controller can be deployed locally or in the cloud. The controller can be composed of more than one processor.

[0093] The surgical robot also includes an input unit. The input unit can be integrated into the main console 2. The input unit can also be integrated into the slave operating device 3. Of course, the input unit can also be independent of the main console 2 and the slave operating device 3. The input unit can be, for example, a mouse, a keyboard, a voice input device, or a touch screen. In one embodiment, a touch screen is used as the input unit. The touch screen is set on the armrest of the main console 2, and configurable information can be displayed on the touch screen, such as the virtual camera to be selected and its virtual camera parameters. In other embodiments, the configurable information can be displayed on the display 22 of the main console 2 or on another external display.

[0094] The operating arm 31 also includes sensors for sensing joint variables of the joints. These sensors include angle sensors for sensing the rotational motion of the joint components and displacement sensors for sensing the linear motion of the joint components. Specifically, the appropriate sensors can be configured according to the type of joint.

[0095] The controller is coupled to these sensors, and is coupled to the input unit and the display 22 of the main console 2 .

[0096] In one embodiment, a graphical display method for a surgical robot is provided, which can be executed by a controller. Figure 4 , the graphical display method comprises the following steps:

[0097] Step S11 : obtaining a feature point sequence of the manipulator and a kinematic model corresponding to the manipulator.

[0098] For example, Figure 5 As shown, a storage unit 311 is installed on the abutting surface of the driving box 310 of the manipulator 31 against the power unit 302 of the power mechanism 301. Correspondingly, a reading unit 303 is installed on the abutting surface of the power unit 302 against the driving box 310, which is matched with the storage unit 311. The reading unit 303 is coupled to the controller. When the manipulator 31 is installed on the power unit 302, the reading unit 303 communicates with the coupled storage unit 311 and reads relevant information from the storage unit 311. The storage unit 311 is, for example, a memory or an electronic tag. The storage unit stores, for example, one or more combinations of the type of manipulator, a sequence of feature points, and a kinematic model pre-constructed based on the connecting rod parameters of the manipulator. The feature point sequence includes multiple feature points, which can represent any characteristic part of the manipulator. The characteristic part can refer to one or more of the end device, joint, or connecting rod of the manipulator.

[0099] For example, the storage unit 311 stores the feature point sequence and the kinematic model of the operating arm, and the required feature point sequence and kinematic model of the operating arm can be directly obtained from the storage unit 311 .

[0100] For another example, the storage unit 311 only stores the type of manipulator, while other storage units coupled to the controller store feature point sequences and kinematic models of different types of manipulators. The feature point sequence and kinematic model of the corresponding manipulator can be obtained based on the acquired type of the manipulator.

[0101] Step S12: obtaining the joint variables of each joint in the operating arm sensed by the sensor.

[0102] Joint variables refer to the joint amount of the rotational joint and / or the joint offset of the translational joint in the joint.

[0103] Step S13: Acquire the virtual camera selected by the input unit.

[0104] As the name implies, a virtual camera is a non-actual camera that does not capture real images of objects. It only reflects a concept of viewpoint, such as Figure 6 As shown, Figure 6 A spatial distribution diagram of a virtual camera relative to the manipulator arm is illustrated. The default virtual camera 100 can be defined as any one of them, for example, the default selection is the virtual camera 100 on the puncture device 4. The virtual camera can be configured with parameters. The virtual camera parameters (i.e., configuration parameters) of the virtual camera include at least a (virtual) posture, corresponding to the camera parameters of the real camera such as focal length and / or aperture. The virtual camera parameters also include a virtual focal length and / or virtual aperture. Typically, the (virtual) focal length corresponds to the field of view of the adjustable (virtual) camera, and the (virtual) aperture corresponds to the depth of field of the adjustable (virtual) camera. In one embodiment, the virtual camera parameters can also be described as including the field of view and / or depth of field. For the virtual camera, the field of view and / or depth of field are also virtual. Even if the camera, focal length, and aperture are virtual, the imaging principle of the real camera can also be used to achieve the purpose of the present invention. Different virtual camera parameters can show different imaging effects to the doctor.

[0105] These virtual camera parameters can be stored in a system configuration file stored in the surgical robot's memory and can be accessed by the controller reading the system configuration file. These virtual camera parameters can also be manually set by the surgeon before or during surgery as needed through an input unit coupled to the controller. This setting method is on-demand. For example, these virtual camera parameters can be obtained by entering relevant data in a text control or by selecting from an option control.

[0106] The virtual camera's position can be the same as that of the real camera (i.e., the image terminal device) so that the manipulator is observed from the same viewpoint as the real camera. The virtual camera's position can also be different from that of the real camera so that the manipulator is observed from a different viewpoint than the real camera. Generally, the virtual camera's position can be selected to be different from that of the real camera for observation, which helps to obtain more comprehensive information about the manipulator. For example, the manipulator can also be a camera arm so that it can be observed by the virtual camera.

[0107] The obtained virtual camera includes the posture of the virtual camera and the virtual camera parameters of the virtual camera.

[0108] From a purely theoretical perspective, the maximum virtual focal length can be infinite, while the minimum can approach 0. For example, the virtual focal lengths available for the virtual camera can be configured to mimic the lens of a real camera with a focal length range of 2mm to 70mm, for example, configurable virtual focal lengths of 2mm to 50mm, such as 2mm, 5mm, 10mm, and 20mm. The position of the virtual camera is then configured based on the shortest and / or longest virtual focal lengths. The smaller the virtual focal length, the larger the projected image, allowing for better viewing of local details; the larger the virtual focal length, the smaller the projected image, allowing for better overall viewing.

[0109] Theoretically, the maximum virtual aperture can be infinite, and the minimum can approach 0 infinitely. For example, the virtual camera's selectable virtual apertures can be configured to mimic the aperture range of real camera lenses, such as F1, F1.2, F1.4, F2, F2.8, F4, F5.6, F8, F11, F16, F22, F32, F44, and F64. For example, virtual apertures of F2.8, F4, F5.6, and F8 can be configured. The larger the virtual aperture, the smaller the depth of field; and the smaller the virtual aperture, the greater the depth of field.

[0110] like Figure 7 , which illustrates a configuration interface for a virtual camera, wherein the virtual camera's position, virtual focal length, and virtual aperture can be selected.

[0111] Step S14 , determining the projection point of each feature point in the feature point sequence of the manipulator on the projection plane of the virtual camera according to the kinematic model and joint variables of the manipulator.

[0112] For example, the first position of each feature point in the feature point sequence in the reference coordinate system can be calculated based on the kinematic model and joint variables of the manipulator, and then the first position can be converted into the second position in the virtual camera coordinate system based on the coordinate transformation relationship between the virtual camera coordinate system and the reference coordinate system, and finally the second position can be projected into the third position of the projection plane of the virtual camera as the projection point. Among them, the projection plane of the virtual camera is usually associated with the virtual focal length of the virtual camera, so the projection plane of the virtual camera can usually be determined according to the virtual focal length of the virtual camera, which is equivalent to obtaining the projection point of each second position on the projection plane according to the virtual focal length. The above-mentioned reference coordinate system can be set anywhere, and is usually considered to be set on the surgical robot, and preferably, it is set on the slave operating device. For example, the reference coordinate system is the base coordinate system of the slave operating device. For another example, the reference coordinate system is the tool coordinate system of the robotic arm in the slave operating device.

[0113] Furthermore, the implementation process of obtaining the projection point of each second position on the projection plane based on the virtual focal length can be divided into the following two steps: the first step is to obtain the contour information of the joint represented (associated) by each feature point, and the contour information includes, for example, size information and / or line type information; and the second step is to obtain the projection point of each second position on the projection plane in combination with the virtual focal length and contour information.

[0114] Among them, the above-mentioned first position, second position and third position can be a point position or an area composed of multiple points. It can be seen that the projection point can be understood as a point or a point set, which is specifically determined according to the selection of the feature point. That is, when the feature point itself is selected as a point of the feature part, the projection point is a point; and when the feature point itself is selected as a point set of the feature part (that is, the concept of "area"), the projection point corresponds to a point set (that is, area). If the point set of the feature point can reflect the geometric size of the feature part, then the projection point can also reflect the geometric size of the feature part, and then the real structure of the operating arm can be approximately displayed, which is more conducive to displaying the motion state of the operating arm.

[0115] Furthermore, the more or more densely selected feature points are, the more closely they can represent the actual structure of the manipulator. For example, by fitting these projection points, the linear features of the manipulator, such as straight lines, curves, and radians, can be more accurately represented.

[0116] In this step, more specifically, the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera can be determined in combination with the virtual focal length (virtual field of view) and / or virtual aperture (depth of field), kinematic model and joint variables of the virtual camera.

[0117] See Figure 8 , Figure 8The operating arm has a feature point sequence, which includes feature points Q1, Q2, Q3, and Q4. Under virtual imaging of a virtual camera, a projection point sequence is obtained on the projection plane, which corresponds to q1, q2, q3, and q4.

[0118] For example, taking feature points Q1 and Q2 as examples, the positions of Q1 and Q2 in space are obtained as Q1(X1, Y1, Z1) and Q2(X2, Y2, Z2) respectively based on the kinematic model and joint variables. The projection points q1(x1, y1) and q2(x2, y2) of feature points Q1 and Q2 on the projection plane can be determined by combining the virtual focal length using the following formula:

[0119] x1=fx*(X1 / Z1)+cx;

[0120] y1=fy*(Y1 / Z1)+cy;

[0121] x2=fx*(X12 / Z12)+cx;

[0122] y2=fy*(Y12 / Z12)+cy;

[0123] Where fx is the horizontal focal length, fy is the vertical focal length, cx is the horizontal offset relative to the optical axis, and cy is the vertical offset relative to the optical axis. The values ​​of fx and fy can be equal or different.

[0124] Step S15 : sequentially fitting and connecting the projection points to generate a projection image of the operating arm.

[0125] This step can connect the projection points in order according to the order of the feature points corresponding to the projection points in the feature point sequence to generate the projection image of the operating arm. The "orderly" here refers to the order of correspondence between the projection points, rather than which projection point is connected first and which projection point is connected later. It is feasible to connect from the proximal end to the distal end corresponding to the real structure of the operating arm, or from the distal end to the proximal end, or from the middle to both ends in sequence according to the order of correspondence between the projection points.

[0126] In addition, this step can combine the contour information of the characteristic parts to orderly fit and connect the projection points to generate the projection image of the manipulator. For example, if the actual geometric dimensions of the characteristic parts of the manipulator are roughly the same, the projection points can be connected by line segments of the same size as the projection points.

[0127] Furthermore, "fitting connection" can refer to a connection method that closely follows the linear characteristics of a characteristic part. For example, for an overall linear manipulator, adjacent projection points can be connected with straight lines. For another example, for a manipulator that is at least partially curved, projection points corresponding to the curved portion can be connected with curved lines. Fitting connections can reflect the linear characteristics of the manipulator.

[0128] Continue reading Figure 8 , by fitting and connecting q1, q2, q3 and q4 in an orderly manner, the projection image corresponding to the operating arm can be obtained.

[0129] Step S16: displaying the projected image on the display.

[0130] Through the above steps S11 to S16, the doctor can observe the motion status of all operating arms and the complete characteristic parts of each operating arm through the projection image without any blind spots, which helps to assist the doctor to perform the operation reliably and continuously. Figure 33 See, Figure 32 In the embodiment, a potential collision between the operating arms 31b and 31c may occur outside the real visual area of ​​the real camera and cannot be observed. However, through the above steps S11 to S16, the potential collision can be observed with the help of the generated projection image.

[0131] Figure 9 A display interface is illustrated that generates only a projected image of the surgical arm. Figure 10 Another display interface is illustrated, which generates projected images of the surgical arm and the camera arm at the same time. Figure 9 and Figure 10 The projection images in the figure reflect the motion state of each feature point of the corresponding operating arm.

[0132] In the above embodiment, since the projection image is formed by a series of orderly connected projection points, these feature points may not be easy to intuitively reflect the structural features of the end device of the operating arm. Therefore, in order to more easily reflect the structural features of the end device, such as Figure 11 As shown, the controller may be configured to execute, in the above step S15, i.e., the step of sequentially fitting and connecting the projection points to generate the projection image of the operating arm, the following steps:

[0133] Step S151: Obtain the icon of the end instrument of the operating arm.

[0134] For example, the type of the manipulator arm can be first obtained, and then the icon of the end-use instrument of the manipulator arm can be matched based on the type of the manipulator arm. For another example, the icon of the end-use instrument of the manipulator arm can be matched based on the obtained feature point sequence. These icons are pre-associated with the type of the manipulator arm and / or the feature point sequence and stored in the storage unit.

[0135] Step S152: determining the position and posture of the end device in the virtual camera coordinate system based on the joint variables and the kinematic model.

[0136] Step S153 : rotating and / or scaling the icon according to the position of the end device in the virtual camera coordinate system.

[0137] The icon is usually scaled according to the position of the end device in the virtual camera coordinate system, and the icon is rotated according to the posture (direction) of the end device in the virtual camera coordinate system.

[0138] In step S154 , the processed icons are spliced ​​at the remote projection point to generate a projection image.

[0139] See Figure 12 , Figure 12 The figure shows a display interface that displays the shape of the end instrument of the corresponding operating arm in the projection image. Of course, the projection image does not reflect the outline shape of the corresponding operating arm. Figure 13 , Figure 13 Another display interface is illustrated, which also displays the shape of the end instrument of the corresponding operating arm in the projection image. Of course, the projection image reflects the contour shape of the corresponding operating arm.

[0140] In one embodiment, in the surgical robot of the present invention, the operating arm includes a camera arm having an image end instrument and / or a surgical arm having an operating end instrument. Figure 14 As shown, the controller is further configured to perform the following steps:

[0141] Step S21 , detecting whether the operating arm has a camera arm.

[0142] This step S21 can be triggered by the user through the input unit. The detection step can also be implemented, for example, by obtaining the type of the operating arm and then determining whether the operating arm includes a camera arm based on the type of the operating arm. Of course, a camera arm must be present during surgery.

[0143] When it is detected in this step that the operating arm has a camera arm, the process proceeds to step S22 .

[0144] Step S22: acquiring the camera parameters of the image end device of the camera arm, and calculating the visible area of ​​the image end device according to the camera parameters.

[0145] Among them, the camera parameters of the image terminal device include focal length and aperture.

[0146] Step S23: determining the position and posture of the image end device in the reference coordinate system according to the joint variables and kinematic model of the camera arm.

[0147] In step S24 , the visible area of ​​the image terminal device is converted into the visible area of ​​the virtual camera according to the conversion relationship between the position and posture of the image terminal device and the position and posture of the virtual camera in the reference coordinate system.

[0148] Step S25 , calculating a boundary line of the visible area of ​​the virtual camera on the projection plane, and displaying the boundary line in the projection image displayed on the display.

[0149] See Figure 15 , Figure 15 A display interface is illustrated, in which the projection image of the display interface shows the visible area of ​​the image end device, and the part outside the visible area is the non-visible area.

[0150] Through the above steps S21 to S25, it is convenient for the doctor to clearly perceive from the projection image which parts of the operating arm are visible in the real field of view and which parts are invisible in the real field of view.

[0151] In one embodiment, Figure 16 As shown, the controller is further configured to perform the following steps in the above step S15, i.e., the step of sequentially fitting and connecting the projection points to generate the projection image of the operating arm:

[0152] Step S151 ′: obtaining an operation image of the surgical area captured by the image end instrument of the camera arm.

[0153] Step S152 ′: identifying characteristic parts of the surgical arm from the operation image.

[0154] Image recognition can be used. More preferably, image recognition can be performed in combination with a neural network such as a convolutional neural network.

[0155] Step S153 ′: matching the associated first feature point from the feature point sequence according to the identified feature part.

[0156] In this feature point sequence, in addition to the first feature points that can be matched, it also includes the second feature points that are not matched. It should be understood that "first feature points" refers to one type of feature points, which in this context refers to all feature points matched based on the identified feature parts, which may be one or more. "Second feature points" refers to another type of feature points, which in this context refers to all feature points in the feature point sequence except the first feature points, which may also be one or more.

[0157] Step S154 ′: sequentially fitting and connecting the projection points and marking the first projection point associated with the first feature point and the line segment connected to the first projection point to generate a projection image of the operating arm.

[0158] This is particularly suitable for situations where feature points are relatively dense, for example, each feature part is represented by more than two feature points. Through the above steps S151' to S154', that is, by marking the first projection point and the line segment connecting it, the visible and invisible parts of the operating arm under the end device of the image can be better displayed.

[0159] In one embodiment, see Figure 17 The controller may also be configured to perform the following steps after step S153', i.e., the step of matching the associated first feature point from the feature point sequence according to the identified feature part:

[0160] Step S155 ′: obtaining unmatched second feature points.

[0161] In simple terms, the second feature point can be obtained by excluding the first feature point from the feature point sequence.

[0162] Step S156 ′: generating an image model of the corresponding feature part by combining the contour information, joint variables and kinematic model of the feature part corresponding to the second feature point.

[0163] This image model can be a reconstructed computer model or a calculated projection model.

[0164] Step S157 ′: convert the image model into a supplementary image in the image end instrument coordinate system.

[0165] Step S158 ′: splicing the supplementary image to the image of the characteristic part corresponding to the first characteristic point according to the order relationship between the second characteristic point and the first characteristic point in the characteristic point sequence to form a complete sub-image of the operating arm in the operating image.

[0166] Step S159 ′: displaying the operating image with the complete sub-image of the operating arm on the display.

[0167] Figure 18 A display interface is shown, which supplements the operating arm with an incomplete operating image.

[0168] Through the above steps S155 ′ to S159 ′, the doctor can also be assisted in viewing some characteristic parts of the operating arm that cannot be viewed by a real camera.

[0169] In one embodiment, see Figure 19 , the controller can also be configured to perform the following steps:

[0170] Step S31: Acquire the maximum motion range of the operating arm in the first direction.

[0171] Step S32 , calculating the movement amount of the operating arm in the first direction according to the joint variables of the operating arm and the kinematic model.

[0172] Step S33: Generate an icon according to the maximum movement range and movement amount in the first direction.

[0173] The maximum motion range may be pre-stored in the aforementioned storage unit.

[0174] Step S34: displaying the icon on the display.

[0175] This chart shows that you can continue to refer to Figure 9 、 Figure 12 and Figure 13 .

[0176] The first direction may be one or more of a forward and backward feed direction, a left and right movement direction, an up and down movement direction, a rotation direction, a pitch direction, and a yaw direction, and may be specifically configured according to the effective degrees of freedom of the manipulator. For example, the first direction is the forward and backward feed direction.

[0177] The icon can be a progress bar or a pie chart. For example, in the progress bar, the maximum range of motion is represented by a fixed-length bar, and the amount of motion is represented by a variable-length bar within the fixed-length bar. As the amount of motion increases or decreases, the color of the variable-length bar can be darkened or lightened accordingly. Furthermore, the proportion of the amount of motion within the maximum range of motion can be calculated, either alone or in combination, and displayed in the display area of ​​the progress bar, such as in the variable-length bar for the amount of motion.

[0178] Through the above steps S31 to S34, the doctor can be prompted to pay attention to the range of motion in the corresponding direction.

[0179] In one embodiment, the controller may be further configured to detect a first manipulator arm currently being controlled from the manipulator arms, and then identify the first manipulator arm in the projected image. This allows for a differentiated display of controlled and uncontrolled manipulator arms on the display. Whether a manipulator arm is controlled may be determined based on whether a start command for actively controlling the manipulator arm is detected.

[0180] In the above embodiment, different virtual cameras selectable by the input unit have different positions and postures in the reference coordinate system to simulate a real camera such as an image end device observing the manipulator from different positions and / or postures (directions).

[0181] In one embodiment, the position of the virtual camera in the reference coordinate system can be determined based on the reachable workspace of the manipulator in the reference coordinate system (hereinafter referred to as the reachable workspace). This allows the position of the virtual camera to be associated with its reachable workspace for easier determination.

[0182] Furthermore, the position and posture of the virtual camera in the reference coordinate system can be determined based on the union space of the reachable workspaces of the manipulators in the reference coordinate system. When there is only one manipulator, this union space is equal to the reachable workspace of the manipulator. When there are more than two manipulators, this union space is the space corresponding to the union of the reachable workspaces of each manipulator. Among them, the reachable workspace of each manipulator in the reference coordinate system can be determined according to the kinematic model of the manipulator, and stored in the aforementioned storage unit for direct call. Of course, the reachable workspace of each manipulator in the reference coordinate system can also be recalculated one or more times each time the surgical robot is started according to the kinematic model of the manipulator.

[0183] Furthermore, the position of the virtual camera in the reference coordinate system is always outside the union space, and the posture of the virtual camera in the reference coordinate system is always toward the union space.

[0184] The position and posture of the virtual camera determined in this way can always fully observe the motion state of each operating arm, including the motion state of each operating arm and the motion state between the operating arms.

[0185] The virtual camera is configured with selectable virtual focal lengths. In one embodiment, the virtual camera's position is sufficient as long as it is outside the area defined by the shortest virtual focal length that is visible across the entire union space. In one embodiment, the virtual camera's position is sufficient as long as it is within the area defined by the shortest virtual focal length that is visible across the entire union space. In one embodiment, the virtual camera's position is defined by both the longest and shortest configurable focal lengths, and is located within the intersection of a first area defined by the longest focal length and a second area defined by the shortest focal length.

[0186] The virtual camera's posture (orientation) is always oriented toward a relatively specific point or region in the union space. In one embodiment, the virtual camera's posture is always oriented toward the center of the union space. This ensures that the virtual camera's virtual imaging plane can always capture a virtual image of each manipulator.

[0187] In one embodiment, the controller may be configured to display the projected image in a first display window of the display and generate icons of selectable virtual cameras in the first display window.

[0188] The relative position of the icon corresponding to the virtual camera and the projected image can be fixed and changes synchronously with the change of the projected image viewpoint. The change of the projected image viewpoint (ie, coordinates) is related to the position of the selected virtual camera.

[0189] Exemplarily, the number of icons corresponding to virtual cameras can be set to six, representing virtual cameras in six different positions. The six icons correspond to virtual imaging of the operating arm from the left, right, top, bottom, front and back sides to generate projection images under the corresponding viewpoints.

[0190] For example, the icon is displayed as an arrow pattern or a camera pattern, and any one of the icons selected by rotation corresponds to a virtual camera. The icon can also be a point, a circle, etc. Figure 22 As shown, the icon is displayed as an arrow pattern, and the arrow indicates the adjustment direction of the field of view angle.

[0191] Exemplarily, the icon is presented as a rotatable sphere, and any position to which the sphere is rotated corresponds to a virtual camera. For example, any position on the surface of the sphere can be matched to certain positions of the aforementioned first area, second area, and / or the intersection area of ​​the first area and the second area, so any position to which the sphere is rotated can represent a virtual camera. Of course, the postures of these virtual cameras are all facing a certain point in the reachable space to ensure that each complete manipulator can be seen. Figure 23 As shown, the icon is presented as a sphere, and the arrows indicate the adjustable directions of the field of view angle.

[0192] In one embodiment, see Figure 22 The controller is configured to execute, in the above step S13, i.e., the step of obtaining the virtual camera selected by the input unit, the following steps:

[0193] Step S131 , obtaining at least two target positions of the virtual camera selected by the input unit and the virtual camera input by the input unit.

[0194] In this step, the selected virtual camera mainly refers to the virtual focal length and / or virtual aperture of the selected virtual camera; the at least two target positions of the input virtual camera can be more than two discrete positions or more than two continuous positions.

[0195] When entering the target position, you can also set the tracking mode, such as single tracking projection mode, multiple tracking projection mode, and reciprocating tracking projection mode. A series of target positions includes a starting position A and an ending position B. For the single tracking projection mode, each target position from A to B is projected only once; for the multiple tracking projection mode, each target position from A to B is projected a specified number of times; for the reciprocating tracking projection mode, each target position from A to B to A to B, etc. is projected repeatedly. For the single tracking projection mode and the multiple tracking projection mode, after the projection process from A to B is completed, the virtual camera can stay at a specified position and continue projecting. The specified position can be any position from A to B, such as A or B, or another default position.

[0196] Step S132 , determining the target projection point of the projection plane of each feature point in the feature point sequence at each target position of the virtual camera according to the preset movement speed of the virtual camera and based on the kinematic model and the joint variables.

[0197] Step S133 : sequentially fitting and connecting the target projection points at each target position to generate a target projection image of the operating arm.

[0198] Step S134: generating an animation according to each target projection image.

[0199] Step S135: Play the animation on the display according to the preset frequency.

[0200] Through the above steps S131 to S135, the doctor can dynamically observe the relative position relationship of the operating arms and the projection information, solve the problem of partial information overlap or projection distortion under a single perspective, and understand the spatial position information from multiple perspectives.

[0201] In one embodiment, the controller is configured to, in the above step S13, i.e., the step of obtaining the virtual camera selected by the input unit, execute: Figure 23 :

[0202] Step S1311 ′: obtaining the motion trajectory of the virtual camera input by the input unit.

[0203] For example, the motion trajectory may be the trajectory of the cursor movement, or for another example, the motion trajectory may be the sliding trajectory of a finger. For ease of implementation, illustratively, the starting position of the motion trajectory is the position of the virtual camera corresponding to the aforementioned icon, and the starting position has coordinates (x0, y0, z0). In the motion trajectory, the coordinates of other positions maintain the Z-axis coordinate unchanged, while only the X-axis coordinate and the Y-axis coordinate are changed. In other embodiments, the starting position of the motion trajectory is not necessarily the position of the virtual camera corresponding to the aforementioned icon, but it is usually necessary to first specify the Z-axis coordinate of the entire trajectory, and then only change the X-axis coordinate and the Y-axis coordinate. For example Figure 24 As shown, it illustrates a configuration interface of a virtual camera motion trajectory.

[0204] In step S1312 ′, the discrete motion trajectory is used to obtain the discrete positions of the virtual camera as the target position.

[0205] Step S132 , determining the target projection point of the projection plane of each feature point in the feature point sequence at each target position of the virtual camera according to the preset movement speed of the virtual camera and based on the kinematic model and the joint variables.

[0206] Step S133 : sequentially fitting and connecting the target projection points at each target position to generate a target projection image of the operating arm.

[0207] Step S134: generating an animation according to each target projection image.

[0208] Step S135: Play the animation on the display according to the preset frequency.

[0209] In one embodiment, Figure 25 As shown, the controller is generally further configured to perform the following steps:

[0210] Step S41: Acquire an operation image of the surgical area captured by an image terminal instrument.

[0211] Step S42: displaying the operation image on the display.

[0212] Step S43: displaying the projection image suspended in the operation image.

[0213] This means the projected image's position within the operating image can be easily changed. For example, a floating window can be created on the display, displaying the projected image while the remaining area of ​​the display shows the operating image. This allows the projected image to be positioned away from key locations within the operating image, facilitating surgical intervention.

[0214] In one embodiment, Figure 26As shown, the controller may also be configured to, in step S43, i.e., in the step of displaying the projected image suspended in the operation image, execute:

[0215] Step S431 : Acquire an overlapping area between the operation image and the projection image, and obtain a first image attribute of the portion of the operation image in the overlapping area.

[0216] Step S432: adjusting the second image attribute of the portion of the projected image in the overlapping area according to the first image attribute.

[0217] These image attributes include one or more of color, saturation, hue, brightness, and contrast, for example, one or more of color, brightness, and contrast.

[0218] Through the above steps S431 and S432, the image attributes of the projected image can be adaptively adjusted according to the image attributes of the operating image. For example, if the operating image is dark, the projected image can be brightened or its color can be changed to make it more prominent relative to the operating image for easier observation by the doctor.

[0219] In one embodiment, Figure 27 As shown, the controller may also be configured to execute, in step S16, that is, before the step of displaying the projection image on the display:

[0220] Step S161: Detect whether the projected image is distorted.

[0221] When it is detected that the projected image is distorted, the process proceeds to step S162 ; and when it is detected that the projected image is not distorted, the process proceeds to step S16 .

[0222] Exemplarily, whether the projected image is distorted can be judged as follows: Step 1, obtain the position of each projection point in the reference coordinate system; Step 2, obtain the number of first projection points among the projection points that fall within the edge area; Step 3, calculate the ratio of the number of first projection points to the total number of projection points, and when the ratio reaches a threshold, determine that the projected image is distorted.

[0223] The edge area can be obtained, for example, by dividing the area based on a display window or a projection plane that displays the projection image.

[0224] Step S162: Increase the virtual focal length of the virtual camera.

[0225] That is, the field of view angle is reduced based on the almost inversely proportional relationship between focal length and field of view angle.

[0226] Step S14 ′ is a step of determining the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera by combining the virtual focal length and / or virtual aperture, the kinematic model and the joint variables.

[0227] Please combine Figures 28 to 28 , Figure 28 A schematic diagram illustrating a method of observing a manipulator with a large field of view is shown; Figure 29 Indicated in Figure 28 The display interface with the first projected image generated under the field of view shown in FIG. 1 shows that the edge area of ​​the projected image is compressed, i.e., distorted. Figure 30 The diagram shows a display interface with a second projected image that is regenerated after the field of view angle is adjusted. It can be seen that the edge area of ​​the projected image is expanded, that is, the distortion problem is eliminated.

[0228] The above step S162 can be illustratively increasing the virtual focal length of the virtual camera by a proportional coefficient. In simple terms, it can be re-determined according to formula (1): F = k*f, where k is the adjustment coefficient, k>1; f is the focal length before adjustment; and F is the focal length after adjustment.

[0229] In one embodiment, the virtual focal length may also be re-determined according to the following formula (2). For example, the re-determination of the virtual focal length in the horizontal direction is used as an example. The formula (2) is:

[0230] fx=k1*Fx*fx0;

[0231] Where fx0 is the focal length at the center of the projection plane; Fx is the distance from a projection point on the projection image along the X-axis; k1 is the setting coefficient; and fx is the x-direction focal length at a projection point. To increase the virtual focal length, simply satisfy k1*Fx>1.

[0232] Formula (2) associates the virtual focal length of the virtual camera with the position of the projection point. That is, the virtual focal length is related to the position of the projection point, and the virtual focal length to be adjusted changes with the change of the projection point. Where x represents any point in the projection plane, and the position of the projection point P in the projection plane is expressed as P(Fx, Fy).

[0233] According to the same principle as formula (2), the virtual focal length in the vertical direction, the horizontal offset relative to the optical axis cx, and the vertical offset relative to the optical axis cx can also be determined, which can be achieved by the following similar methods:

[0234] fy=k2*Fy*fy0;

[0235] cx=k3*Fx*cx0;

[0236] cy=k4*Fy*cy0;

[0237] Through the above steps S161 to S162, by expanding the projected image, the problem that the feature points at the edge of the field of view have projection compression under a large field of view angle, thereby losing the effectiveness of observation information, can be solved.

[0238] In one embodiment, the controller may be further configured to execute:

[0239] An operation instruction for displaying or hiding the image of the corresponding operating arm is obtained, and then the image of the corresponding operating arm is displayed or hidden according to the operation instruction.

[0240] Specifically, when an operation instruction for the display image of the operating arm is obtained, the projection point corresponding to the operating arm is determined in step S14. When an operation instruction for the hidden image of the operating arm is obtained, it is not necessary to determine the projection point corresponding to the operating arm in step S14. This is equivalent to customizing the configuration of the projected image to achieve the purpose of simplifying the projected image and removing interfering sub-images. In one embodiment, a similar purpose can also be at least partially achieved by adjusting the virtual aperture (virtual depth of field) of the virtual camera. For example, by adjusting the virtual aperture, the operating arm away from the virtual camera can be blurred and only the operating arm adjacent to the virtual camera can be clearly virtually imaged.

[0241] In one embodiment, the above-mentioned graphical display method may further include:

[0242] When a first manipulator arm among the manipulator arms reaches a threshold of an event, at least a portion of the first manipulator arm is identified in a projected image and displayed on a display.

[0243] The first operating arm also refers to a class and is not limited to a specific operating arm. The threshold is a warning threshold, and the event is a situation to be avoided.

[0244] In a specific embodiment, the warning threshold is based on the distance between the first operating arm and the second operating arm in the operating arm. For example, the warning threshold can be a numerical value. The situation to be avoided is a collision between the first operating arm and the second operating arm. For example, the situation to be avoided can be a numerical value. The second operating arm also refers to a category and is not limited to a specific operating arm. For example, Figure 31 As shown, the method can be implemented by the following steps:

[0245] Step S51: Acquire the minimum distance between the first operating arm and the second operating arm.

[0246] This step S51 is performed in real time.

[0247] Step S52: Determine the relationship between the minimum distance, the warning threshold, and the situation to be avoided.

[0248] The warning threshold and the situation to be avoided are both represented by numerical values, and in the case where the situation to be avoided is a collision between the first operating arm and the second operating arm, the numerical value d represented by the warning threshold is lim Greater than the value d representing the situation to be avoided min , that is, d lim >d min , the minimum distance between the first operating arm and the second operating arm is represented by d. In one embodiment, d min =0, which means collision has occurred.

[0249] In step S52, if d>d lim , that is, the minimum distance does not reach the warning threshold, then proceed to step S51; if d min <d≤d lim , that is, the minimum distance reaches the warning threshold but does not reach the situation to be avoided, go to step S53; if d=d min , that is, the minimum distance exceeds the warning threshold and reaches the situation to be avoided, and enters step S54.

[0250] Step S53 : performing a first identification on the minimum distance point on the projection image of the first operating arm and the second operating arm.

[0251] like Figure 32 As shown, the operating arm includes a camera arm 31a and surgical arms 31b and 31c, and the minimum distance between the surgical arms 31b and 31c reaches the warning threshold. At this time, in step S53, the minimum distance points P1 and P2 in the projection images of the surgical arms 31b (i.e., the first operating arm) and the surgical arms 31c (i.e., the second operating arm) can be marked with colors or graphic boxes such as circles, as shown in FIG. Figure 33 As shown. If the minimum distance is detected again and does not reach the warning threshold, the marking of the minimum distance point on the projection image of the first operating arm and the second operating arm is usually removed. If the minimum distance is detected again and reaches the situation to be avoided, the process proceeds to step S54, i.e., performing a second marking.

[0252] In addition, during the first identification process, that is, when d min <d≤d lim When the condition is met, the first mark can be changed as the minimum distance gradually decreases or increases. For example, the color is gradually changed, but it can be different from d=d min For example, the first mark is strobed, but it can be different from d=d min When the flash.

[0253] Step S54 : performing a second marking on the minimum distance point on the projection image of the first operating arm and the second operating arm.

[0254] The first mark is different from the second mark. In step S54, for example, the marks of the minimum distance points P1 and P2 in the models of the first operating arm and the second operating arm can be strengthened, such as by deepening the color; or, the marks of the minimum distance points in the projection images of the first operating arm and the second operating arm can be flashed; or, the marks of the minimum distance points in the projection images of the first operating arm and the second operating arm can be changed in type, such as by changing the type of the graphic box, such as Figure 34 As shown, Figure 34 Replace the dotted circle with Figure 33 When it is detected again that the minimum distance has reached the warning threshold but has not reached the situation to be avoided, the process proceeds to step S53, i.e., performing the first identification.

[0255] Steps S51 to S54 help doctors understand the collision position between the operating arms.

[0256] More specifically, if Figure 35 As shown, the above step S51 can be implemented by the following steps:

[0257] Step S511 : constructing corresponding geometric models of the first operating arm and the second operating arm according to their respective kinematic models and structural features.

[0258] In step S511, a slightly larger basic geometric body can be used instead of the actual model for interference analysis to improve subsequent detection efficiency. The geometric models of the first manipulator and the second manipulator can be simplified to, for example, a sphere, a cylinder, a cuboid, a convex polyhedron, or a combination of two or more.

[0259] Step S512: Discrete the geometric models of the first manipulator and the second manipulator to obtain external information point sets of the first manipulator and the second manipulator in the reference coordinate system.

[0260] In step S512, the geometric models of the first operating arm and the second operating arm are digitized to obtain respective external information point sets.

[0261] Step S513 : determining a minimum distance between the first operating arm and the second operating arm according to respective external information point sets of the first operating arm and the second operating arm.

[0262] In step S513, a distance tracking method may be used to determine the minimum distance between the two. More specifically, a traversal algorithm may be used to determine the minimum distance between the first operating arm and the second operating arm from their respective external information point sets.

[0263] More specifically, if Figure 36 As shown, the above step S53 can be implemented by the following steps:

[0264] Step S531 : determining a minimum distance point on the projection image of the first operating arm and the second operating arm corresponding to the minimum distance between the first operating arm and the second operating arm.

[0265] Step S532: first mark the minimum distance point on the projection image of the first operating arm and the second operating arm.

[0266] In one embodiment, Figure 37 As shown, when the minimum distance reaches the warning threshold, the graphical display method may further include the following steps:

[0267] Step S533 : determining the collision direction according to the position of the minimum distance point on the projection images of the first operating arm and the second operating arm in the reference coordinate system.

[0268] Step S534: Mark the collision direction between the first operating arm and the second operating arm in the projection image.

[0269] By marking the minimum distance point between the first operating arm and the second operating arm and the collision direction in the projection image, for example, the collision direction can be marked with an arrow vector direction, visual feedback can be provided to the doctor to avoid collision.

[0270] The handle of the main operating console adopts a mechanical handle. In one embodiment, Figure 38 As shown, corresponding to the situation of the above step S53, that is, when the minimum distance reaches the warning threshold but does not reach the situation to be avoided, the following steps are included:

[0271] Step S533 : determining the collision direction according to the position of the minimum distance point on the projection images of the first operating arm and the second operating arm in the reference coordinate system.

[0272] Step S535 : generating resistance to hinder the movement of the mechanical handle in the associated direction according to the collision direction.

[0273] This can provide force feedback to the doctor to avoid collision when there is a tendency for the operating arms to collide.

[0274] Specifically, the robotic handle comprises a plurality of joint assemblies, sensors coupled to a controller for sensing the status of each joint assembly, and drive motors coupled to the controller for driving the joint assemblies. Generating resistance to hinder movement of the robotic handle in the associated direction based on the direction of the collision more specifically involves causing the drive motor in the associated direction to generate a reverse torque based on the resistance.

[0275] When the minimum distance is between the warning threshold and the situation to be avoided, for example, the reverse torque can be of a constant magnitude; for another example, the magnitude of the reverse torque is negatively correlated with the magnitude of the minimum distance. In the case where the magnitude of the reverse torque is negatively correlated with the magnitude of the minimum distance, specifically, when the minimum distance gradually decreases, the reverse torque is increased to generate a greater resistance; and when the minimum distance gradually increases, the reverse torque is reduced to generate a smaller resistance. For example, the change of the reverse torque is linear; for example, the change of the reverse torque is nonlinear such as a step-like. When the minimum distance reaches the situation to be avoided, the reverse torque generated can be at least as small as to completely hinder the movement of the mechanical handle in the collision direction. In one embodiment, the force or torque applied by the doctor can be detected by the force sensors provided in each joint assembly of the mechanical arm handle, and then a reverse torque that can at least offset the force applied by the doctor can be generated according to the force or torque applied by the doctor. In one embodiment, a force large enough to make a doctor with normal strength insufficient to move the mechanical handle in the collision direction can also be generated suddenly and directly.

[0276] In one embodiment, the warning threshold can also be based on the range of motion of at least one joint component in the first manipulator arm, and the situation to be avoided is the limitation of the range of motion of at least one joint component in the first manipulator arm. Similarly, when the first manipulator arm reaches the warning threshold, at least the relevant joint component of the model of the first manipulator arm can be identified in the first display window or the second display window. In addition, a resistance can be generated at the mechanical handle to prevent the first manipulator arm from crossing the warning threshold and moving toward the situation to be avoided. This resistance is also achieved by generating a reverse torque by the associated drive motor.

[0277] The surgical robot of the above embodiment can also be a multi-port surgical robot. The difference between a multi-port surgical robot and a single-port surgical robot mainly lies in the operating equipment. Figure 39The figure shows a slave operating device of a multi-hole surgical robot. The mechanical arm of the slave operating device in the multi-hole surgical robot has a main arm 110, an adjustment arm 120 and a manipulator 130 connected in sequence. There are more than two adjustment arms 120 and manipulators 130, for example, four. The distal end of the main arm 110 has a directional platform, the proximal ends of the adjustment arms 120 are connected to the directional platform, and the proximal end of the manipulator 130 is connected to the distal end of the adjustment arm 120. The manipulator 130 is used to detachably connect the operating arm 150, and the manipulator 130 has multiple joint components. In the multi-hole surgical robot, different operating arms 150 are inserted into the patient's body through different puncture devices. The operating arm 150 of the multi-hole surgical robot generally has fewer degrees of freedom than the operating arm 31 of the single-hole surgical robot. Usually, the operating arm 150 only has posture freedom (i.e., orientation freedom). Of course, changes in its posture generally also have an impact on the position, but because the impact is small, it can usually be ignored. The position of the operating arm 150 is often achieved with the assistance of the manipulator 130. Since the manipulator 130 and the operating arm 150 are linked to achieve posture changes, the two can be considered as a manipulator component, which is equivalent to the operating arm 31 in the single-port surgical robot.

[0278] In some embodiments, such as Figure 40 As shown, the graphical control device may include: a processor (processor) 501 , a communications interface (Communications Interface) 502 , a memory (memory) 503 , and a communication bus 504 .

[0279] The processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0280] The communication interface 502 is used to communicate with other devices such as various sensors, motors, solenoid valves, or network elements of other clients or servers.

[0281] The processor 501 is configured to execute a program 505 , and specifically may execute the relevant steps in the above method embodiment.

[0282] Specifically, the program 505 may include program codes, which include computer operation instructions.

[0283] The processor 505 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), one or more integrated circuits configured to implement an embodiment of the present invention, or a graphics processing unit (GPU). The one or more processors included in the control device may be processors of the same type, such as one or more CPUs or one or more GPUs; or they may be processors of different types, such as one or more CPUs and one or more GPUs.

[0284] The memory 503 is used to store the program 505. The memory 503 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0285] Program 505 can be specifically used to enable the processor 501 to perform the following operations: obtain the feature point sequence of the operating arm and its corresponding kinematic model; obtain the joint variables sensed by the sensor, and obtain the virtual camera selected by the input unit; determine the projection point of each feature point in the feature point sequence on the projection plane of the virtual camera based on the kinematic model and the joint variables; orderly fit and connect each projection point to generate a projection image of the operating arm; and display the projection image on the display.

[0286] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0287] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A surgical robot, characterized in that: include: Input unit; monitor; An operating arm comprising a plurality of joints and sensors for sensing joint variables of the joints, wherein the operating arm has a feature point sequence consisting of a plurality of feature points arranged in an orderly manner and used to associate with the corresponding joints; and A controller, coupled to the input unit, the display, and the operating arm, is configured to: Obtain at least two target positions of the virtual camera; generating a target projection image of the operating arm in association with each target position; generating an animation according to each of the target projection images; Play the animation on the display at a preset frequency; Wherein, obtaining at least two target positions of the virtual camera includes: Obtaining at least two target positions of the virtual camera selected by the input unit and the virtual camera input by the input unit; and / or Acquiring a motion trajectory of the virtual camera input by the input unit, and discretizing the motion trajectory to obtain discrete positions of the virtual camera as target positions of the virtual camera; Generating a target projection image of the operating arm includes: Determining a target projection point of each feature point in the feature point sequence on the projection plane of the virtual camera by combining the obtained virtual focal length and / or virtual aperture of the virtual camera, the kinematic model of the manipulator, and the joint variables of the manipulator; Orderly fitting and connecting the target projection points to generate a target projection image of the operating arm; or Generating a target projection image of the operating arm includes: Determining a target projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to a preset movement speed of the virtual camera and based on a kinematic model of the manipulator and joint variables of the manipulator; The target projection points are sequentially fitted and connected to generate a target projection image of the operating arm.

2. The surgical robot according to claim 1, characterized in that: Corresponding to different target positions, the virtual camera has different projection planes.

3. The surgical robot according to claim 1, wherein: The orderly fitting and connecting of the target projection points to generate the target projection image of the operating arm includes: Obtaining the type of the operating arm, and matching an icon of the end instrument of the operating arm according to the type; Determining the position and posture of the end instrument on the projection plane of the virtual camera according to the joint variables and the kinematic model; Rotating and / or scaling the icon according to the posture of the end instrument on the projection plane of the virtual camera; The processed icons are spliced ​​at the projection point at the far end to generate the projection image.

4. The surgical robot according to claim 1, wherein: The orderly fitting and connecting of the target projection points to generate the target projection image of the operating arm includes: Matching an icon of the end instrument of the operating arm according to the sequence of feature points; Determining the position and posture of the end instrument on the projection plane of the virtual camera according to the joint variables and the kinematic model; Rotating and / or scaling the icon according to the posture of the end instrument on the projection plane of the virtual camera; The processed icons are spliced ​​at the projection point at the far end to generate the projection image.

5. The surgical robot according to claim 3, characterized in that: The surgical robot further includes a storage unit, in which each of the icons is stored in advance in association with the type and / or the feature point sequence.

6. The surgical robot according to claim 1, characterized in that: The controller is further configured to: A set tracking mode is acquired, and when generating the target projection image of the operating arm, the target projection image of the operating arm is generated in combination with the tracking mode.

7. The surgical robot according to claim 6, characterized in that: The step of generating a target projection image of the operating arm in combination with the tracking mode includes: When the tracking mode includes a single tracking projection mode, the operating arm is projected once at each target position to generate a target projection image of the operating arm corresponding to each target position; or When the tracking mode includes a multiple tracking projection mode, performing a specified number of cyclic projections on the operating arm at each target position to generate a target projection image of the operating arm corresponding to each target position; or When the tracking mode includes a reciprocating tracking projection mode, the operating arm is repeatedly projected at each target position to generate a target projection image of the operating arm corresponding to each target position.

8. A computer-readable storage medium, characterized in that Applicable to a surgical robot, the surgical robot comprising an input unit, a display, an operating arm, and a controller, the operating arm comprising a plurality of joints and sensors for sensing joint variables of the joints, the operating arm having a feature point sequence consisting of a plurality of feature points arranged in an orderly manner for associating corresponding joints, the controller being coupled to the input unit, the display, and the operating arm, the computer-readable storage medium storing a computer program, the computer program being configured to be loaded and executed by a processor to implement the following steps: Obtain at least two target positions of the virtual camera; generating a target projection image of the operating arm in association with each target position; generating an animation according to each of the target projection images; Play the animation on the display at a preset frequency; Wherein, obtaining at least two target positions of the virtual camera includes: Obtaining at least two target positions of the virtual camera selected by the input unit and the virtual camera input by the input unit; and / or Acquiring a motion trajectory of the virtual camera input by the input unit, and discretizing the motion trajectory to obtain discrete positions of the virtual camera as target positions of the virtual camera; Generating a target projection image of the operating arm includes: Determining a target projection point of each feature point in the feature point sequence on the projection plane of the virtual camera by combining the obtained virtual focal length and / or virtual aperture of the virtual camera, the kinematic model of the manipulator, and the joint variables of the manipulator; Orderly fitting and connecting the target projection points to generate a target projection image of the operating arm; or Generating a target projection image of the operating arm includes: Determining a target projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to a preset movement speed of the virtual camera and based on a kinematic model of the manipulator and joint variables of the manipulator; The target projection points are sequentially fitted and connected to generate a target projection image of the operating arm.

9. A graphical control device for a surgical robot, characterized in that: Applicable to a surgical robot, the surgical robot comprising an input unit, a display, an operating arm, and a controller, the operating arm comprising a plurality of joints and sensors for sensing joint variables of the joints, the operating arm having a feature point sequence consisting of a plurality of feature points arranged in an orderly manner for associating the corresponding joints, the controller coupled to the input unit, the display, and the operating arm, comprising: memory for storing computer programs; and a processor for loading and executing the computer program; The computer program is configured to be loaded and executed by the processor to implement the following steps: Obtain at least two target positions of the virtual camera; generating a target projection image of the operating arm in association with each target position; generating an animation according to each of the target projection images; Play the animation on the display at a preset frequency; Wherein, obtaining at least two target positions of the virtual camera includes: Obtaining at least two target positions of the virtual camera selected by the input unit and the virtual camera input by the input unit; and / or Acquiring a motion trajectory of the virtual camera input by the input unit, and discretizing the motion trajectory to obtain discrete positions of the virtual camera as target positions of the virtual camera; Generating a target projection image of the operating arm includes: Determining a target projection point of each feature point in the feature point sequence on the projection plane of the virtual camera by combining the obtained virtual focal length and / or virtual aperture of the virtual camera, the kinematic model of the manipulator, and the joint variables of the manipulator; Orderly fitting and connecting the target projection points to generate a target projection image of the operating arm; or Generating a target projection image of the operating arm includes: Determining a target projection point of each feature point in the feature point sequence on the projection plane of the virtual camera according to a preset movement speed of the virtual camera and based on a kinematic model of the manipulator and joint variables of the manipulator; The target projection points are sequentially fitted and connected to generate a target projection image of the operating arm.

Images