Method for controlling a robotic system for medical or surgical remote operation and related robotic system

Abstract

In any robotic system for the remote operation of medical or surgical procedures, having at least one master device, at least one slave device moved by an actuator and configured to be controlled by the master device, a central microprocessor unit configured to control the slave device, and a display configured to visualize the surgical site where the slave device operates at its central portion, a control method usable is provided. The central unit determines the spatial orientation between the master device and the slave device, and when the central unit receives an alignment assistance request signal, generates on the display a graphical user interface including at least one alignment assistance graphic element visually representing the relative spatial orientation of the master device with respect to the slave device, and modifies at least one graphical aspect of the alignment assistance graphic element in a manner corresponding to the relative spatial orientation of the master device with respect to the slave device.

Description

Technical Field

[0001] The present invention relates to a method for controlling a robot system for medical or surgical teleoperation.

[0002] Specifically, the present invention relates to a control method for initiating medical or surgical teleoperation.

[0003] The present invention also relates to a related robot system for medical or surgical teleoperation.

Background Art

[0004] In master-slave teleoperation where both the master device and the slave device have rotational degrees of freedom, the operator generally gives commands for activating the teleoperation, such as pressing a foot pedal and / or pressing buttons available on the master device body. As is known, during master-slave teleoperation, the master device position is mapped as a target position in the slave working space.

[0005] When preparing for teleoperation, there is a common problem for various master-slave solutions. It is the alignment of each orientation. Because during the medical or surgical teleoperation phase, it is highly desirable that the master and the slave are aligned, that is, perceived as being aligned, for example, by the operator performing a surgical gesture.

[0006] This problem is particularly felt when a non-actuated or "flying" master device is used for one-way teleoperation, that is, without force feedback, for example, when the master device is not mechanically linked to the master operation console.

[0007] In fact, ideally, remote operation should be initiated only when the master and slave are perfectly aligned in orientation or are misaligned by a very small angle, in order to maintain and ensure consistency between the direction and orientation of the user's movement controlling the master device and the direction and orientation of the movement of the slave device.

[0008] Conversely, remote operation by a master and slave that are misaligned beyond a certain misalignment threshold (e.g., a solid angle of 15 degrees) is generally considered undesirable because this results in counter-intuitive movement of the slave device and the potential for resulting hazards.

[0009] For these reasons, generally, an alignment stage is provided that serves as a sufficient preparation for the remote operation stage, which has the purpose of minimizing the misalignment between the master device and the slave device before allowing entry into the remote operation.

[0010] During the alignment step as preparation for remote operation, further requirements for usability arise, such that it is desirable to be able to enter the remote operation anyway, even if there is remaining master-slave misalignment.

[0011] The solution proposed in the document PCT / IB2022 / 051226 in the name of the same applicant provides, in the alignment step as preparation for remote operation, an alignment sub-step involving movement, and the slave device is enabled to move itself to align with the master device when the misalignment is lower than a certain determinable threshold and when one or more further checks are satisfied. In this case, during the alignment step, it is also felt necessary to avoid or at least minimize the movement of the slave device for alignment with the master device. This document also discloses an alignment step in which the slave device follows the master device only in terms of orientation and avoids translation.

[0012] Solutions have also been proposed in which remote operation is made possible even in the presence of a master-slave misalignment, which is gradually corrected during remote operation.

[0013] From a constructive point of view, a master console with mechanically constrained and motorized appendages acting as a "master controller" device is known in master-slave robotic systems for medical or surgical remote operation. In such robotic systems, generally, the motors of the "master controller" appendage impose alignment conditions on the master device with respect to the current orientation of the slave device in the slave workspace, restricting movement in the orientation of the master device. In other words, these systems prevent misalignment by impeding the master device linked to the console.

[0014] A master device for medical or surgical remote operation, mechanically linked to the console by a gimbal (universal support and stabilization system equipped with a gyroscope), is also known.

[0015] As shown, for example, in WO-2016-030767, there has also been proposed a master device for medical or surgical teleoperation that is mechanically directly linked to the one or more slave robot arms for moving the one or more slave robot arms.

[0016] There have recently emerged solutions for master devices for medical or surgical teleoperation that are not mechanically linked to the console of the robotic system, i.e., uncoupled or "ungrounded" or "groundless" master devices, i.e., of the types shown, for example, in documents WO-2019-220407, WO-2019-220408, WO-2019-220409, WO-2021-161158, WO-2021-161185 and WO-2021-161177 in the name of the same applicant, and of the types shown, for example, in documents US-8521331, US-2020-0360097 and WO-2016-137527.

[0017] Another category of master devices are those of the non-operating or "flying" type, i.e., without a feedback system from the slave device, which can physically limit its ease of manipulation. This category can include both the above-mentioned non-constrained type of master device and the type of master device constrained to the operating console, for example, provided with a gimbal (a "gimbal") for support and stabilization.

[0018] Known robotic systems generally comprise at least one master device adapted to be moved by an operator and graspable, at least one slave device moved by an actuator and configured to be controlled by the master device, a central microprocessor unit configured to control the slave device, and a display.

[0019] In known robotic systems, mechanically unconstrained master devices are tracked within a magnetic field and / or by an optical tracking system.

[0020] A robotic system display may be configured to show the surgical site where the slave device is operating.

[0021] When alignment conditions are met, typically following pressing a control pedal, the robotic system can be enabled to enter master-slave teleoperation.

[0022] Before starting master-slave teleoperation, it is important to obtain alignment between the master device and the slave device and have correctly and consistently associated master-slave references such that in all such robotic systems, both those with non-operating or flying master devices and those with non-linked master devices, and in some cases those with master devices linked to an operating console, the surgeon's surgical gestures are implemented in a manner that is perceived as corresponding by the slave surgical instrument.

[0023] For example, the prior art document EP-3305236 shows a robotic system solution for endoscopic surgery, where information regarding the orientation of the articulated end effector portion of the slave surgical instrument is shown on a display together with information regarding the orientation of the rotary joint of an addendum to the master control device.

[0024] For example, the document US-2016-0235489 shows a solution for tracking the orientation of a surgeon's hand to control a slave robotic surgical instrument.

[0025] For example, the initial document WO-2016-137527 discloses an example of master-slave misalignment display in a robotic system with a non-restrained type and freely movable master device (UID) in space.

[0026] For example, the prior document WO-2019-103954 shows a surgical master-slave teleoperation system in which the orientation of a master control device with respect to its reference corresponds to the orientation of a slave surgical instrument with respect to an image acquisition device.

Prior Art Documents

Patent Documents

[0027]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Patent Document 9

Patent Document 10

Patent Document 11

Patent Document 12

Patent Document 13

Patent Document 14

[0028] An object of the present disclosure is to provide a control method that overcomes the limitations of known methods in the master-slave alignment step prior to remote operation.

[0029] A further object of the present disclosure is to propose a solution that can make the alignment step before starting the master-slave remote operation step simpler and more intuitive.

[0030] The proposed solution enables a visual alignment aid element that can guide the surgeon in the alignment step in order to start the remote operation phase by combining the demand for usability and at the same time minimizing the risk.

[0031] In particular, the proposed solution can reduce the time required for the alignment step, and thus reduce the time required for preparing the fully slave remote operation step.

[0032] In a robotic system provided with an alignment step that includes an alignment sub-step in which the proposed solution and, in particular, the slave device is allowed to move to align itself with the orientation of the master device, reducing or resetting the master-slave misalignment in this way minimizes the need to move the slave device during the alignment step and thus the path of the slave during alignment with the master device during the alignment step involving movement, and even eliminates it.

[0033] A further object is that the invention can be used in any robotic system for the remote operation of medical or surgical procedures, having at least one master device, at least one slave device moved by an actuator and configured to be controlled by the master device, a central unit microprocessor configured to control the slave device, and a display configured to view these "helpers".

[0034] The display can also be configured to visualize, for example simultaneously, the surgical site where the slave device operates.

[0035] These and other objects are achieved, at least in part, by the method defined in claim 1.

Means for Solving the Problems

[0036] According to the method of the present disclosure, when the central unit receives a help - alignment request signal, a graphical user interface comprising at least one help - alignment graphic element displayed in a part of the display is generated on the display.

[0037] In one aspect, at least one help - alignment graphic element represents information regarding the relative spatial orientation of the master device with respect to the slave device.

[0038] According to one aspect, at least one help - alignment graphic element represents instructions for enabling the operator to achieve the alignment condition between the master device and the slave device.

[0039] According to one embodiment, the master device is of a non - actuating type.

[0040] According to one embodiment, the master device is of a type that is not mechanically linked to the console, i.e., the master device is of the "flying" type within the master work space.

[0041] According to one embodiment, the master device has N-fold symmetry along an axis, i.e., the master device is equal to itself for rotations equal to a fraction from the circumferential angle N.

[0042] According to one embodiment, the step of generating a graphical user interface with at least one alignment assistance graphic element on the display is activated during the remote operation preparation step.

[0043] According to one embodiment, the method is performed during the alignment step as preparation for remote operation.

[0044] According to one embodiment, the step of generating a graphical user interface with at least one alignment assistance graphic element on the display is activated when the master device is detected inside and / or outside a predefined volume.

[0045] According to one embodiment, the step of generating a graphical user interface with at least one alignment assistance graphic element on the display is activated when the master device receives a predefined hand gesture at a predefined time interval. The hand gesture is preferably detected by detection of the master device (e.g., by optical and / or electromagnetic tracking).

[0046] According to one embodiment, the step of generating a graphical user interface with at least one alignment assistance graphic element on a display is such that the master device is detected within a predefined volume and, simultaneously, the open / close command of the master device is activated, preferably when the open / close commands of two master devices (right and left) are simultaneously activated. For example, the open and close commands may be repeatedly activated ("double tap").

[0047] According to one embodiment, the step of generating a graphical user interface with at least one alignment assistance graphic element on a display is such that the master device is detected within a predefined volume and, contextually, (to indicate that the surgeon's gesture is about to spontaneously start the alignment phase, "top-down") when turned over, preferably when two master devices (right and left) are simultaneously turned over, it is activated.

[0048] According to one embodiment, the orientation of the slave device is calculated from internal measurements of the sensors or actuators of the robotic system.

[0049] According to one embodiment, the orientation of the slave device is calculated starting from an algorithm that analyzes an image of the slave collected by a vision system.

[0050] According to one embodiment, the orientation of the slave device is calculated based on a combination of the internal information of the sensors or actuators of the robotic system and that of the vision system.

[0051] A robotic system configured to implement the described method for a suitable computer program loaded into a central microprocessor unit is also described.

[0052] Further embodiments are defined in the appended claims.

Brief Description of the Drawings

[0053]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 13C

Figure 14A

Figure 14B

Figure 14C

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

DETAILED DESCRIPTION OF THE INVENTION

[0054] In the following description, a control method of a robot system 100 for medical or surgical remote operation will be described with reference to FIGS. 1 to 24. Generally, the robot system 100 in which the control method of the present disclosure can be implemented is of the type schematically shown in FIG. 1 and includes at least - a master device 110 adapted to be handled by an operator 150 and capable of being held, - a slave device 170 that can have at least one surgical instrument 170, is controlled by the master device 110, and is configured to be moved by an actuator to track the movement of the master device. - A central microprocessor unit configured to control a slave device and move the slave device via one or more dedicated actuators, a display 180, including.

[0055] The display 180 may be configured to display the surgical site where the slave device 170 is operating.

[0056] The robotic system 100 may include two slave devices (i.e., two slave surgical instruments) and two master devices for controlling them.

[0057] According to one embodiment, the display 180 is configured to display at least one alignment assistance graphic element 190 and avoid displaying the surgical site.

[0058] According to one embodiment, at least two displays are provided, one display is configured to display the surgical site, and the other display is configured to display at least one alignment assistance graphic element 190. For example, the display that displays at least one alignment assistance graphic element 190 may be placed in proximity to the surgeon and may be worn by the surgeon, for example, as gloves, masks, glasses, bracelets, bands, belts, and / or the like.

[0059] In the example shown in FIG. 1, reference is made to a particular case of interest where the master device held by the surgeon is not mechanically constrained to the console. This type of master device is generally not actuated, i.e., those master devices do not have a feedback system (joints) from the slave devices, which could physically limit the ease of manipulation of those master devices, for example, if a master - slave position deviation exceeding a certain threshold is detected. The method of the present disclosure is presented with respect to this particular case of interest that is common in the field of telemedicine, but it should be understood that what is described applies when the master device is of the non - actuated type, as well as when the master device is mechanically linked to the console, and as well as when the master device is mechanically linked to a slave device such as a slave robotic arm, regardless of whether it is linked to the console or not.

[0060] In particular, in the case of the unconstrained or “flying” master device 110, the operator feels the need to check whether the master device held by the user is aligned with the slave device as seen by the user, which is represented on the display 180 of the system robot. This is because the orientation of the master device 110 with respect to the user's view may not correspond to the orientation of the surgical instrument 170 of the slave device with respect to the view of the vision system (e.g., video camera 160) that records the orientation of the surgical instrument 170 of the slave device. This need is particularly felt during the preparation stage for medical or surgical teleoperation, and thus, when entering the master - slave teleoperation, the position deviation between the master device 110 controlled by the operator and the slave device 170, as shown on the display 180, is minimal or zero.

[0061] An example of a master device and a slave device is shown in FIG. 2A, along with coordinate systems MF, SF that identify the respective orientations of the master device and the slave device in space relative to reference MFO, SFO. In the example of FIG. 2A, the body of the master device 110 includes two rigid parts 134, 135 that are integral with respective sensors or markers (not shown) that are constrained at a rotational joint 136 like a gripper so as to rotate about a common axis while the master device is being operated by an operator 150.

[0062] In the example shown in FIG. 2A, possible alignment steps as preparation for remote operation are schematically shown, and the slave device 170 is enabled to move in a slave work space (identified in the figure by reference SFO) to align itself with the master device 110. (In the example shown, the starting posture of the slave surgical instrument 170 is shown with shading and solid lines, and the target posture of the slave surgical instrument 170 aligned with the posture of the master device 110 is shown with dashed lines without shading.)

[0063] In the example shown in FIG. 2B, possible alignment steps are shown in which the slave device 170 follows the master device 110 only in terms of orientation. In other words, the control point 600 of the slave surgical instrument 170 can perform only a pure rotational movement to align the slave surgical instrument 170 itself with the posture of the master device 110, independent of the positioning kinematic chain of the control point 600.

[0064] To determine the positions of the master device and the slave device and their relative misalignment, a control point 600 for placing the slave device can be conveniently identified. In the example of FIG. 2A, the point 600 can be identified as a control point belonging to the slave surgical instrument and / or as a virtual point strictly associated with the slave surgical instrument. The surgical instrument of the slave device 170 comprises a plurality of rotational joints P, Y, G, for example, a pitch joint P and a yaw joint Y, and may also comprise an opening / closing joint G, and / or the surgical instrument of the slave device can comprise a plurality of stacked vertebrae ("snake"), while the master device 110 preferably does not comprise a corresponding number of rotational joints, and thus, by providing a control point 600 that can identify at least six degrees of freedom (three translations and three rotations, and, if necessary, an opening / closing degree of freedom G) of the slave device, it is possible to determine the correlation between the posture of the master device and the posture of the slave device.

[0065] The relative orientation (misalignment) between the master device 110 and the slave device 170 is generally understood as a relative rotation in the rotation space SO(3). This relative rotation can be mathematically represented in different ways, for example, Euler angles, axis angles, or twist-swing, and these display modes can be associated with different visual solutions for assisting the operator with alignment. The display by Euler angles represents the relative rotation by three rotations with respect to orthogonal axes, and the angle-axis display represents the direction of rotation and the total angle along the geodesic. The twist-swing display is constructed by examining two main directions in the slave and in the master, the swing angle is what carries one main direction over to the other, and the twist angle is the relative rotation angle along the said main direction.

[0066] In the robot system 100, a vision system 160 may be provided to capture a physical actual slave device 170 in real time, and the collected image is displayed on the display 182. In this way, a display (helper graphic element alignment 190) on the display 180 of the orientation of the slave device 170 (represented, for example, in roll-pitch-yaw and / or twist-sway coordinates) can be derived from the collected image. Alternatively or additionally, the display (alignment assistance graphic element 190) on the display 180 can be derived by implementing computer vision and machine learning algorithms on such images. In this case, the vision system 160 preferably comprises a microscope capable of magnifying the images to be collected.

[0067] For example, when the master device comprises two rigid parts pincer-hinged about a common axis, the "alignment" preferably means an alignment between the longitudinal axis (for example, the "x" axis in Figure 2A) extending between the two rigid parts 134, 135 and the control point 600 identifying the slave device 170 by the constrained rotation 136 of the master device 110.

[0068] According to one aspect, the method of the present disclosure provides an operation of generating a graphical user interface on the display 180 by the central unit when the central unit receives an alignment assistance request signal. This graphical interface can be, for example, as shown in Figure 3, and comprises alignment assistance graphic elements 190, which are shown as helpers in Figure 3 and can look like those illustrated in Figures 5 to 14. Through interface graphics such as those shown in Figure 3, the surgeon 150 can display one or more graphic elements that clarify what the orientation of the master device 110 with respect to the corresponding slave device 170, as seen by the surgeon, represented on the display 180.

[0069] According to a preferred embodiment, the slave device 170 is displayed in the substantially central portion 182 of the display 180, while the alignment assistance graphic element 190 is in a portion of the display 180 that does not overlap the central portion 182 of the display, for example, at the corner 183 of the display, or in another peripheral portion 184 that does not obstruct the surgeon's 150 view of the slave device 170 of the robotic system 100. Preferably, the robotic system 100 preferably includes at least two slave devices 170 that are controlled by at least two respective master devices 110.

[0070] The slave device 170 is not necessarily displayed in the substantially central portion 182 of the display 180. However, in one embodiment, the alignment assistance graphic element 190 is displayed in a portion of the display 180 that does not overlap the portion of the display 180 where the slave device 170 is represented.

[0071] According to one embodiment, the slave device 170 is not shown on the display 180, and at least one alignment assistance graphic element 190 is shown on the display 180. For example, a second additional display (not shown) may be provided, the slave device 170 is displayed on the second additional display, and at least one alignment assistance graphic element 190 is displayed on the first display 180.

[0072] According to one aspect of the method of the present disclosure schematically shown in FIG. 4, the first spatial orientation of one or more master devices and the second spatial orientation of one or more corresponding slave devices are determined by a central unit (the "control unit"). Next, a possible misalignment (i.e., relative orientation) Δ between the master and the slave is determined, and the central unit reveals how the master is oriented with respect to the slave and provides visual information to the surgeon that enables the surgeon to achieve the alignment condition between the master device and the slave device. The graphic aspect of the alignment assistance graphic element 190 is modified in a manner corresponding to the relative spatial orientation of the master device with respect to the slave device. For example, the status or posture of the master and / or other inputs generate an alignment assistance request signal (the "trigger"), the master is mapped to slave coordinates, and three additional inputs are processed to create a visual display of a characteristic alignment assistance graph, in particular, the rotation of the mapped master (the "master rotation param"), the rotation of the slave (the "slave rotation param"), and their relative rotation in SO(3).

[0073] FIGS. 15 to 18 show state diagrams showing how remote operation should be started after a graphical user interface according to the method of the present invention is opened on the display 180 of the robot system 100, such as the robot system shown in FIG. 1. When this graphical user interface is shown on the display, various situations as illustrated in the following figures can occur. - FIG. 15: When, after the graphical interface is opened, the relative spatial orientation between the master device and the slave device happens to be within a predefined range, the graphical interface is closed and the remote operation is automatically started. - Figure 16: After the graphical interface is opened, if it happens that the relative spatial orientation between the master device and the slave device is within a predetermined range, the graphical interface is not automatically closed, and the graphical interface notifies the operator that remote operation is ready to be enabled. Thus, the robot system waits for an enabling signal ("trigger") for medical or surgical remote operation to be generated. - Figure 17: After the graphical interface is opened, if the relative spatial orientation between the master device and the slave device is outside the acceptable tolerance range but within a second maximum range ("funnel") that includes the acceptable tolerance range, when an enable signal ("trigger") is generated (e.g., an alignment enable signal and / or a medical or surgical remote operation enable signal), the alignment phase begins with, for example, the operator holding the master device fixed and the robot system automatically and / or autonomously moving the slave device until the relative spatial orientation between the master device and the slave device is within the acceptable tolerance range. At that point, the graphical interface closes and remote operation begins. - Figure 18: When the relative spatial orientation between the master device and the slave device is outside the second maximum range (the "funnel") after the graphical interface is opened, for an enable signal ("trigger") to enter medical or surgical teleoperation, for example, keep the slave device stationary and request the operator via the graphical interface to move the master device until the relative spatial orientation between the master device and the slave device is no longer within the second maximum range, thereby starting the alignment phase. In this situation, the graphical interface notifies the operator to keep the master device stationary while the slave device is automatically and / or autonomously moved until the relative spatial orientation between the master device and the slave device is within the acceptable tolerance range. When this occurs, the GUI closes and teleoperation begins.

[0074] According to one aspect, the graphical user interface closes when the maximum time has elapsed since the alignment assistance request signal was generated and the graphical user interface has been open, and / or is closed anyway, for example without entering teleoperation.

[0075] According to a preferred embodiment, the robot system generates the alignment assistance request signal by the central unit outside of teleoperation and, preferably, in the teleoperation preparation step.

[0076] According to one embodiment, the robot system generates the alignment assistance request signal by the central unit in an interrupted teleoperation phase, for example in the pause from teleoperation envisaged between two consecutive fully enslaved teleoperation phases.

[0077] According to one embodiment, the robot system generates the alignment assistance request signal by the central unit in a restricted remote operation stage in which a slave device remains slave to the master device while a subset of the degrees of freedom of the slave device is blocked by a second subset of degrees of freedom.

[0078] Of course, the aspects described below for generating the alignment assistance request signal can be provided in any combination thereof. In fact, one requirement is that it be possible to activate the alignment procedure between the (one or more) master device(s) and the (one or more) slave device(s) in a simple and safe manner.

[0079] According to one aspect, the requirement for generating the alignment assistance request signal can be the detection of the master device or two master devices within a determinable workspace. The determinable workspace in which the master device 110 must be arranged for generating the alignment assistance request signal can coincide with the entire workspace 115 of the master device (defined, for example, by the limitation of the "tracking" field), or the determinable workspace can be a subspace 116 defined by the central unit. In this way, the determinable workspace 116 in which the master device must be arranged to generate the alignment assistance request signal is a subset of the master workspace 115, as shown, for example, in FIG. 24. For example, the determinable workspace can exclude areas for storing the master device when not in use, such as storage areas. For example, the determinable workspace can have the shape of a sphere, and / or two spheres, such as one sphere for each master device, and / or a box. An external region can be provided, and an internal region can be included in the external region of the workspace, and the determinable workspace in which the master device must be arranged to generate the alignment assistance request signal can be the external region or outer region.

[0080] According to one aspect, the alignment assistance request signal is generated by the central unit when the open / close command of the master device 110 is activated. For example, as shown in FIG. 21, slightly pressing the open / close command (grip, G) of the master device generates the alignment assistance request signal. When two master devices are provided, such as a right master device for the surgeon's right hand and a left master device for the surgeon's left hand, the alignment assistance request signal can be generated by the central unit when the open / close commands of both master devices are activated simultaneously or within a time window from the first close / open. According to one embodiment, the alignment assistance request signal is generated by the central unit when the open / close command of the master device is repeatedly activated, such as a "double tap".

[0081] According to one aspect, the alignment assistance request signal is generated by the central unit when the master device is detected upside down, i.e., "from top to bottom", as shown in FIG. 22 for example. In this case, additional conditions can be provided, such as when the master device 110 is detected upside down and stopped, i.e., not in free fall, for example, the alignment assistance request signal is generated. For example, the alignment assistance request signal can be generated in the inverted master condition and the activated open / close command. Further, an additional check to establish that the surgeon's gesture is attempting to initiate the alignment phase, as schematically shown in FIG. 22, can be, for example, a check regarding the vertical position of the master device to detect whether the surgeon has turned the master device over by lifting the arms simultaneously.

[0082] According to one aspect, the alignment assistance request signal is generated by the central unit when the line-of-sight recognition device 187 installed in the robot system detects that the operator is looking in a predetermined direction, such as towards the screen 180 or an area or a part where the screen 180 is located.

[0083] According to one aspect, the alignment assistance request signal is generated by the central unit when the master device 110 is directed towards the screen 180 or the display 180 according to a predefined orientation.

[0084] According to one aspect, the alignment assistance request signal is generated by the central unit when an alignment is detected between, for example, as shown in FIG. 23, the hand 151 of the surgeon 150 holding the master device 110, at least one eye 152 of the surgeon, and at least a portion of the screen 180. For example, a camera can detect such an alignment. For example, the line-of-sight detection device 187 detects such an orientation.

[0085] According to one aspect, the method includes the operation of generating an alignment assistance request signal by the central unit when the central unit detects that a predetermined movement has been performed by the master device. In practice, the central unit determines the position of the master device(s), and when the central unit itself detects that the master device(s) is / are in a predefined spatial orientation or in a predefined master working space, the central unit activates the graphical user interface.

[0086] In one aspect, the central unit determines the position of the master device(s), and when the central unit itself detects that the master device(s) is / are in a predefined spatial orientation or in a predefined master working space, the central unit activates the graphical user interface. By doing so, the surgeon can activate the graphical user interface without having to keep holding the master device(s) and without taking his eyes off the display showing the surgical site where the slave device(s) operates.

[0087] The alignment assistance request signal that activates the graphical user interface in the central microprocessor unit can be generated in other ways.

[0088] According to one aspect, the robotic system can be equipped with a line-of-sight detection device 187 of the user of the robotic system for the user to generate an alignment assistance request signal (and thus open the graphical user interface) when looking in a predetermined direction.

[0089] According to another aspect, at least one or each master device 110 of the robotic system can be equipped with a specific switch 158, such as a button, for generating an alignment assistance request signal when activated (e.g., pressed).

[0090] According to one aspect, each master device may be equipped with a capacitive or other sensor that generates an alignment assistance request signal for the central microprocessor unit when touched by the surgeon holding the master device, and the central microprocessor unit opens the graphical user interface on the display. By placing a button or capacitive sensor on the master device in a position that can be reached by a simple movement of the finger without changing the grip, the operator can generate an alignment assistance request signal without taking their eyes off the display.

[0091] According to one aspect, if the robotic system includes a foot switch 158 functionally connected to the central unit, when this is pressed, an alignment assistance request signal is generated and then the graphical user interface can be displayed on the display.

[0092] In one aspect, the graphical user interface can be closed when the pedal is released, regardless of whether alignment between the master device and the slave device has been achieved.

[0093] According to one aspect, the alignment assistance request signal can be generated by the central unit when the command of the master device 110 is suitable for controlling the opening / closing degree of freedom G of the slave device 170.

[0094] When the master device 100 includes, for example, as shown in FIG. 2A, two rigid portions 134 and 135 that are hinged to each other like a pair of tweezers around a common axis, and the two rigid portions 134 and 135 are angularly kept away from the spring 137 when the spring 137 is stationary and can approach each other only by overcoming the elastic repulsive force from the spring, when the angle between the two rigid portions is defined and the two rigid portions are brought closer to each other until it becomes smaller than a predetermined angle threshold value taking into account the elastic influence action of the spring, the alignment assistance request signal can be generated by the central unit.

[0095] According to yet another aspect, in the master device 110 of a robotic system having two rigid gripper-hinged portions 134 and 135, the alignment assistance request signal can be generated by continuously closing those rigid gripper-like portions quickly twice ("double tap"). The master device 110 may include an elongated body given an open / close command, and this alignment assistance request signal can be generated by continuously activating the open / close command quickly twice ("double tap").

[0096] In one aspect, the alignment assistance request signal is automatically generated by the central unit. According to one implementation option, the alignment assistance request signal is automatically generated by the central unit when a determinable condition is detected during master-slave teleoperation. Determinable conditions detectable during teleoperation may include that the slave device is close to the physical limitations of the slave work space (joint limitations, e.g., pitch P joint and / or yaw Y joint and / or roll R joint limitations of the articulated end of the surgical instrument 170), and / or that the posture commanded to the slave device is outside the work space of the slave device (e.g., outside the work space of the pitch P joint and / or yaw Y joint and / or roll R joint). Determinable conditions detectable during teleoperation may include, for example, a state transition from a restricted teleoperation state where the slave device follows the master device in only orientation and / or only translation, and / or in a subgroup of degrees of freedom including orientation and / or translation, to or from such a state.

[0097] Various examples of graphical interfaces that can be visualized on a display in the manner diagrammed in FIG. 3 are shown in FIGS. 5-14 with respect to cases where a robotic system includes two master devices intended to be grasped by a surgeon and two corresponding slave devices operating at the surgical site. It should be noted, however, that the description also applies to cases where there is only one master device and one slave device, and thus the graphical user interface includes only one graphic element. In the cases shown with two master devices and two slave devices, the graphical user interface that is displayed will include a first graphic alignment assistance element for the first master-slave pair and a second graphic alignment assistance element for the second master-slave pair. The central microprocessor unit determines the positions of the master and slave devices and modifies the appearance of the two graphic elements to visually represent the current relative orientation of each master device from the corresponding slave device.

[0098] Graphic elements that visually represent misalignment between the master and slave devices or the respective orientations of the master and slave updated in real time can be represented in various ways.

[0099] For example, as shown in FIG. 5, two alignment assistance graphic elements 190 can be arranged at opposite corners of the screen 180, each being shown as a concentric circle having circular sectors of different colors C1, C2, C3 and having at least one indicator F, F1, F2 (such as an arrow, a cursor, etc.). The alignment assistance graphic element 190 of FIG. 5 is shown in more detail in FIG. 6A, where the inner circle represents the twist angle, the circular sector outside the inner circle represents the swing angle, the arrow F represents the orientation of the master device, the color C1 represents a region of misalignment lower than an acceptability threshold (such as 5 to 8 degrees) for enabling remote operation, etc., the color C2 represents a region of misalignment lower than a "funnel" threshold (such as 40 to 60 degrees) for enabling the movement of the slave device to align with the master device, etc., and the color C3 represents a region outside the slave working space. In this example, it should be noted that due to the two-fold rotational symmetry about the roll x-axis of the master device 110, two possible acceptability thresholds (color C1) for the twist angle are shown ("master flip" mode).

[0100] For example, in FIG. 6B, the twist angle and the swing angle are represented on two different and separate geometric shapes, where the circle represents the swing angle and the bar represents the twist angle. In this figure, the color C1 corresponding to the acceptability threshold is placed in the "12 o'clock" configuration, and the arrows F1, F2 represent the master device 110. In this display, the colors C1, C2, C3 represent the same concepts as those referred to with reference to FIG. 6A.

[0101] For example, in FIG. 6C, the twist angle and the swing angle are represented on a concentric ring, similar to that described with reference to FIG. 6A.

[0102] For example, in FIG. 6D, the swing angle is represented on a ring and the twist angle is represented on a bar, similar to that described with reference to FIG. 6B.

[0103] For example, in FIG. 6E, the pitch angle is represented by a ring (angle display), and the twist angle is represented as a radial extension of the same ring. Thus, in this example, the F indicator can move both angularly and radially within the ring, i.e., the ring becomes thicker / thinner according to the twist angle.

[0104] For example, in FIG. 6F, the twist angle and the pitch angle are represented on a Cartesian plane as two orthogonal axes, and the F indicator is represented as a crosshair.

[0105] Of course, one skilled in the art may choose to exchange the graphical representation of the twist angle and the graphical representation of the pitch angle with respect to what has been described with reference to these exemplary FIGS. 6A - 6F.

[0106] According to one aspect, each graphic element can be represented by two cones, as shown in FIGS. 7 and 8, one representing the master device and the other representing the slave device. These cones visually provide information about the misalignment between the master device and the slave device from the perspective of the angular coordinates of the spherical reference system, and the information regarding the misalignment is indicated by the display of the master device and the slave device. More specifically, FIG. 8 shows the details of an alignment assistance graphic element consisting of two three - dimensional cones representing the orientation of the master updated in real - time and the relative orientation of the slave. In one embodiment, these cones have different colors.

[0107] In one aspect, each graphic element can be shown by two long cones, as shown in FIGS. 9 and 10. For example, the coloring of the cones can include displaying information regarding the misalignment along the roll axis or the twist axis, i.e., along the longitudinal dimension of the master device and / or the slave device. More specifically, FIG. 9 shows how the two right and left graphic elements for alignment assistance according to the present disclosure in the shape of three - dimensional cones with two - color pole discrimination of the roll should be visualized on a display respectively.

[0108] According to a preferred embodiment, each graphic element is of the type shown in FIGS. 11 and 12, showing a virtual, i.e., synthetic, slave device, which appears as an actual slave device partially superimposed on another graphic element showing the corresponding master device, as seen by the surgeon in the central part of the display. In an exemplified case where the slave device is substantially constructed by two rigid parts 134, 135 integral with respective sensors or markers constrained in a rotary joint such as a gripper to rotate about a common axis while the master device is being operated by the operator 150, the graphic element showing the master device and the graphic element showing the slave device can be superimposed at the rotary joint, and thus, the misalignment between the rigid scissor hinge-coupled part of the master device held by the operator and the rigid scissor hinge-coupled part of the slave device operating at the surgical site.

[0109] For example, in FIG. 13A, the graphic element can be represented with an arrow F identifying the misalignment between the master device and the slave device, and with at least one cone visually identifying the maximum allowable misalignment between the master device and the slave device. For example, the size of the arrow F can increase with the misalignment. For example, two or more concentric cones colored with the colors C1, C2 can be provided to indicate the acceptance threshold and the "funnel" threshold (e.g., 5 - 8 degrees and 40 - 60 degrees, respectively). Similarly, in FIG. 13B, a frustum of a cone is shown instead of a cone. For example, in FIG. 13C, two arrows respectively showing the master device and the slave device are shown, and the feet of the arrows indicate the twist or roll angle.

[0110] According to one aspect shown in FIGS. 14A - 14C, the graphic element indicates an instruction to reach an alignment condition and can show the operator how the master device 110 should be moved (rotated) to align the master device 110 with the orientation of the slave device. For example, FIG. 14A shows two arrows, FIG. 14B shows arrows superimposed on the display of the hand 151 of the surgeon 150 to indicate a twist or roll rotation instruction, and FIG. 14C shows arrows superimposed on the display of the master device 110 to indicate a twist or roll instruction.

[0111] In one aspect, to increase the intuitiveness for the surgeon, real - time master - slave misalignment in the form of abstract and simplified images is shown. According to one embodiment, the misalignment is shown in real - time as a cursor extending along a line or bar that can exhibit color - coded coloring indicating the misalignment, as shown, for example, in FIGS. 6b or 6c. For example, when the misalignment is lower than a certain tolerance threshold, the cursor is positioned corresponding to the green - colored section of the line or bar, and when the misalignment is greater than this certain tolerance threshold, the cursor is positioned corresponding to the yellow, orange, and / or red - colored sections of the line or bar.

[0112] According to one embodiment, the misalignment is shown in real - time as a hand rotating along a circle, for example, a portion of a ring. Providing a simplified and abstract representation of the relative orientation (misalignment) between the master and the slave makes it possible to avoid representing redundant information and makes the process of aligning the master device with the slave device intuitive.

[0113] According to a preferred embodiment, the alignment assistance graphic element is not an overlay (in technical terms, "ghosting") on the image of the "actual" slave device 170 obtained by the vision system 160. In other words, preferably, the alignment assistance graphic element is avoided from being like a "ghost" overlay. This choice enables obtaining a more effective display, which is independent of the current posture of the slave device and is thus more understandable in any kinematic configuration of the slave device. In fact, generally, "ghosting" is based on augmented reality technology, that is, the system uses "computer vision" technology to identify the slave in the image and superimposes a 3D transparent model ("ghost") showing the instrument posture mapped by the corresponding master posture on these images. This display has the drawback of being affected by the problem of self-occlusion that makes understanding difficult, the problem of being potentially positioned at variable positions on the screen 180 close to the edge, or the problem of mutual occlusion between two close instruments. FIG. 12 shows the alignment assistance graphic element according to the present disclosure, which shows the orientation in a composite image of the slave device partially superimposed on the image of each master device to represent the relative orientation of the slave device with respect to the corresponding master device. The composite image can be a simulated or CAD replica of the slave device or the master device in one embodiment, but is not associated with what is potentially represented by the vision system. One of the drawbacks of ghosting in augmented reality is that the roll axis of the actual surgical instrument, that is, the longitudinal axis of the instrument, is generally common, which is not advantageous for proper illustration, especially when the articulated end portion (pitch, yaw, grip) of the surgical instrument is miniaturized.

[0114] In certain embodiments, the above-described requirements that satisfy the above-mentioned advantages can be met by the characteristics described above provided separately or together, and in particular, - When a remote operation preparation stage including an alignment stage by movement of a slave device that enables the slave to move to reach the orientation of the master device as described above is assumed, the provision of at least one graphic element of alignment assistance makes it possible to minimize the movement of the slave device during the alignment stage by movement (Figure 20). - The alignment assistance graphic element can be generated in response to an accurate command by the user. - The graphic alignment assistance element can be automatically generated according to the state of the robot system. - The graphic element can appear, for example, during the remote operation stage, and the command of the master device requires the slave device to reach the physical limit of the working space of the rotary joint of the articulated surgical instrument of the slave device. - Unnecessary movement of the slave device is avoided or at least minimized. - A simple and robust solution is provided that enables a clear and intuitive display for any posture of the slave device, that is, for the spatial configuration of the joints of the surgical instrument of the slave device. - In particular, by avoiding the creation of the at least one alignment assistance graphic element in the form of a "ghost" on the actual slave collected by the vision system, the kinematic configuration of the joints of the slave device is not preferable, that is, for example, when the resulting control point 600 becomes invisible due to the body of the surgical instrument itself or the body of a further second surgical instrument, clear, intuitive, and thus effective visualization is enabled.

[0115] Various embodiments of the robot system control method of the present disclosure can be implemented by creating a computer program that causes the central microprocessor unit to perform the operations of the control method described above when loaded into and activated in the internal memory of the central microprocessor unit of the robot system.

Explanation of Reference Numerals

[0116] JPEG2025518673000002.jpg43165 JPEG2025518673000003.jpg241164

Claims

1. A method for controlling a robotic system (100) for medical or surgical teleoperation, the robotic system comprising at least one master device (110) adapted to be handled and moved by an operator, at least one slave device (170) configured to be moved by an actuator and controlled by the master device, a central microprocessor unit configured to control the slave devices, and a display (180), The method comprises the following operations: determining, by the central unit, a first spatial orientation of the master device and a second spatial orientation of the slave device; an operation of opening, by the central unit, a graphic user interface on the display when the central unit receives an alignment assistance request signal, the graphic user interface comprising at least one alignment assistance graphic element (190) displayed in a portion of the display visually representing information regarding a real-time updated relative spatial orientation of the master device with respect to the slave device and / or instructions for enabling the operator to achieve an alignment condition between the master device and the slave device; updating the alignment assistance graphic element shown on the display in a manner corresponding to the relative spatial orientation of the master device with respect to the slave device; The following conditions are met: - said relative spatial orientation is within a predetermined interval, - an enabling signal for medical or surgical teleoperation is generated, - the maximum time that the graphical user interface remains open from the time the registration assistance request signal was generated has elapsed; closing, by the central unit, the graphic user interface when at least one of the following conditions is met; including carrying out Control method.

2. 2. The method of claim 1, wherein the master device (110) is of the non-actuated type, preferably of the untethered type.

3. the display is configured to display, at least in a central portion thereof (182), a surgical site in which the slave device operates; The method comprises the following steps: displaying the surgical site and the slave device in the central portion of the display; displaying the alignment assistance graphic element in a peripheral portion (184) of the display that is not superimposed on the central portion (182) of the display showing the slave device; Including, The method of claim 1.

4. displaying the at least one alignment-assistance graphic element on the display (180) and avoiding showing the surgical site; and / or the robotic system includes at least a first display and a second display, the method including the operations of viewing the surgical site on the first display and displaying the at least one alignment assistance graphic element (190) on the second display. The method of claim 1.

5. The method of claim 1 , including generating the alignment assistance request signal by the central unit outside of teleoperation, such as during an alignment step in preparation for a teleoperation step.

6. The following conditions are met: when a close / open command of said master device is activated, preferably when the close / open commands of two master devices are activated simultaneously or within a time window from the first close / open command, when the open / close command of said master device is repeatedly activated, when the master device is flipped over by a hand gesture at a defined time interval, and preferably when two master devices are flipped over simultaneously; when a gaze detection device installed in the robot system detects that the operator is looking in a predetermined direction, for example, towards the display; when said master device receives a predetermined hand gesture (151) at a predefined time interval, 2. The method of claim 1, further comprising generating the alignment assistance request signal by the central unit in one of the following ways:

7. The following conditions are met: when the central unit detects that the master device is in a predetermined workspace, for example a sub-space (116) of the workspace (115) of the master device that can be calculated by the central unit, when the central unit detects that a predetermined movement has been performed by the master device, when the central unit detects that the master device is in a predefined spatial location, when the central unit detects that a hand activation switch of the robotic system (158) is pressed, the hand activation switch being located on the handle of the master device (110) and connected to the central unit; a foot switch (158) of the robot system, said foot switch being connected to said central unit, when said central unit detects that said foot switch (158) has been pressed; 2. The method of claim 1, further comprising generating the alignment assistance request signal by the central unit in one of, or a combination thereof.

8. - visually representing the relative spatial orientation between the master and slave devices; The method of claim 1 , further comprising the act of generating an alignment-assistance graphic element for:

9. visually representing the first spatial orientation of the master device and the second spatial orientation of the slave device; The method of claim 1 , further comprising the act of generating an alignment-assisting graphic element such that:

10. visually presenting information, such as a graphic indicator, for moving the master device to achieve alignment with the second spatial orientation of the slave device; The method of claim 1 , further comprising the act of generating an alignment-assisting graphic element such that:

11. The method of claim 1 , wherein an enabling signal for medical or surgical teleoperation is generated when the relative spatial orientation is within a predetermined interval.

12. a slave device movement enable signal is generated to align with the master device orientation when the relative spatial orientation is within a predetermined range; Preferably, when said relative spatial orientation is within a narrow range included in said predetermined range, an enabling signal for medical or surgical teleoperation is generated. The method of claim 1.

13. The method of claim 1 , wherein the enabling signal for medical or surgical teleoperation is generated by a request from the operator, for example by activating a control interface such as a control pedal.

14. The method of claim 1 , wherein the signal for enabling medical or surgical teleoperation is generated automatically when the relative spatial orientation is within a predetermined range.

15. A computer program installable into an internal memory of a microprocessor unit, the computer program comprising software code which, when executed, causes the microprocessor unit to perform the operations of the method of claim 1.

16. at least, a master device (110) that can be handled by an operator; a slave device (170) comprising at least one surgical instrument configured to be moved by an actuator and controlled by said master device; a central microprocessor unit configured to control said slave devices; a display (180); A robotic system (100) for medical or surgical teleoperation, comprising: A robotic system (100) in which the computer program of claim 15 is installed on the central microprocessor unit.

17. 17. The robotic system of claim 16, further comprising a foot switch operatively connected to the central unit, the foot switch configured to generate an alignment assistance request signal when pressed.

18. 17. The robotic system of claim 16, wherein the master device is of a non-actuated type, preferably wherein the master device is of the non-actuated type and is not mechanically constrained to a console.

19. 17. The robotic system of claim 16, comprising two displays, one display configured to display alignment-assistance graphic elements and another display configured to show the surgical site.

Images