Control method of robot system for remote operation of internal medicine or surgery department and related robot system

Abstract

The invention provides a control method of a robot system capable of being used for any internal medicine or surgical remote operation. The robotic system includes at least one master device; at least one slave device moved by an actuator and controlled by the master device; the central microprocessor unit is used for controlling the slave equipment; and a display which visually displays a surgical site operated by the slave device at a central portion thereof, the control method comprising the following operations: determining spatial orientations of the master device and the slave device through the central microprocessor unit; when the central microprocessor unit receives an alignment assistance request signal, generating a graphical user interface on the display, the graphical user interface comprising at least one alignment assistance graphical element to intuitively represent a relative spatial orientation of the master device relative to the slave device; at least one graphical aspect of the assist alignment graphical element is modified in a manner corresponding to a relative spatial orientation of the master device relative to the slave device.

Description

Technical Field

[0001] The present invention relates to a control method for a robot system used for medical or surgical remote operation, and in particular to a control method for starting medical or surgical remote operation, and also to a related robot system used for medical or surgical remote operation. Background Art

[0002] In master-slave teleoperation, both the master and slave devices have rotational degrees of freedom, and the operator typically provides instructions to enable teleoperation, such as depressing a foot pedal and / or pressing an available button on the master device body. As is known, during master-slave teleoperation, the position of the master device is mapped into the slave device workspace as a target position.

[0003] In preparation for teleoperation, various master-slave solutions share the common problem of alignment of the respective positions, since it is highly desirable that the master and slave devices are aligned during the medical or surgical teleoperation phase, e.g., that the operator performing the surgical gestures perceives the two devices as aligned.

[0004] This problem is particularly evident when using an undriven or "flying" master device in unilateral teleoperation, that is, without force feedback, such as when the master device is not mechanically connected to the host operating station.

[0005] In fact, ideally, remote operation should only be initiated when the master and slave devices are perfectly aligned or slightly misaligned in orientation to ensure consistency between the movement direction and positioning of the user-controlled master device and the movement direction and positioning of the slave device.

[0006] On the contrary, it is generally considered undesirable to perform remote operation when the master device and the slave device are offset beyond a certain offset threshold (such as 15 degrees solid angle) because this will cause the slave device to move in the opposite direction, thus creating potential risks.

[0007] For the reasons stated above, the fully slaved remote operation phase is usually preceded by an alignment phase, the purpose of which is to minimize the offset between the master and slave devices before being able to enter remote operation.

[0008] During the alignment step in preparation for teleoperation, a further availability requirement arises, namely the desire to be able to enter teleoperation under any circumstances, even if there is a legacy master-slave offset.

[0009] The technical solution proposed in the application document PCT / IB2022 / 051226 in the name of the same applicant provides: in the alignment phase in preparation for remote operation, in the alignment sub-phase of movement, when the offset is below a predetermined threshold and if one or more further inspection conditions are met, the slave device can move to align with the master device. In this case, it is also necessary to avoid the movement of the slave device during the alignment phase or at least reduce it to minimize the movement of the slave device in order to align with the master device. The application document also discloses an alignment step in which the slave device only follows the master device in direction, thereby avoiding translation.

[0010] The following technical solution is also proposed: even if there is a master-slave offset, remote operation is still allowed, and this offset will gradually recover during the remote operation.

[0011] From a structural point of view, in master-slave robotic systems for medical or surgical teleoperation, a master console with mechanical constraints and motorized attachments is known as a "master controller" device. In such robotic systems, the motors of the "master controller" attachments usually impose the following conditions on the master device: align with the current position of the slave device in the slave device workspace to limit the positional movement of the master device; in other words, these systems prevent deviation by blocking the master device connected to the console.

[0012] In addition, there is a main device for medical or surgical teleoperation, which is mechanically connected to the console via a universal support and stabilization system equipped with a gyroscope ("gimbal").

[0013] For example, as shown in WO-2016-030767, a master device for medical or surgical teleoperation is also proposed, which is directly mechanically connected to one or more slave robotic arms to move the one or more slave robotic arms.

[0014] Technical solution Recently, a master device for medical or surgical remote operation has emerged, which has no mechanical connection with the console of the robotic system, that is, an unbound or "ungrounded" or "ungrounded" master device, such as the type shown in application documents WO-2019-220407, WO-2019-220408, WO-2019-220409, WO-2021-161158, WO-2021-161185 and WO-2021-161177 in the name of the same applicant, and the type shown in application documents US-8521331, US-2020-0360097 and WO-2016-137527.

[0015] Another type of master is the undriven or "flying" type, i.e., there is no feedback system from the slave that can physically limit its maneuverability. This type of master includes both the mechanically unconstrained type of master described above and the console-constrained type of master, for example, providing a universal joint ("gimbal") for support and stabilization.

[0016] Known robotic systems generally include at least one master device, hand-held and adapted to be moved by an operator; at least one slave device, configured to be moved by actuators and controlled by the master device; a central microprocessor unit, configured to control the slave device; and a display.

[0017] In known robotic systems, a mechanically unconstrained master device is tracked in a magnetic field and / or by an optical tracking system.

[0018] The display of the robotic system may be used to show the surgical site being operated on by the slave device.

[0019] Once alignment conditions are achieved, the robotic system can enter master-slave teleoperation, typically achieved by following the pressure of a control pedal.

[0020] In all these robotic systems, i.e., those with non-driven or flying masters and those with non-connected masters and in some cases masters connected to an operator console, it is critical to obtain the alignment between the master and slave devices before starting master-slave teleoperation in order to obtain a correct and consistent master-slave association reference so that the operator's surgical gestures are executed in a perceptibly corresponding manner by the slave surgical instruments.

[0021] For example, the prior application document EP-3305236 shows a robotic system solution for endoscopic surgery, in which the position information of the components of the articulated end effector of the surgical instrument of the slave device and the position information of the rotary joint of the master device supporting the auxiliary device are shown on the display.

[0022] For example, application document US-2016-0235489 shows a technical solution for tracking the position of a surgeon's hands to control a slave robotic surgical instrument.

[0023] For example, the earlier application document WO-2016-137527 discloses an example of a master-slave offset display in a robot system, which includes a master device (UID) of an unconstrained type that can move freely in space.

[0024] For example, the prior application document WO-2019-103954 shows a master-slave surgical remote operation system, in which the orientation of the master device relative to the reference object corresponds to the orientation of the slave device surgical instrument relative to the image acquisition device. Summary of the invention

[0025] An object of the present disclosure is to provide a control method that overcomes the limitations of known methods in the master-slave alignment step prior to remote operation.

[0026] Another object of the present disclosure is to propose a technical solution that can make the alignment step simpler and more intuitive before starting the master-slave remote operation step.

[0027] By means of the proposed technical solution, a visually assisted alignment element can be provided, capable of guiding the surgeon during the alignment phase in a way that takes into account the usability needs and at the same time minimizes the risks to allow the start of the teleoperation phase.

[0028] In particular, by means of the proposed technical solution, the time required for the alignment step can be shortened, thereby reducing the time required for preparing a fully slave remote operation step.

[0029] Through the proposed technical solution, especially in those robot systems that provide an alignment stage, the alignment stage includes an alignment sub-stage with movement, in which the slave device can move so as to be aligned with the position of the master device, thereby reducing or resetting the master-slave device offset to minimize - or even eliminate - the need to move the slave device during the alignment stage, thereby shortening the path of the slave device during the alignment process with the moving alignment stage.

[0030] Another object is that the present invention can be used in any robotic system for medical or surgical teleoperation, comprising at least one master device; at least one slave device, moved by actuators and controlled by said master device; a central microprocessor unit for controlling said slave devices; and a display for viewing these "helps".

[0031] The display may also be used to display, for example simultaneously display, the surgical site being operated on by the slave device.

[0032] The above objectives and other objectives can be at least partially achieved through the methods defined in the above embodiments.

[0033] According to the method of the present disclosure, when the central microprocessor unit receives an alignment assistance request signal, a graphical user interface is generated on the display, the graphical user interface including displaying at least one alignment assistance graphic element in a portion of the display.

[0034] On the one hand, the at least one alignment-aiding graphic element intuitively represents the relative spatial orientation information of the master device with respect to the slave device.

[0035] According to one aspect, the at least one alignment assistance graphical element represents instructions for allowing the operator to implement an alignment condition between the master device and the slave device.

[0036] According to one of the embodiments, the master device is a driverless type.

[0037] According to one of the embodiments, the main device is of the type that has no mechanical connection with the console, ie is of the "flying" type in the main working area.

[0038] According to one of the embodiments, the master device has N-fold symmetry along the axis, that is, when the rotation angle is equal to one Nth of the circumferential angle, the master device is equal to itself.

[0039] According to one of the embodiments, the step of generating on the display a graphical user interface comprising at least one graphical element to assist alignment is activated during a remote operation preparation step.

[0040] According to one of the embodiments, the method is performed in an alignment step prior to remote operation.

[0041] According to one of the embodiments, when the host device is detected to be located inside and / or outside a predetermined range, the step of generating a graphical user interface including at least one alignment-aiding graphical element on the display will be activated.

[0042] According to one of the embodiments, the step of generating a graphical user interface comprising at least one alignment aid graphical element on the display is activated when the master device is subjected to a predetermined gesture within a predetermined time interval. Preferably, the gesture is detected by detection of the master device (e.g., by optical and / or electromagnetic tracking).

[0043] According to one of the embodiments, when the master device is detected within a predetermined range and the opening / closing instructions of the master devices are activated simultaneously, preferably when the opening / closing instructions of two master devices (right and left) are activated simultaneously, the step of generating a graphical user interface including at least one alignment aid graphical element on the display is activated. For example, the opening and closing instructions may be activated repeatedly ("double click").

[0044] According to one of the embodiments, when the master device is detected within a predetermined range and turned upside down ("top-down" meaning that the surgeon's gesture is active in order to start the alignment phase), preferably when both master devices (right and left) are inverted at the same time, a step of generating on the display a graphical user interface comprising at least one graphical element assisting alignment will be activated.

[0045] According to one of the embodiments, the position of the slave device is calculated by internal measurements of sensors or actuators of the robotic system.

[0046] According to one of the embodiments, the position of the slave device is calculated by an algorithm analyzing an image of the slave device acquired by a vision system. According to one of the embodiments, the position of the slave device is calculated based on internal information of sensors or actuators of the robotic system combined with internal information of the vision system.

[0047] A robotic system is also described for implementing the method on the basis of a suitable computer program loaded into a central microprocessor unit.

[0048] Further embodiments are defined in the accompanying embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An embodiment of a robotic system for medical and / or surgical teleoperation is schematically illustrated;

[0050] Figure 2A schematically illustrates an example of interaction between a master device and a slave device, wherein the slave device is adapted to be movable so as to align with the master device;

[0051] Figure 2B A slave device with an articulated terminal is schematically shown, wherein a control point is defined which can only perform pure rotational movements within the working area of ​​said slave device during an alignment step;

[0052] Figure 3 is an example diagram of how to display two alignment aid graphic elements according to the present disclosure to indicate the relative orientation of each slave device with respect to the corresponding master device;

[0053] Figure 4 is a schematic diagram of a process for updating the appearance of an aid-alignment graphical element based on the relative orientation between the master device and the corresponding slave device according to the present disclosure;

[0054] Figure 5 shows how to display two alignment-aiding graphical elements according to the present disclosure in the form of a doughnut diagram;

[0055] 6A to 6FAn example of a circular crown-shaped aid alignment graphical element for indicating the relative orientation of the slave device with respect to the corresponding master device is shown;

[0056] Figure 7 Two examples of graphics to help align elements are shown, three-dimensional cones on the right and left;

[0057] Figure 8 Shows Figure 7 A detailed example of the graphical element to help with alignment is shown, wherein two three-dimensional cones respectively represent the orientation of the master device and the relative orientation of the slave device;

[0058] Fig. 9 Two examples of graphical elements that aid alignment are shown, which are three-dimensional cones on the right and left and are two-color;

[0059] Fig.10 Shows Fig. 9 A detailed example of the graphical element to help with alignment is shown, wherein two three-dimensional cones respectively represent the orientation of the master device and the relative orientation of the slave device;

[0060] Fig.11 Two examples of graphical elements that aid alignment according to the present disclosure are shown to depict the orientation of the master device and each slave device superimposed in a composite image;

[0061] Fig.12 Shows Fig.11 An example of detail of a graphical element to aid alignment is shown in which the orientation of a slave device is shown in a composite image partially superimposed on an image of a corresponding master device;

[0062] Fig.13A , Fig. 13B and Fig. 13C Other examples of alignment-aiding graphical elements according to the present disclosure are shown, for indicating the relative orientation of the slave device with respect to the corresponding master device;

[0063] Fig.14A , Fig. 14B and Fig. 14C Other examples of alignment-assisting graphical elements according to the present disclosure are shown, for providing instructions to an operator to achieve alignment between the master device and the slave device;

[0064] Figures 15 to 18 A state diagram showing different ways of initiating teleoperation after a graphical user interface is opened on a display of a robotic system according to the method of the present invention;

[0065] Fig.19 is a graph showing the trend of the master-slave offset over time according to possible operation methods;

[0066] Fig. 20 is a graph showing the trend of the master-slave offset over time according to possible operation methods;

[0067] Figure 21 to Figure 24 The steps of generating an alignment assistance request signal according to some possible operation modes are schematically illustrated. DETAILED DESCRIPTION

[0068] In the following description, reference will be made to Figures 1 to 24 A control method of a robotic system 100 for medical or surgical teleoperation is described. Generally, the robotic system 100 that can implement the control method of the present disclosure is Figure 1 of the type shown and including at least:

[0069] The main device 110 can be held by the operator 150 and is suitable for operation by the operator.

[0070] The slave device 170 may have at least one surgical instrument 170 and is used to be controlled by the master device 110 and moved by an actuator to track the movement of the master device.

[0071] A central microprocessor unit for controlling the slave device and moving it via one or more dedicated actuators, and a display 180.

[0072] The display 180 may be used to display the surgical site being operated on by the slave device 170 .

[0073] The robotic system 100 may include two slave devices (ie, two slave surgical instruments) and two master devices for controlling the two slave devices.

[0074] According to one embodiment, the display 180 is configured to display at least one graphical element 190 to aid alignment so as to avoid displaying the surgical site.

[0075] According to one embodiment, at least two displays are provided, one of which is used to display the surgical site and the other is used to display at least one alignment aid graphic element 190. For example, the display displaying at least one alignment aid graphic element 190 may be placed near the surgeon, for example, may be worn by the surgeon, such as gloves, masks, glasses, bracelets, wristbands, belts and / or the like.

[0076] exist Figure 1In the example shown, it is particularly noteworthy that the master device held by the surgeon is not mechanically constrained to the console. This type of master device is usually undriven, that is, the master device has no feedback system (joints) from the slave device, which may physically limit their operability, for example in the event of a master-slave offset exceeding a certain threshold being detected. The method of the present disclosure will be described with reference to this typical and particularly noteworthy case in the field of telemedicine, but it will be understood that the content also applies to the case where the master device is undriven, whether the master device is connected or not mechanically connected to the console, or the master device is mechanically connected to the console, and the master device is mechanically connected to the slave device (such as a slave robotic arm).

[0077] In particular, when the master device 110 is of the unconstrained or "flying" type, the operator considers it necessary to check whether the master device held by the user is aligned with the slave device seen on the display 180 of the robotic system, because the orientation of the master device 110 relative to the user's point of view may not correspond to the orientation of the surgical instrument 170 of the slave device relative to the point of view of the visual system recording it, such as the camera 160. This need is particularly manifested in the stage of preparation for medical or surgical teleoperation, so that once entering the master-slave teleoperation, the offset between the master device 110 controlled by the operator and the slave device 170 shown on the display 180 is minimal or zero.

[0078] Figure 2A Examples of a master device and a slave device are shown, as well as coordinate systems MF, SF that determine their respective orientation in space relative to reference coordinate systems MFO, SFO. Figure 2A In the example of , similar to a clamp, the body of the master device 110 includes two rigid components 134 , 135 integrated with respective sensors or markers (not shown) constrained in a rotating joint 136 to rotate about a common axis when the master device is manipulated by an operator 150 .

[0079] exist Figure 2A In the example shown, a possible alignment step is schematically shown in preparation for remote operation, in which the slave device 170 is able to move in the slave device working area (identified by the reference coordinate system SFO in the figure) so as to be aligned with the master device 110 (in the example, the starting posture of the slave device surgical instrument 170 is shown by shading and solid lines, and the target posture of the slave device surgical instrument 170 aligned with the posture of the master device 110 is shown without shading and dashed lines).

[0080] exist Figure 2BIn the example shown, a possible alignment step is shown, in which the slave device 170 only follows the master device 110 in direction. In other words, the control point 600 of the slave surgical instrument 170 can only perform a pure rotational movement to align with the posture of the master device 110, regardless of the positioning kinematic chain of the control point 600.

[0081] In order to determine the positions of the master and slave devices and their relative offset, it is convenient to identify a control point 600 which is used to locate the slave device. Figure 2A In the example, point 600 can be identified as a control point, which can belong to the surgical instrument of the slave device and / or a virtual point rigidly connected to the surgical instrument of the slave device. The surgical instrument of the slave device 170 may include multiple rotation joints P, Y, G, such as a pitch joint P, a yaw joint Y, and an open / close joint G, and / or the surgical instrument of the slave device may include multiple stacked vertebrae ("snake"), while the master device 110 preferably does not include a corresponding number of rotation joints. Therefore, at least six degrees of freedom of the slave device (three translations and three rotations, and, if necessary, the opening / closing degree of freedom G) can be identified through the control point 600, so that it is possible to establish an association between the posture of the master device and the posture of the slave device.

[0082] The relative orientation (offset) between the master device 110 and the slave device 170 is usually understood as a relative rotation in the rotation space SO(3). This relative rotation can be represented mathematically in different ways, such as Euler angles, axis angles or yaw angles: these representation modes can be associated with different visual technology schemes to help the operator with alignment. The Euler angle representation represents the relative rotation based on three rotations about orthogonal axes, and the axis angle representation represents the direction of the rotation and the total angle along the geodesic. The yaw angle representation is constructed by studying the two main directions in the slave device and the master device: the yaw angle is the angle of one main direction relative to the other main direction, while the torsion angle is the relative rotation angle along the said main directions.

[0083] In the robotic system 100, a visual system 160 may be provided that captures the physical, real slave device 170 in real time and displays the captured image on a display 182. In this way, a representation of the orientation of the slave device 170 on the display 180 (to assist in aligning the graphical element 190) may be obtained from the captured image (e.g., in roll-pitch-yaw and / or yaw coordinates). In addition, the representation on the display 180 (to assist in aligning the graphical element 190) may be obtained by applying computer vision and machine learning algorithms to these images. In this case, the visual system 160 preferably includes a microscope capable of magnifying the captured image.

[0084] For example, when the main device comprises two rigid parts hinged about a common axis, preferably, "alignment" refers to alignment about a longitudinal axis (e.g. Figure 2A alignment between the “x” axis in the figure) and a control point 600, wherein the longitudinal axis extends between the two rigid components 134, 135 and passes through the constrained rotation joint 136 of the master device 110, and the control point 600 is used to identify the slave device 170.

[0085] According to one aspect, the method of the present disclosure provides an operation of generating a graphical user interface on the display 180 by the central microprocessor unit when the central microprocessor unit receives the help alignment request signal. For example, the graphical interface can be as follows Figure 3 As shown, and includes a graphical element 190 to assist in alignment, Figure 3 It is represented as HELPER in the figure and its appearance can be Figure 5 to Figure 1 4. By Figure 3 As shown in the interface graphics, the surgeon 150 can display one or more graphical elements to clearly display the orientation of the master device 110 relative to the corresponding slave device 170, and the surgeon can see the relative orientation on the display 180.

[0086] According to a preferred aspect, the slave device 170 is displayed in a substantially central portion 182 of the display 180, while the alignment aid graphical element 190 is displayed in a portion of the display 180 that is not superimposed on the central portion 182 of the display, for example, in a corner 183 of the display or in another peripheral portion 184, so as not to obstruct the surgeon's 150 view of the slave device 170 of the robotic system 100. Preferably, the robotic system 100 comprises at least two slave devices 170, which are preferably controlled by at least two master devices 110, respectively.

[0087] The slave device 170 need not be displayed in the substantially central portion 182 of the display 180 , however in one embodiment the aid alignment graphical element 190 is displayed in a portion of the display 180 that is not superimposed on the same portion of the same display 180 where the slave device 170 is displayed.

[0088] According to one embodiment, the slave device 170 is not shown on the display 180, and at least one alignment-aiding graphical element 190 is displayed on the display 180. For example, a second additional display (not shown) may be provided, wherein the slave device 170 is displayed on the second additional display, and at least one alignment-aiding graphical element 190 is displayed on the first display 180.

[0089] According to one aspect of the disclosed method, Figure 4As shown, a first spatial orientation of a master device (or multiple master devices) and a second spatial orientation of a corresponding slave device (or multiple slave devices) are determined by a central microprocessor unit ("control unit"). A possible offset (i.e., relative orientation) Δ between the master device and the slave device is then determined, and the central microprocessor unit modifies the graphic aspects of the alignment assistance graphic element 190 in a manner that is in accordance with the relative spatial orientation of the master device relative to the slave device, so as to provide the surgeon with visual information that clearly understands the positioning of the master device relative to the slave device and allows him to achieve alignment conditions between the master device and the slave device. For example, the state or posture of the master device and / or other inputs generate the alignment assistance request signal ("trigger"), the master device is mapped into the coordinate system of the slave device, and three other inputs are processed to generate a visual representation of the feature alignment assistance graphic, in particular, the three other inputs are: the rotation of the mapped master device ("master device rotation parameter"), the rotation of the slave device ("slave device rotation parameter"), and the relative rotation in SO (3).

[0090] Figures 15 to 18 A state diagram is shown to illustrate the state of the robot system 100 (eg Figure 1 How to start remote operation after opening the graphical user interface according to the method of the present invention on the display 180 of the robot system shown in the figure. Once the graphical user interface is shown on the display, various situations illustrated in the figure will appear:

[0091] Fig.15 : After the graphical interface has been opened, if the relative spatial orientation between the master device and the slave device is within the pre-defined range, the graphical interface is closed and remote operation is automatically started;

[0092] Fig.16 : After opening the graphical interface, if the relative spatial orientation between the master device and the slave device is within a predetermined range, the graphical interface will not be automatically closed, and the graphical interface will notify the operator that the teleoperation is ready to start. Therefore, the robotic system waits for the generation of an enabling signal ("trigger") for medical or surgical teleoperation;

[0093] Fig.17 : After the graphical interface has been opened, if the relative spatial orientation between the master device and the slave device exceeds the acceptable tolerance range, but is contained within a second maximum range ("funnel"), which includes the acceptable tolerance range, then when an enable signal ("trigger") is generated (e.g., an alignment enable signal and / or a medical or surgical teleoperation enable signal), an alignment phase is initiated, for example, the operator keeps the master device stationary, and the robotic system automatically and / or autonomously moves the slave device until the relative spatial orientation between the master device and the slave device falls within the acceptable tolerance range. At this point, the graphical interface is closed, and teleoperation is started;

[0094] Fig.18 : After the graphical interface has been opened, if the relative spatial orientation between the master device and the slave device exceeds the second maximum range ("funnel"), when there is an enabling signal ("trigger") for entering medical or surgical teleoperation, the alignment phase is started by the following steps: for example, the slave device is kept stationary and the operator is asked to move the master device via the graphical interface until the relative spatial orientation between the master device and the slave device is not included in the second maximum range. In this case, the graphical interface will inform the operator to keep the master device stationary and automatically and / or autonomously move the slave device until the relative spatial orientation between the master device and the slave device falls within an acceptable tolerance range. At this point, the graphical user interface is closed and teleoperation is started.

[0095] According to one aspect, the graphical user interface is closed in any case when the maximum time for which the graphical user interface is open has been reached, due to the generation of an alignment assistance request signal, and / or for example a situation in which remote operation is not entered.

[0096] According to a preferred embodiment, the robot system generates the help alignment request signal by the central microprocessor unit outside of the teleoperation, preferably in a teleoperation preparation step.

[0097] According to an embodiment, the robotic system generates the auxiliary alignment request signal by means of the central microprocessor unit during a paused teleoperation phase, for example a pause envisaged for teleoperation between two consecutive fully slaved teleoperation phases.

[0098] According to an embodiment, the robotic system generates the help alignment request signal via the central microprocessor unit during a limited teleoperation phase in which a subset of the slave device degrees of freedom remain slaved to the master device, while a second subset of the degrees of freedom are blocked.

[0099] Of course, the various aspects described below for generating an alignment assistance request signal may be provided in any combination thereof. Indeed, a need exists to be able to activate an alignment procedure between a master device (s) and a slave device (s) in a simple and safe manner.

[0100] According to one aspect, the need to generate the alignment assistance request signal may be to detect the master device or the two master devices within a predetermined operating area. The predetermined operating area may coincide with the entire operating area 115 of the master device (e.g. defined by the limits of the "tracking" area) or may be a sub-area 116 defined by the central microprocessor unit, wherein the master device 110 must be located in the predetermined operating area in order to generate the alignment assistance request signal. In this way, as Fig.24As shown, the predetermined working area 116 in which the master device must be located in order to generate the help alignment request signal is a subset of the master device working area 115. For example, the predetermined working area can exclude areas where the master device is stored when not in use, such as containers and / or the like. For example, the shape of the predetermined working area can be a sphere and / or two spheres, such as a sphere for each master device, and / or a box. An outer area can be provided and the inner area is included in the outer area of ​​the working area, wherein the predetermined working area in which the master device must be located in order to generate the help alignment request signal can be the outer area or the peripheral area.

[0101] According to one aspect, when the on / off command of the master device 110 is activated, an assist alignment request signal is generated by the central microprocessor unit. Fig.21 As shown, a slight pressure of the opening / closing command (grip force, G) of the master device generates an assisting alignment request signal. In the case where two master devices are provided, for example a right master device for the surgeon's right hand and a left master device for the surgeon's left hand, an assisting alignment request signal may be generated by the central microprocessor unit when the opening / closing command of both master devices is activated simultaneously or within the time window of the first closing / opening command. According to one embodiment, an assisting alignment request signal is generated by the central microprocessor unit when the opening / closing command of the master device is repeatedly activated, for example a "double click".

[0102] According to one aspect, for example, Fig. 22 As shown, when it is detected that the master device is inverted (i.e. "top-down"), a help alignment request signal is generated by the central microprocessor unit. In this case, more conditions can be provided, for example, when it is detected that the master device 110 is inverted and stationary, i.e. in a non-free fall state, the help alignment request signal is generated. For example, the help alignment request signal can be generated under a reverse master device condition and an activated open / close command. In addition, in order to determine that the surgeon's gesture is intended to indicate the intention to start the alignment phase, the vertical position of the master device can be further checked, for example, Fig. 22 As shown, it is detected whether the surgeon flips the main device by simultaneously raising his arms.

[0103] According to one aspect, when a gaze detection device installed in the robotic system 187 detects that the operator is looking in a predetermined direction, such as towards the screen 180 or a region or portion of the screen 180, an aid-alignment request signal is generated by the central microprocessor unit.

[0104] According to one aspect, when the host device 110 is directed toward the screen 180 or display 180 in a predetermined direction, the help alignment request signal is generated by the central microprocessor unit.

[0105] According to one aspect, for example Fig.23 As shown, when alignment is detected between the surgeon's hand 151 grasping the main device 110, the surgeon's at least one eye 152 and at least a portion of the screen 180, a help alignment request signal is generated by the central microprocessor unit. For example, a camera can detect such alignment. For example, a gaze detection device 187 can detect such direction.

[0106] According to one aspect, the method includes the operation of generating an alignment assistance request signal by the central microprocessor unit when the central microprocessor unit detects a predetermined movement by the master device. In actual operation, the central microprocessor unit determines the position of the master device (or multiple master devices), and when the central microprocessor unit itself recognizes that the surgeon has performed a predetermined movement of the master device, the central microprocessor unit activates the graphical user interface.

[0107] On the one hand, the central microprocessor unit determines the position of the master device (or multiple master devices), and when the central microprocessor unit itself detects that the master device is in a predetermined spatial posture or is located in a predetermined master device working area, the central microprocessor unit activates the graphical user interface. Through the above arrangement, the surgeon can activate the graphical user interface by continuing to hold the master device (or multiple master devices) without moving his eyes away from the display, which shows the surgical site operated by the slave device (or multiple slave devices).

[0108] The alignment assistance request signal causing the central microprocessor unit to activate the graphical user interface may also be generated in other ways.

[0109] According to one aspect, the robotic system may be equipped with a gaze detection device 187 of a user using the robotic system to generate an aid in aiming request signal (and thereby open a graphical user interface) when the user looks in a predetermined direction.

[0110] According to another aspect, at least one or each master device 110 of the robotic system may be equipped with a specific switch 158, such as a button, to generate a help alignment request signal when the switch is activated (eg pressed).

[0111] According to one aspect, each master device may be equipped with a capacitive or other sensor which, when touched by the surgeon holding the master device, generates an alignment assistance request signal for the central microprocessor unit, thereby opening a graphical user interface on the display. The alignment assistance request signal may be generated by placing a button or capacitive sensor on the master device in a position accessible by a simple movement of a finger, without changing the grip and without the operator having to take his eyes off the display.

[0112] According to one aspect, if the robot system includes a foot switch 158 functionally connected to the central microprocessor unit, when the foot switch is depressed, an assist alignment request signal may be generated and then a graphical user interface may be displayed on the display.

[0113] On the one hand, the graphical user interface can be closed when the pedal is released regardless of whether alignment is achieved between the master device and the slave device.

[0114] According to one aspect, when the instruction of the master device 110 is adapted to control the opening / closing degree of freedom G of the slave device 170, an assist alignment request signal may be generated by the central microprocessor unit.

[0115] The main device 100 comprises two rigid parts 134, 135 hinged to each other around a common axis like a clamp, for example Figure 2A As shown, the two rigid components 134 and 135 remain angularly separated from the spring 137 when the spring is stationary, and can only approach each other by overcoming the elastic repulsive force of the spring. When the two rigid components approach each other until the angle between them is less than a predetermined angle threshold, a request signal for assisting alignment can be generated by the central microprocessor unit, wherein the angle threshold takes into account the elastic influence of the spring.

[0116] According to another aspect, because the main device 110 of the robot system has two rigid clamp-shaped articulated parts 134, 135, the help alignment request signal can be generated by quickly closing the rigid clamp-shaped parts twice in succession ("double click"). The main device 110 may include an elongated body that provides an open / close command, and the help alignment request signal can be generated by quickly activating the open / close command twice in succession ("double click").

[0117] On the one hand, the help alignment request signal is automatically generated by the central microprocessor unit. According to an optional embodiment, when a predetermined condition is detected during the master-slave remote operation, the help alignment request signal is automatically generated by the central microprocessor unit. The predetermined conditions detected during the remote operation may include: the slave device approaches the physical limit of the slave device's working area (the limit of the joint, such as the limit of the pitch P and / or yaw Y and / or roll R joint of the articulated terminal of the surgical instrument 170), and / or the posture instruction issued to the slave device exceeds the working area of ​​the slave device (for example, it exceeds the working area of ​​the pitch P and / or yaw Y and / or roll R joint). The predetermined conditions that can be detected during the remote operation may include: a state transition from a limited remote operation state or a state transition to a limited remote operation state, for example, during this state transition process, the slave device only follows the master device in azimuth, and / or translation, and / or a subset of degrees of freedom including azimuth and / or translation.

[0118] Figure 5 to Figure 14 shows that Figure 3 Various examples of graphical interfaces visualized on a display are shown schematically, with reference to the case where the robotic system comprises two master devices for the surgeon to grasp and corresponding two slave devices to operate at the surgical site, but the content described below also applies to the case where there is only one master device and one slave device, so the graphical user interface only includes one graphical element. In the illustrated case with two master devices and two slave devices, the displayed graphical user interface will include a first graphical alignment aid element for the first pair of master and slave devices and a second graphical alignment aid element for the second pair of master and slave devices. The central microprocessor unit determines the position of the master devices and the slave devices and modifies the appearance of the two graphical elements in order to intuitively represent the current relative position of each master device and the corresponding slave device.

[0119] The graphic element may be represented by a variety of methods, wherein the graphic element intuitively represents the offset between the master device and the slave device or the respective positions of the master device and the slave device updated in real time.

[0120] For example, Figure 5 As shown, the two alignment aid graphic elements 190 may be located at opposite corners of the screen 180, and each element may be depicted as a sector having a different color C1, C2, C3 and at least one concentric circle of an indicator F, F1, F2 (such as an arrow, cursor and the like). Fig. 6A Shown in more detail Figure 5 , wherein the inner circle represents the twist angle, and the outer sector thereof represents the swing angle; the arrow F represents the orientation of the master device; the color C1 represents the area where the offset is below an acceptable threshold, such as a threshold that enables remote operation (e.g., 5-8 degrees), the color C2 represents the area where the offset is below a "funnel" threshold, such as enabling the slave device to move to align with the master device (e.g., 40-60 degrees), and the color C3 represents the area outside the slave device working area. It should be noted that in this example, due to the 2-fold rotational symmetry around the roll x-axis of the master device 110, two acceptable thresholds (color C1) for the twist angle ("master device flip" mode) are displayed.

[0121] For example, in Figure 6B In the figure, the torsion angle and the swing angle are represented on two different, independent geometric figures, where the circle represents the swing angle and the bar represents the torsion angle; in this figure, the color C1 corresponding to the acceptable threshold is placed in the "12 o'clock" configuration; the arrows F1 and F2 represent the master device 110; in this representation, the colors C1, C2, and C3 represent the same Fig. 6A Similar to what is mentioned in .

[0122] For example, in Figure 6C In, with Fig. 6ASimilar to what is mentioned in , the torsion angle and the swing angle are represented on concentric circles.

[0123] For example, in Fig.6D In, with Figure 6B Similar to what is mentioned in , the ring represents the swing angle, while the bar represents the torsion angle.

[0124] For example, in Fig. 6E In the diagram, the swing angle is represented by a ring (angular representation), and the torsion angle is represented as a radial extension of the same ring; thus in this example, the indicator F can move both angularly and radially in the ring, i.e. the ring becomes thicker / thinner as a function of the torsion angle.

[0125] For example, in Fig. 6F In FIG. 1 , the twist angle and the oscillation angle are represented as two orthogonal axes on a Cartesian plane, and the indicator F is represented as a crosshair.

[0126] Naturally, those skilled in the art can refer to Figures 6A-6F As described in the example in , choose to exchange the graphical representation of the torsion angle and the swing angle.

[0127] According to one aspect, Figure 7 and Figure 8 As shown, each graphical element can be represented by two cones, one representing the master device and the other representing the slave device, and the offset information between the master device and the slave device is intuitively provided in the angular coordinates of the spherical reference system, wherein the offset information is displayed by the representation of the master device and the slave device. In more detail, Figure 8 Details of a graphical element to aid alignment are shown, the graphical element comprising two three-dimensional cones representing the position of the master device and the relative position of the slave device updated in real time, respectively, and in one embodiment the two cones have different colors.

[0128] On the one hand, if Fig. 9 and Fig.10 As shown, each graphical element can be represented by two long cones. For example, the coloring of the cones can include information showing the offset along the rolling axis or the torsion axis, that is, the offset information along the longitudinal dimension of the master device and / or the slave device. In more detail, Fig. 9 It is shown how two left and right graphic elements are visually displayed on a display, the two graphic elements being respectively used to assist alignment according to the content of the present disclosure, and having the shape of a three-dimensional cone with a two-color polar scroll logo.

[0129] According to a preferred aspect, each graphic element is Fig.11 and Fig.12The type shown, depicts a virtual, i.e. synthetic slave device, as if the surgeon saw a real slave device in the central part of the display, partially superimposed on another graphical element depicting a corresponding master device. In the case of this example, the slave device is essentially composed of two rigid parts integrated with respective sensors or markers 134, 135 constrained in a rotating joint (such as a clamp) so as to rotate around a common axis when the master device is manipulated by an operator 150, and the graphical element describing the master device and the graphical element describing the slave device can be superimposed at the rotating joint so that there is no offset between the master device held by the operator and the rigid clamp-shaped articulated part of the slave device operating at the surgical site.

[0130] For example, in Fig.13A In , the graphical elements may be represented by an arrow F and at least one cone, the arrow F representing the offset between the master and the slave, the cone visually representing the maximum tolerable offset between the master and the slave. For example, the size of the arrow F may increase with the offset. For example, two or more concentric cones colored with colors C1, C2 may be provided to illustrate acceptable thresholds and "funnel" thresholds (e.g., 5-8 degrees and 40-60 degrees, respectively). Similarly, in Fig. 13B , a frustum is shown instead of a cone. For example, in Fig. 13C In the figure, the two arrows represent the master device and the slave device respectively, and the foot of the arrow represents the torsion angle or the rolling angle.

[0131] according to Figures 14A-14C In one aspect shown, the graphical element may depict instructions for achieving the alignment condition, instructing the operator how to move (rotate) the master device 110 to align it with the orientation of the slave device. For example, Fig.14A Two arrows are shown; Fig. 14B showing arrows superimposed on the representation of the surgeon's 150 hand 151 to indicate twisting or rolling rotation instructions; Fig. 14C Arrows are shown superimposed on the representation of the master device 110 to indicate twist or roll instructions.

[0132] On the one hand, the real-time master-slave offset is described in the form of an abstract and simplified image to improve the surgeon's intuition. According to one embodiment, such as shown in Figure 6b or 6c, the offset is displayed in real time as a cursor moves along a line or bar, and the cursor can display a color-coded color representing the offset. For example, when the offset is less than a certain acceptable threshold, the position of the cursor corresponds to the green portion of the line or bar, and when the offset is greater than a certain acceptable threshold, the position of the cursor corresponds to the yellow, orange and / or red portion of the line or bar.

[0133] In one embodiment, the offset is displayed in real time as the hand rotates along a portion of a circle, such as a ring. Providing a simplified abstract representation of the relative orientation (offset) between the master and slave devices allows redundant information to be avoided, making the alignment process of the master and slave devices more intuitive.

[0134] According to a preferred embodiment, the alignment aid graphic element is not an overlay (technical term: "ghost") on the "real" slave device 170 image captured by the vision system 160. In other words, it is best to avoid overlaying the alignment aid graphic element as a "ghost". This choice allows a more effective representation to be obtained, which is independent of the current posture of the slave device and is therefore easier to understand in any movement configuration of the slave device. In fact, "ghosting" is usually based on augmented reality technology: the system uses "computer vision" technology to identify the slave devices in the images and overlays three-dimensional transparent models ("ghosts") on these images to show the instrument posture mapped by the corresponding master device posture. This representation method has the disadvantages of the following problems: problems of self-occlusion that are difficult to understand, problems of variable positions on the screen 180 that may be close to the edge, or problems of mutual occlusion between two close instruments. Fig.12 A graphical element to aid alignment according to the present disclosure is shown depicting the orientation of a slave device in a composite image partially superimposed on a corresponding master device image to represent the relative orientation of the slave device relative to the corresponding master device, wherein in one embodiment, the composite image may be a simulation or CAD rendition of the slave device or the master device, but may not be related to the device represented by the visual system. One of the disadvantages of ghosting in augmented reality technology is that the roll axis of a real surgical instrument, i.e., the longitudinal axis of the instrument, is ubiquitous, which is not conducive to adequate illustration, especially in the case of miniaturization of the articulated end portions (pitch, yaw, grip) of the surgical instrument.

[0135] In certain embodiments, the above-mentioned requirements may be met by providing the above-mentioned features individually or collectively, so as to obtain the above-mentioned advantages, in particular:

[0136] As described above, the remote operation preparation phase includes a slave device movement alignment phase, in which the slave device can be moved to a position to reach the master device, and at least one graphical element that helps alignment is provided to minimize the physical movement of the slave device during the movement alignment phase. Fig. 20 );

[0137] Graphical elements that aid alignment may be generated in response to precise instructions from the user;

[0138] Graphic elements that help with alignment can be automatically generated based on the status of the robot system;

[0139] For example, during the teleoperation phase, in which the master device's instructions require the slave device to reach the physical limits of the working area of ​​the rotational joint of the slave device's articulated surgical instrument, the graphical element will appear;

[0140] Unnecessary movement of the slave device can be avoided or at least reduced to a minimum;

[0141] Providing a simple and robust technical solution that allows clear and intuitive representation of any posture of the slave device, i.e. the spatial configuration of the joints of the surgical instruments of the slave device;

[0142] In particular, by avoiding the creation of at least one alignment aid graphic element in the form of a "ghost" on the actual slave device captured by the visual system, a clear, intuitive and effective visual display can be achieved even when the motion configuration of the slave device joint is unfavorable, such as when the control point 600 is obscured by the surgical instrument body itself or the second surgical instrument body itself.

[0143] The various embodiments of the control method of the robot system disclosed in the present invention can be implemented by writing a computer program, which is loaded into the internal memory of the central microprocessor unit of the robot system and enables the central microprocessor unit to perform the operations of the above control method after startup.

[0144] Reference Number List

[0145]

[0146]

[0147]

Claims

1. A control method for a robotic system (100) for medical or surgical teleoperation, characterized in that: The robotic system comprises: at least one master device (110) operable by an operator and adapted to be moved by the operator, at least one slave device (170) adapted to be moved by an actuator and controlled by the master device, a central microprocessor unit for controlling the slave device, and a display (180), The method comprises performing the following operations: determining, by the central microprocessor unit, a first spatial position of the master device and a second spatial position of the slave device; When the central microprocessor unit receives the alignment assistance request signal, a graphical user interface is opened on the display by the central microprocessor unit, the graphical user interface comprising at least one alignment assistance graphic element (190) displayed in a portion of the display to visually represent information of the relative spatial orientation of the master device relative to the slave device updated in real time and / or instructions allowing the operator to achieve an alignment condition between the master device and the slave device; updating the alignment aid graphical element shown on the display in a manner corresponding to the relative spatial orientation of the master device with respect to the slave device; When at least one of the following conditions is met, the graphical user interface is closed by the central microprocessor unit: The relative spatial orientation is within a predetermined interval; generating an enabling signal for medical or surgical teleoperation; A maximum time that the graphical user interface has remained open has elapsed since the assist-with-alignment request signal was generated.

2. The method according to claim 1, characterized in that The master device (110) is of a non-drive type and a non-constraint type.

3. The method according to claim 1 or 2, characterized in that: The display is used to display at least the surgical site operated by the slave device in its central part (182), The method comprises the following steps: displaying the surgical site and the slave device at a central portion of the display; The alignment aiding graphical element is displayed at a peripheral portion (184) of the display, the alignment aiding graphical element not being superimposed on a central portion (182) of the display showing the slave device.

4. The method according to claim 1 or 2, characterized in that: comprising displaying said at least one alignment aid graphical element on said display (180) to avoid viewing said surgical site; and / or wherein The robotic system includes at least a first display and a second display, and the method includes the following operations: viewing the surgical site on the first display and displaying the at least one alignment aid graphical element on the second display (190).

5. The method according to any one of the preceding claims, characterized in that The invention also includes an operation of generating the alignment assistance request signal by the central microprocessor unit in an alignment step in preparation for the remote operation step in addition to the remote operation.

6. The method according to any one of the preceding claims, characterized in that The operation of generating the alignment assistance request signal by the central microprocessor unit comprises one of the following conditions or a combination thereof: When the closing / opening instruction of the master device is activated, and when the closing / opening instructions of two master devices are activated simultaneously or within the time window of the first closing / opening instruction; When the on / off command of the master device is repeatedly activated; When the master device is turned over by a gesture within a specified time interval, and when two master devices are turned over at the same time; When a gaze detection device installed in the robot system detects that the operator is looking in a predetermined direction, wherein, when looking towards the display; When the master device is subjected to a predetermined gesture (151) within a predetermined time interval.

7. The method according to any one of the preceding claims, characterized in that The operation of generating the alignment assistance request signal by the central microprocessor unit comprises one of the following conditions or a combination thereof: When the central microprocessor unit detects that the master device is in a predetermined operating area, wherein when a sub-area (116) of the operating area of ​​the master device (115) calculated by the central microprocessor unit; When the central microprocessor unit detects that a predetermined action is performed by the master device; When the central microprocessor unit detects that the main device is in a predetermined spatial position; When the central microprocessor unit detects that a manual switch of the robot system is pressed, wherein the manual switch (158) is mounted on a handle of the main device (110) and connected to the central microprocessor unit; When the central microprocessor unit detects that a foot switch (158) of the robot system is pressed, wherein the foot switch is connected to the central microprocessor unit.

8. The method according to any one of the preceding claims, characterized in that Included are operations that generate help to align graphic elements so that: The relative spatial orientation between the master device and the slave device is intuitively represented.

9. The method according to any one of claims 1 to 7, characterized in that: Included are operations that generate help to align graphic elements so that: The first spatial position of the master device and the second spatial position of the slave device are intuitively represented.

10. The method according to any one of the preceding claims, characterized in that Included are operations that generate help to align graphic elements so that: The information of the graphical indicator is visually represented, and the master device is moved to achieve a second spatial orientation alignment with the slave device.

11. The method according to any one of the preceding claims, characterized in that When the relative spatial orientation is within a predetermined interval, an enable signal for medical or surgical teleoperation is generated.

12. The method according to any one of claims 1 to 10, characterized in that When the relative spatial orientation is within a predetermined range, generating a slave device movement enable signal to align with the orientation of the master device; When the relative spatial orientation is within a narrow range, the narrow range is included in the predetermined range, and an enabling signal for medical or surgical teleoperation is generated.

13. The method according to any one of the preceding claims, characterized in that The medical or surgical teleoperation enable signal is generated by a request from the operator, wherein the request from the operator includes activating a control interface, wherein the control interface includes a control pedal.

14. The method according to any one of the preceding claims, characterized in that When the relative spatial orientation is within a predetermined range, a signal for enabling medical or surgical teleoperation is automatically generated.

15. A computer program installable in the internal memory of a microprocessor unit, characterized in that The method comprises software code which, when executed, causes the microprocessor unit to perform a plurality of operations in the method according to any one of the preceding claims.

16. A robotic system (100) for medical or surgical teleoperation, characterized in that: At least: The main device (110), controlled by an operator, a slave device (170) comprising at least one surgical instrument adapted to be moved by an actuator and controlled by the master device, a central microprocessor unit for controlling the slave devices; Display (180); Wherein, the computer program according to claim 15 is installed in the central microprocessor unit.

17. The robot system according to claim 16, characterized in that: Also included is a foot switch operatively connected to the central microprocessor unit, the foot switch being configured to generate an assist alignment request signal when depressed.

18. The robot system according to claim 16 or 17, characterized in that: The master device is a driverless type, wherein the master device is a driverless type and has no mechanical constraints with the console.

19. The robot system according to claim 16, 17 or 18, characterized in that Includes two monitors, one for displaying graphical elements to aid alignment and the other for viewing the surgical site.

Images