Surgical robot and method of controlling same, control device

Abstract

This application relates to a surgical robot and its control method and control device. The control method includes: acquiring a first target rotation vector input by an input device; determining the target position and / or orientation of the target imaging center of an imaging instrument in a reference coordinate system based on the first target rotation vector; determining target joint variables of the joints in the manipulator and imaging instrument based on the target position and / or orientation; and controlling the movement of the joints in the manipulator and imaging instrument according to the target joint variables, so that the imaging center of the imaging instrument reaches the target imaging center. Through the above implementation, it is possible to ensure that the image acquired by the imaging instrument is the desired image.

Description

Technical Field

[0001] This application relates to the field of medical devices, and in particular to a surgical robot and its control method and control device. Background Technology

[0002] Minimally invasive surgery refers to a surgical procedure performed inside the human body using modern medical instruments and equipment such as laparoscopes and thoracoscopes. Compared to traditional surgical methods, minimally invasive surgery has advantages such as less trauma, less pain, and faster recovery.

[0003] With advancements in technology, minimally invasive surgical robot systems have matured and are widely used. A surgical robot system includes a master control console and slave operating devices. The slave operating devices include multiple medical devices with end effectors, such as imaging devices with imaging end effectors and surgical instruments with manipulating end effectors. The master control console includes a monitor and an operating unit. The surgeon operates the operating unit to move the imaging devices or surgical instruments, guided by the field of view provided by the imaging devices displayed on the monitor.

[0004] When adjusting the field of view, surgeons typically expect to rotate the imaging device around its distal end to obtain the desired image. However, due to the structural and motion characteristics of minimally invasive surgical robotic systems, it is usually not possible to directly control the imaging device to rotate around its distal end. Instead, it needs to rotate around other centers of rotation, such as a specific distal fixed point during laparoscopic surgery. Because the center of rotation changes during the movement of the imaging device, the image actually obtained by the device may not be what the surgeon expects. Summary of the Invention

[0005] Therefore, it is necessary to provide a surgical robot capable of obtaining the desired image, as well as its control method and control device.

[0006] On one hand, this application provides a surgical robot, comprising: an imaging instrument for insertion into a body opening to acquire images; a manipulator for manipulating the imaging instrument to rotate about a telecentric fixed point located at the body opening and to feed along the axial direction of the imaging instrument; an input device for inputting a first target rotation vector of the imaging instrument rotating about its distal end in a target pose degree of freedom; and a control device coupled to the manipulator and the input device, configured to: acquire the first target rotation vector input by the input device; determine the target position and / or pose of the target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector; determine the target joint variables of the joints in the manipulator and the imaging instrument based on the target position and / or pose; and control the joint movement in the manipulator and the imaging instrument according to the target joint variables so that the imaging center of the imaging instrument reaches the target imaging center.

[0007] The step of determining the target joint variables of the joints in the manipulator and the imaging device based on the target position and / or attitude includes: obtaining the configuration parameters of the imaging optical axis of the imaging device; constructing a kinematic model associated with the manipulator, the imaging device, and the imaging optical axis by combining the configuration parameters of the imaging optical axis; and determining the target joint variables of the joints in the manipulator and the imaging device based on the target position and / or attitude and the kinematic model.

[0008] The configuration parameters include the length of the imaging optical axis and / or the angle of the imaging optical axis relative to the imaging surface of the imaging instrument.

[0009] The control device is configured to: acquire the imaging distance range of the imaging instrument; generate a configuration interface based on the imaging distance range, including one or more selectable target imaging distances, the target imaging distances being between the minimum and maximum imaging distances of the imaging distance range; and, in response to selecting the target imaging distance through the configuration interface, configure the selected target imaging distance as the length of the imaging optical axis.

[0010] The control device is configured to: generate a configuration interface including one or more selectable target angles, the target angles being between 0° and 90°; and, in response to selecting the target angle through the configuration interface, configure the selected target angle as the angle of the imaging optical axis relative to the imaging surface.

[0011] The step of determining the target position and / or orientation of the target imaging center of the imaging device in the reference coordinate system based on the first target rotation vector includes: determining a first position and / or orientation of the target imaging center relative to the distal end point of the imaging device; and determining the target position and / or orientation of the first position and / or orientation in the reference coordinate system based on the current position and / or orientation of the distal end point in the reference coordinate system.

[0012] Wherein, the movement of the user's head in the target pose degree of freedom is associated with the acquired first target rotation vector, and the surgical robot further includes: an image host, including an observation component and a pose adjustment component, the pose adjustment component being used to adjust the pose of the observation component, the observation component being used to observe the image acquired by the imaging instrument; the control device is coupled to the pose adjustment component and is further configured to: control the movement of the pose adjustment component according to the first target rotation vector, so that the observation component follows the movement of the user's head in the target pose degree of freedom.

[0013] The observation component is configured to move in sync with the movement of the imaging instrument.

[0014] The attitude adjustment component includes: a base; a first pivot member pivotally connected to the base and configured to rotate relative to the base in a first attitude degree of freedom; and a second pivot member pivotally connected to the first pivot member and configured to rotate relative to the first pivot member in a second attitude degree of freedom; the observation component is fixedly connected to the second pivot member.

[0015] Wherein, the base includes a first curved slide rail, the first pivot member includes a second curved slide rail and a third curved slide rail, the second pivot member includes a fourth curved slide rail, the first curved slide rail and the second curved slide rail are slidably engaged to rotate in a first degree of freedom, and the third curved slide rail and the fourth curved slide rail are slidably engaged to rotate in a second degree of freedom; or, the base includes a first curved slide rail, the first pivot member includes a second curved slide rail, the first curved slide rail and the second curved slide rail are slidably engaged to rotate in a first degree of freedom, the attitude adjustment joint further includes a rotational joint, and the second pivot member and the first pivot member are connected through the rotational joint to rotate in a second degree of freedom.

[0016] The input device includes one or more base stations capable of receiving and / or transmitting wireless signals, and also includes a wearable device that can be worn on a user's head. The wearable device includes one or more beacons capable of transmitting and / or receiving wireless signals. Based on sensing the distance between the one or more beacons and the one or more base stations, a first target rotation vector associated with the movement of the user's head can be determined.

[0017] The imaging device includes a rigid imaging device.

[0018] The imaging device includes a flexible imaging device.

[0019] On the other hand, this application provides a control method for a surgical robot, the surgical robot comprising:

[0020] An imaging device is inserted into a body opening to acquire images; a manipulator is used to manipulate the imaging device to rotate around a telecentric fixed point located at the body opening and to feed it along the axial direction of the imaging device; an input device is used to input a first target rotation vector for the imaging device to rotate around its distal end point in a target pose degree of freedom; the control method includes: acquiring the first target rotation vector input by the input device; determining the target position and / or pose of the target imaging center of the imaging device in a reference coordinate system based on the first target rotation vector; determining the target joint variables of the manipulator and the joints in the imaging device based on the target position and / or pose; and controlling the movement of the joints in the manipulator and the imaging device according to the target joint variables so that the imaging center of the imaging device reaches the target imaging center.

[0021] On the other hand, this application provides a computer-readable storage medium storing a computer program configured to be loaded by a processor and execute steps implementing the control method as described in any of the above embodiments.

[0022] On the other hand, this application provides a control device for a surgical robot, comprising: a memory for storing a computer program; and a processor for loading and executing the computer program; wherein the computer program is configured to be loaded by the processor and execute steps implementing the control method as described in any of the above embodiments.

[0023] The surgical robot, its control method, and control device disclosed in this application have the following beneficial effects:

[0024] By aligning the imaging device with the target imaging center based on the first target rotation vector rotated around its distal end, the target joint variables of the joints in the manipulator and imaging device are determined. Then, the movement of the joints in the manipulator and imaging device is controlled according to the target joint variables, so that the imaging center of the imaging device can reach the target imaging center. This achieves the same imaging effect as when the imaging device is manipulated to rotate around its distal end with the first target rotation vector, which ensures that the image actually acquired by the imaging device is consistent with the image expected to be acquired. Attached Figure Description

[0025] Figure 1 This is a simplified diagram of the device relationships of a surgical system according to one embodiment;

[0026] Figure 2 This is a schematic diagram of the structure of a surgical system according to one embodiment;

[0027] Figure 3 This is a schematic diagram of the structure of an imaging device according to one embodiment;

[0028] Figure 4 This is a schematic diagram of a doctor's main control console according to one embodiment;

[0029] Figure 5 This is a flowchart illustrating a control method for a surgical robot according to one embodiment;

[0030] Figure 6 This is a schematic diagram of the structure of another imaging device according to one embodiment;

[0031] Figure 7 yes Figure 4 A schematic diagram illustrating the structural principle of an imaging device shown.

[0032] Figure 8 yes Figure 6 A schematic diagram illustrating the structural principle of an imaging device shown.

[0033] Figure 9 yes Figure 7 A schematic diagram of the motion state of an embodiment of the imaging device shown;

[0034] Figure 10 yes Figure 7 A schematic diagram of the motion state of another embodiment of the imaging device shown;

[0035] Figure 11 yes Figure 7 A schematic diagram of the motion state of another embodiment of the imaging device shown.

[0036] Figure 12 This is a flowchart illustrating another control method for a surgical robot according to one embodiment;

[0037] Figure 13 This is a flowchart illustrating yet another control method for a surgical robot according to one embodiment;

[0038] Figure 14 This is a schematic diagram illustrating the structural principle of an input device according to one embodiment;

[0039] Figure 15 This is a schematic diagram of the screen of an image host according to an embodiment;

[0040] Figure 16 This is a schematic diagram of the structure of an image host according to one embodiment;

[0041] Figure 17 yes Figure 16 The diagram shows a structural schematic of an embodiment of the attitude adjustment component in the image host.

[0042] Figure 18 yes Figure 17 An enlarged schematic diagram of the second pivot component in the attitude adjustment assembly shown;

[0043] Figure 19 This is a schematic diagram of the structure of another embodiment of the attitude adjustment component in an image host, according to one embodiment;

[0044] Figure 20 This is a schematic diagram illustrating the structural principle of a portion of the input device according to an embodiment;

[0045] Figure 21 This is a schematic diagram of the control device of a surgical robot system according to an embodiment of this application. Detailed Implementation

[0046] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0047] It should be noted that when an element is referred to as being "set on" another element, it can be directly on the other element or there may be an intervening element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intervening element. When an element is considered to be "coupled" to another element, it can be directly coupled to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this application are for illustrative purposes only and do not represent the only possible implementation. The terms "distal" and "proximal" used in this application are directional terms commonly used in the field of interventional medical devices, where "distal" refers to the end away from the operator during surgery, and "proximal" refers to the end closer to the operator during surgery. The terms "first / second," etc., used in this application can refer to a component or two or more components with common characteristics.

[0048] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The terms "each" and "a plurality" as used in this application include one or more.

[0049] Figure 1 This is a simplified diagram of the equipment relationships of a surgical system according to one embodiment. Figure 1As shown, the surgical system 100 includes a surgical robot and an operating table 105. The surgical robot includes a bedside robotic arm system 101, a doctor's main control panel 103, and an imaging cart imaging system 108. It can be understood that the composition of the surgical robot is not limited to this.

[0050] The bedside robotic arm system 101 includes a drive arm with multiple joints. A puncture device is mounted at the distal end of the drive arm. The puncture device is inserted into a body opening of a patient 106 lying on the surface of an operating table 105. The puncture device provides a channel between the surgical robot and the organism (including humans / animals). Medical instruments are inserted into the organism through this connection channel. The medical instruments include imaging instruments providing a field of view and surgical instruments providing surgical operations such as cutting, suturing, dicing, cauterizing, and cleaning. The body opening includes surgical incisions and / or natural cavities in the organism, such as a patient's body. Exemplary natural cavities include the mouth, nose, and anus.

[0051] The doctor's main control console 103 and the bedside robotic arm system 101 communicate in real time via data transmission path 120. The surgeon's actions on the main control console 103 are performed by manipulating the medical devices in the bedside robotic arm system 101 through a master-slave mapping relationship based on a kinematic model. Simultaneously, the main control console 103 can monitor the status of the bedside robotic arm system 101, such as monitoring the motion information of each joint. When the operating table 105 performs movements of corresponding degrees of freedom, the patient 106, fixed to the surface of the operating table 105, remains relatively stationary relative to the surface. Changes in the patient's position are achieved by the operating table 105 performing movements of corresponding degrees of freedom. The motion information of each joint of the operating table 105 is recorded and stored in real time, and transmitted to the bedside robotic arm system 101 via data transmission path 150. Data transmission between the main control console 103 and the operating table 105 occurs via data transmission path 130. Images of the surgical site on patient 106 are acquired by an imaging device installed in the bedside robotic arm system 101. This imaging device is connected to the imaging cart system 108. The images acquired by the imaging device are transmitted to the imaging cart system 108 via data transmission path 110. Then, the imaging cart system 108 transmits the images acquired by the imaging device back to the doctor's main control panel 103 in real time via data transmission path 160, providing the doctor with a surgical field of view and facilitating the smooth execution of the surgery. In practice, data transmission paths 110, 120, 130, 150, and 160 can be either wired or wireless.

[0052] Figure 2 This is a schematic diagram of a surgical system according to one embodiment. Figure 2The diagram illustrates the structure of the bedside robotic arm system 101 and the operating table 105 of the surgical robot. The bedside robotic arm system 101 includes a motion chassis 201, a robotic arm 250, and drive arms. The motion chassis 201 can move the bedside robotic arm system 101 as a whole in any direction on a horizontal surface. The robotic arm 250 is used to position one or more drive arms as a whole. The drive arms include an adjusting arm 260 and a manipulator 270, which is sometimes referred to as a manipulator assembly. The robotic arm 250, adjusting arm 260, and manipulator 270 typically each include one or more joints, including one or more of rotary and translational joints.

[0053] The motion chassis 201 can adopt a wheeled moving structure, which makes the relative positional relationship between the bedside robotic arm system 101 and the operating table 105 more flexible. There are no restrictions on the designated location area. On-site medical staff can push it to complete the positioning operation and locking operation after positioning according to the actual surgical needs. While being able to get close enough to the operating table 105, it is convenient for the preoperative positioning of each manipulator 270 above the patient's body.

[0054] The robotic arm 250 includes a fixed support column 203 fixedly connected to the motion chassis 201 for supporting all the motion joints, a lifting column 204 for performing the overall lifting linear motion J1 of the robotic arm 250, a large arm 205 and a small arm 206 for performing rotational motions J2 and J3 respectively, and a directional platform 207 for controlling one or more adjustment arms 260 to perform the overall rotational motion J4. The movement of these joints can quickly reach the expected preoperative positioning area, which helps to shorten the docking time between the preoperative bedside robotic arm system 101 and the patient 106.

[0055] One or more adjusting arms 260 are connected to the orientation platform 207 individually or in parallel via rotary joints J5. In some examples, the bedside robotic arm system 101 has multiple adjusting arms 260. Considering that the configurations of the multiple adjusting arms 260 are basically the same and the motion descriptions of each joint are basically the same, therefore... Figure 2 The structure is presented using only one adjusting arm 260 and one manipulator 270 as examples, along with the description of the joint motion relationships below. In some examples, the adjusting arm 260 includes a small rotating platform 208, a telescopic arm 209 that performs linear translation J6 in a horizontal direction parallel to the ground, a fixed vertical arm 210 fixedly connected to the telescopic arm 209, a movable vertical arm 211 that performs up-and-down lifting J7 in a vertical direction perpendicular to the ground, a turning head 212 that performs rotational movement J8, and a cyclone joint 213 that performs rotational movement J9.

[0056] The manipulator 270 includes a deflection joint 214 that rotates J10 with the cyclone joint 213, a parallelogram linkage base 215, a first link 216 and a second link 217 that perform the rotational motion J11, and a holding arm 218 for performing linear motion J12 of the medical device 219 along the guide rail direction. A trocar 229 is mounted at the distal end of the manipulator 270. The distal fixed point 220 of the puncture device 229, which is at the same location as the body opening of the patient 106, is defined by the intersection of the axis of the cyclone joint 213 and the axis of the deflection joint 214. These two axes also converge at the distal fixed point 220 of the puncture device 229, where they intersect with the lateral center plane of the parallelogram linkage device base 215. Furthermore, the first link 216 and the second link 217, as two adjacent sides, together with two virtual adjacent sides parallel to them, form a parallelogram motion mechanism. This mechanism is controlled by a motor and performs folding and opening movements of the parallelogram motion around the rotational motion J11 axis. The distal fixed point of the parallelogram also intersects the distal fixed point 220 of the puncture device 229 at a single point, and this intersection point is located on the central axis of the medical device 219. The distal end 221 of the medical device is inserted into the body of the patient 106 and performs surgical actions performed by the doctor on the main control panel based on a master-slave mapping relationship. Figure 2 As shown, controlling the rotational movement J10 of the yaw joint 214 relative to the cyclone joint 213 causes the puncture device 229 and the medical instrument 219 to move in the yaw degree of freedom around the telecentric fixed point 220; controlling the rotational movement J11 of the second link 217 relative to the first link 216 causes the puncture device 229 and the medical instrument 219 to move in the pitch degree of freedom around the telecentric fixed point 220. For example, the pitch joint controlling the rotation of the second link 217 relative to the first link 216 can be located on the parallelogram linkage base 215. The movement of the puncture device 229 and the medical instrument 219 around the telecentric fixed point 220 is mainly used to avoid stress tearing of the body opening caused by changes in the position of the puncture device 229 and the medical instrument 219.

[0057] In applicable Figure 2 Among the medical devices shown, the bedside robotic arm is... Figure 3Taking the rigid endoscope (i.e., imaging device) 219 as an example, it generally includes four degrees of freedom: one translational degree of freedom and three rotational degrees of freedom around the telecentric fixed point 220. The manipulator 270 provides the imaging device 219 with pitch, yaw, and translational degrees of freedom along J12, while the imaging device 219 itself provides a tumble degree of freedom. The telecentric fixed point 220 is typically the intersection of the central axis of the imaging device 219 and the body opening. This telecentric fixed point 220 is usually defined by the physical structure of the manipulator 270. During surgery, this telecentric fixed point 220 needs to be kept stationary to minimize the impact on the human body during the procedure, such as preventing the incision from enlarging.

[0058] The manipulator assembly includes a manipulator 270 and a medical device 219. Exemplarily, controlling the medical device, such as an imaging device 219, in its pitch degree of freedom about a telecentric fixed point 220 can be achieved by controlling the movement of the first joint of a joint in the manipulator assembly; for example, controlling... Figure 2 The pitch joint movement shown can control the imaging device 219 to rotate in pitch freedom around the telecentric fixed point 220. For example, when controlling the imaging device 219 to rotate in yaw freedom around the telecentric fixed point 220, the movement of the second joint in the manipulator assembly can be controlled; for example, controlling the movement of the yaw joint 214 can control the imaging device 219 to rotate in yaw freedom around the telecentric fixed point 220.

[0059] In some embodiments, the surgical robot also includes an input device configured to input a target motion vector of the medical device 219. This target motion vector can be configured to be the motion vector of the distal end effector of the medical device 219 relative to a reference point or a reference coordinate system. The input device can be integrated into the doctor's main control panel 103 or independent of it. In some embodiments, the input device includes one or more of the following: a linkage-type operating unit, a magnetic navigation-type operating unit, a voice recognition device, an eye-tracking device, and a head-tracking device. Figure 4 The doctor's control panel 103 shown includes a linkage-type operating unit 1031.

[0060] In some embodiments, the surgical robot also includes a control unit comprising one or more processors. These processors are coupled to the bedside robotic arm system 101, the surgeon's main control panel 103, and the imaging cart system 108. The processors may be integrated into one of the bedside robotic arm system 101, the surgeon's main control panel 103, and the imaging cart system 108, or they may be distributed across two or more of these systems, or they may be deployed in the cloud.

[0061] In some embodiments, a method for controlling a surgical robot is provided, the method being configured to be executed by a control device of the surgical robot, see reference. Figure 5 The method includes:

[0062] Step S11: Obtain the first target rotation vector input by the input device, which indicates that the desired imaging instrument rotates around its distal end in the target attitude degree of freedom.

[0063] The distal end of the imaging instrument typically includes an image end effector of the forming instrument, and the distal end of the imaging instrument is exemplarily a point in the image end effector, which exemplarily includes the center point of the imaging surface.

[0064] The target attitude degree of freedom may include a single attitude degree of freedom or two or more combined attitude degrees of freedom.

[0065] Step S12: Determine the second target rotation vector based on the first target rotation information, showing the rotation of the imaging device around the telecentric fixed point in the target attitude degrees of freedom.

[0066] In surgical robots, it is difficult to expect the distal end of the imaging instrument to rotate around its distal endpoint in the target orientation degree of freedom. Therefore, it is desirable for the distal end of the imaging instrument to rotate around other rotation centers in the target orientation degree of freedom. Considering the safety and reliability requirements of surgical robots during surgery, for example, to meet the need to avoid stress tearing of the body opening caused by changes in the position of puncture devices and medical instruments, the desired rotation of the distal end of the imaging instrument around its distal endpoint can be actually transformed into rotation around a telecentric fixed point to achieve adjustment of the desired field of view.

[0067] Step S13: Control the imaging device to rotate around the telecentric fixed point according to the second target rotation vector control manipulator.

[0068] The imaging instrument can be manipulated to rotate around the telecentric fixed point in a variety of ways.

[0069] In some embodiments, the target joint and its target joint amount in the manipulator, which are associated with the target attitude degrees of freedom, can be determined based on the second target rotation vector, and then the target joint can be controlled to move the target joint amount. For example, in... Figure 2In the bedside robotic arm system 101 shown, when the second target rotation vector is associated with the yaw degree of freedom, the yaw joint 214 can be identified as the target joint and its target joint value can be determined; when the second target rotation vector is associated with the pitch degree of freedom, the pitch joint, i.e., the joint that controls the rotation of the second link 217 relative to the first link 216, can be identified as the target joint and its target joint value can be determined; when the second target rotation vector is associated with both the yaw and pitch degrees of freedom, the yaw joint 214 can be identified as one target joint and its target joint value can be determined, while the pitch joint can be identified as another target joint and its target joint value can be determined. Then, by controlling these target joints in the manipulator 270 to move their corresponding target joint values, the imaging device 219 can be manipulated to rotate around the telecentric fixed point 220.

[0070] In some embodiments, the imaging device 219 can also be rotated around the telecentric fixed point 220 based on the kinematic control manipulator 270. For example, the current pose of the distal end of the imaging device 219 can be obtained, and the target pose of the distal end of the imaging device 219 can be determined based on the current pose and the second target rotation vector. Then, based on the target pose and using inverse kinematics, the target joint amount of the joint in the manipulator assembly can be determined, and finally, the joint movement in the manipulator assembly can be controlled based on the target joint amount.

[0071] Step S14: In response to the imaging distance deviation caused by the rotation vector of the second target around the telecentric fixed point relative to the rotation vector of the first target around the distal point, the control manipulator manipulates the imaging device to feed along its axis to compensate for the imaging distance deviation.

[0072] The feeding and rotation of the imaging device are synchronized to ensure stable performance throughout the entire process of adjusting the imaging device.

[0073] Step S14 includes: obtaining the imaging distance deviation of the second target rotation vector of the imaging instrument rotating around the telecentric fixed point relative to the first target rotation vector rotating around the distal end point; and based on the imaging distance deviation, controlling the manipulator to maneuver the imaging instrument to feed along the axis of the imaging instrument.

[0074] In some embodiments, the imaging device includes a link and an image end effector coupled to the distal end of the link. When there is no wrist joint between the link and the image end effector, the imaging device is a rigid imaging device, and the image end effector does not have the ability to move relative to the distal end of the link; when there is a wrist joint between the link and the image end effector, the imaging device is a flexible imaging device, and the image end effector has the ability to move relative to the distal end of the link. Figure 3 and Figure 6The imaging device 219 (219') shown does not have a wrist joint between the link 2191 (2191') and the image end effector 2192 (2192'), making it a rigid imaging device.

[0075] In some embodiments, imaging devices can also be further categorized. For example, they can be classified based on whether the imaging optical axis of the imaging device is parallel (including coincident) to the extension direction of the connecting rod. Wherein, if the imaging optical axis is parallel to the extension direction of the connecting rod, the imaging device is a zero-degree endoscope; if it is not parallel to the extension direction of the connecting rod, i.e., there is an angle, the imaging device is a non-zero-degree endoscope. Figure 7 and Figure 8 In the imaging device 219 (219') shown, for example, the straight line between the telecentric fixed point 220 (220') and the distal end point 2193 (2193') can be taken as the extension direction 2196 (2196') of the connecting rod 2191 (2191'), and the optical axis perpendicular to the imaging surface, i.e., the mirror surface 2194 (2194'), is the imaging optical axis 2195 (2195'). The imaging optical axis includes a plurality of optical axes that are parallel or non-parallel to each other. For example, the imaging optical axis described in this application may refer to the optical axis that passes through the center of the imaging surface and / or the center of the image end effector and is perpendicular to the imaging surface, and may also be called the central imaging optical axis. Figure 7 In the middle, the imaging optical axis 2995 is parallel to the extension direction of the connecting rod 2196, and the imaging instrument 219 is a zero-degree endoscope. Figure 8 In this configuration, the imaging optical axis 2995' is parallel to the extension direction of the connecting rod 2196', and the imaging instrument 219' is a non-zero-degree endoscope. Specifically, the angle between the imaging optical axis 2995' and the extension direction of the connecting rod 2196' is 30°, and the imaging instrument 219' is a 30-degree endoscope.

[0076] in, Figure 7 Yes Figure 3 The indication, Figure 8 Yes Figure 6 The illustration, therefore. Figure 3 or Figure 7 The imaging device 219 shown is a rigid zero-degree endoscope. Figure 6 or Figure 8 The imaging device 219' shown is a rigid 30-degree endoscope.

[0077] by Figure 7The above embodiment will be explained using a rigid zero-degree endoscope (also referred to as a rigid zero-degree endoscope) as an example. The imaging device 219 is currently aligned with the actual imaging center point A. Without changing the target imaging distance of the imaging device 219, the user expects to rotate the imaging device 219 around its distal end point 2193 to reach the first target rotation vector α in order to see the target imaging center B. However, in reality, since the distal end of the imaging device 219 cannot rotate around the distal end point 2193, but can only rotate around the telecentric fixed point 220, the imaging device 219 can only be controlled to rotate around the telecentric fixed point 220.

[0078] See Figure 9 If the imaging device 219 is rotated around the telecentric fixed point 220 to reach the first target rotation vector α, the amplification effect of the rigid link 2191 during rotation can easily cause the actual imaging center point C to deviate significantly from the desired target imaging center point B. Therefore, the imaging device 219 cannot be directly controlled to rotate around the telecentric fixed point 220 to reach the first target rotation vector α. Consequently, considering the user's expectations, the second target rotation vector β can be determined based on the first target rotation vector.

[0079] See Figure 10 Furthermore, after determining the second target rotation vector β, if the imaging device 219 is rotated around the telecentric fixed point 220 to reach the second target rotation vector β, it is easy to cause a distance difference B between the actual aligned imaging center D and the target imaging center to be aligned, i.e., an imaging distance deviation. This imaging distance deviation will affect the imaging effect; at the very least, there will be a deviation between the center of the field of view that the user expects to see through the imaging device and the center of the field of view that is actually seen, for example, a depth-of-field deviation on the optimal imaging plane.

[0080] Therefore, while controlling the manipulator 270 to rotate the imaging device 219 around the telecentric fixed point 220, the manipulator 270 can also control the imaging device 219 to feed along its axis J12 to compensate for the aforementioned imaging distance deviation. This allows for consistency between the actual observed and the desired imaging effect, meaning that both the actual imaging center point and the target imaging center point are B. Figure 11 As shown. For example, it can ensure the desired sharpness of the boundary of the circle centered on the distal end of the imaging instrument and with the imaging distance as the radius, as well as the area outside that boundary.

[0081] In some embodiments, determining a second target rotation vector based on a first target rotation vector, allowing the imaging instrument to rotate around a telecentric fixed point in the target attitude degrees of freedom, includes:

[0082] Step 121: Obtain a first distance and a second distance. The first distance includes the distance between the imaging device's telecentric fixed point and its distal end point, and the second distance includes the target imaging distance of the imaging device. In some embodiments, the target imaging distance includes the imaging distance emitted from the center of the image end effector perpendicular to its imaging surface, i.e., the mirror surface. For example, this imaging distance is the optimal imaging distance of the imaging device or other suitable imaging distance. The first target rotation vector input by the user can be understood, for example, as the target rotation vector expected to rotate the center. In some embodiments, the center of the image end effector can be a feature point associated with the distal end point of the imaging device, which may coincide with or be slightly offset from the distal end point. For example, the distal end point of the imaging device can be used as the center of the image end effector. Alternatively, the center of the imaging surface in the image end effector can be used as the center of the image end effector. Step 122: Combine the first target rotation vector, the first distance, and the second distance to determine a second target rotation vector for the imaging device to rotate around the telecentric fixed point in the target attitude degree of freedom.

[0083] In one embodiment, the second target rotation vector can be determined using trigonometric functions and based on the first target rotation vector, the first distance, and the second distance. See also... Figure 10 For example, the arctangent trigonometric function can be used to determine the second target rotation vector, which can be determined by the following formula:

[0084]

[0085] Where α represents the rotation vector of the first target, β represents the rotation vector of the second target, L represents the first distance, and d represents the second distance.

[0086] For example, based on the same principle, the second target rotation vector can also be determined using trigonometric function formulas such as inverse cotangent, arcsine, and inverse cosine. These will not be illustrated here.

[0087] In some embodiments, step S14 above, controlling the manipulator to manipulate the imaging device to feed along its axis to compensate for imaging distance deviation, includes:

[0088] S141, Obtain the initial position and / or initial attitude of the imaging instrument.

[0089] The initial position and / or initial orientation of the imaging device includes the position and / or orientation at the moment the input device and the manipulator assembly establish a master-slave mapping relationship, representing the zero-position state of the imaging device. The initial position and / or initial orientation of the imaging device can refer to the initial position and / or initial orientation of a feature region on the imaging device. In this embodiment, the feature region includes a preset region located at the distal end of the rotation point. The feature region comprises an area consisting of one or more points.

[0090] S142, Obtain the current position and / or current attitude of the imaging instrument.

[0091] The current position and / or current orientation of the imaging device can be determined based on the current joint variables of the manipulator components, such as the joints in the manipulator, and using positive kinematics.

[0092] S143, based on the target position and / or target orientation of the imaging device compared with its current position and / or orientation relationship with its initial position and / or initial orientation, determine the target direction of the imaging device's feed along its axis.

[0093] The target position and / or target orientation of the imaging device are determined based on a second target rotation vector. More specifically, the target position and / or target orientation of the imaging device can be determined based on the second target rotation vector and the current joint variables of the manipulator components, such as joints in the manipulator, using forward kinematics.

[0094] Specifically, when the target position and / or target posture of the imaging device is relatively far from the initial position and / or initial posture compared to its current position and / or current posture, the target direction is determined as the direction in which the imaging device retracts into the body opening along its axis; or, when the target position and / or target posture is relatively close to the initial position and / or initial posture compared to its current position and / or current posture, the target direction is determined as the direction in which the imaging device inserts into the body opening along its axis.

[0095] S144 controls the imaging device to feed along its axis in the target direction to compensate for imaging distance deviation.

[0096] Among them, the feeding direction of the imaging device is basically consistent with the target direction, and the feeding amount of the imaging device is basically consistent with the deviation of the imaging distance.

[0097] In some embodiments, the imaging distance deviation can be determined based on the following principle:

[0098] A first length of the imaging instrument between the telecentric fixed point and the current imaging center is obtained, and a second length between the telecentric fixed point and the actual imaging center is obtained when the imaging instrument is assumed to rotate around the telecentric fixed point by a second rotation vector. The imaging distance deviation can be determined based on the difference between the first length and the second length.

[0099] For example, based on the above principle, the imaging distance deviation can be determined as follows:

[0100] Step S21: Obtain the first distance and the second distance.

[0101] The first distance includes the distance between the telecentric fixed point and the distal point of the imaging instrument, and the second distance includes the target imaging distance of the imaging instrument.

[0102] In one embodiment, the method for obtaining the first distance includes:

[0103] Obtain the joint variables of the joints in the manipulator assembly; combine the joint variables and the first kinematic model of the manipulator assembly, and use forward kinematics to determine the first position of the distal end point in the reference coordinate system, and determine the second position of the telecentric fixed point in the reference coordinate system; determine the first distance based on the first position and the second position.

[0104] In one embodiment, the method for obtaining the second distance includes:

[0105] The imaging instrument's imaging distance range is obtained; a configuration interface is generated based on the imaging distance range, including one or more selectable target imaging distances; in response to the selection of a target imaging distance through the configuration interface, the selected target imaging distance is used as a second distance. By flexibly configuring the second distance, different imaging effects can be achieved to meet different surgical needs.

[0106] Continue reading Figure 10 The first length can be determined based on the sum of the first distance and the second distance. Wherein:

[0107] L1 = L + d (Equation 2)

[0108] Where L1 represents the first length, L represents the first distance, and d represents the second distance.

[0109] Step S22: Combine the first target rotation vector, the first distance, and the second distance to determine the imaging distance deviation.

[0110] For example, trigonometric functions can be used to determine the second length based on the first target rotation vector, the first distance, and the second distance. The calculation formula can be exemplarily expressed as follows:

[0111]

[0112] Where L2 represents the second length and α represents the first target rotation vector.

[0113] ΔL=L1-L2 Equation (4)

[0114] Where ΔL represents the imaging distance deviation, and L1 represents the first length. For example, in Figure 10 In this context, ΔL represents the distance between DB.

[0115] Substituting equations (2) and (3) into equation (4), we get:

[0116]

[0117] That is, the imaging distance deviation can be determined according to equation (5).

[0118] The above-described embodiment, which uses trigonometric functions to determine the rotation vector of the second target and / or the imaging distance deviation to control the imaging device, is well-suited for use with zero-degree endoscopes, especially rigid zero-degree endoscopes. Figure 2 The bedside robotic arm system shown can use a rigid zero-degree endoscope or a rigid non-zero-degree endoscope.

[0119] In some embodiments, rigid zero-degree endoscopes and rigid non-zero-degree endoscopes are typically implemented through structural design. For example, in a rigid zero-degree endoscope, the mirror surface is configured to be perpendicular to the extension direction of the connecting rod. As another example, in a rigid non-zero-degree endoscope, the mirror surface is configured to be inclined to the extension direction of the connecting rod.

[0120] In some embodiments, flexible endoscopes are typically adapted to and manipulated by the manipulators of a single-port surgical robot to provide greater degrees of freedom of movement. The telecentric fixed point of the single-port surgical robot can be controlled by a software algorithm or by a similar mechanism. Figure 2 The illustrated bedside robotic arm system is physically defined by a manipulator with a parallelogram mechanism. Flexible endoscopes include flexible zero-degree endoscopes and flexible non-zero-degree endoscopes. A flexible zero-degree endoscope can be defined as a flexible endoscope in which, in the initial (i.e., zero-position) state, the imaging optical axis is parallel (or coincident) to the extension direction of the link. In this state, the imaging surface is set perpendicular to the extension direction of the link. A flexible non-zero-degree endoscope can be defined as a flexible endoscope in which, in the initial state, the imaging optical axis forms an angle with the extension direction of the link. In this state, the imaging surface is set inclined to the extension direction of the link.

[0121] Specifically, the configuration of a flexible zero-degree endoscope can be changed by controlling the movement of the wrist joint to meet the usage requirements of a non-zero-degree endoscope; conversely, the configuration of a flexible non-zero-degree endoscope can also be changed by controlling the movement of the wrist joint to meet the usage requirements of a zero-degree endoscope.

[0122] In some embodiments, maintaining the wrist joint and connecting rod in a straight line position for the flexible zero-degree endoscope can be applied to the embodiments described in S11-S14 above. For example, when it is necessary to apply the technical solutions described in the embodiments of S11-S14 above, if the wrist joint and connecting rod in the flexible zero-degree endoscope are not in a straight line, the wrist joint movement can be controlled to restore it to a zero-position state relative to the connecting rod, i.e., in a straight line.

[0123] For example, the initial joint variables of the wrist joint can be recorded when the wrist joint and the link are in a straight line. When controlling the movement of the wrist joint to make the wrist joint and the link in a straight line, the corresponding wrist joint can be reset directly based on the current joint variables and the initial joint variables.

[0124] In some embodiments, when the rotation vector of the imaging device around its distal end is converted into rotation around another rotation center, such as a telecentric fixed point, other methods can be used to achieve the desired imaging effect or maintain the imaging effect unchanged. This imaging effect refers to the situation where, when the rotation center of the imaging device is changed, the distance between the distal end of the imaging device and the target imaging center remains consistent with the target imaging distance. The control method generally includes: obtaining the target position and / or orientation of the target imaging center of the imaging device in a reference coordinate system, such as the base coordinate system of a bedside robotic arm system; determining the target joint amount of the joints in the manipulator assembly based on inverse kinematics; and then controlling the joint movement in the manipulator assembly according to the target joint amount, so that the imaging center of the imaging device reaches the target position and / or orientation from its current position and / or orientation. This method is applicable to imaging devices with or without wrist joints and is a general method. For examples, please refer to... Figure 12 The control method includes:

[0125] Step S11': Obtain the first target rotation vector input by the input device, which is intended for the imaging instrument to rotate around its distal end in the target attitude degree of freedom.

[0126] The first target rotation vector includes the target rotation vector input by the input device, which represents the rotation of the distal end of the desired imaging instrument around its distal end in the target attitude degree of freedom.

[0127] Step S12': Determine the target position and / or orientation of the imaging instrument's target imaging center in the reference coordinate system based on the first target rotation vector.

[0128] Step S13': Determine the target joint variables of the joints in the manipulator assembly based on the target position and / or attitude.

[0129] This involves determining the target joint variables of the joints in the manipulator and imaging device.

[0130] Step S14': Control the joint movement in the manipulator assembly according to the target joint variable so that the imaging center of the imaging instrument reaches the target imaging center.

[0131] This refers to controlling the joints in the manipulator and imaging device to move according to the target joint variables.

[0132] By using the steps S11' to S14' described above, it is not necessary to determine the second target rotation vector and imaging distance deviation required in steps S11 to S14, and the same imaging effect can still be achieved.

[0133] In some embodiments, determining the target position and / or orientation of the imaging instrument's target imaging center in the reference coordinate system based on a first target rotation vector includes:

[0134] Step 131': Determine the first position and / or orientation of the target imaging center relative to the distal end of the imaging instrument.

[0135] For example, suppose the current position and / or orientation of the distal endpoint in the reference coordinate system is P0(P x0 ,P y0 Assuming the imaging device rotates by a first target rotation vector α around its distal end in a two-dimensional xy-plane, the first position and / or orientation of the target imaging center relative to the distal end of the imaging center is P1(P x0 +dcosα,P y0 +dsinα).

[0136] Step 132': Based on the current position and / or attitude of the distal endpoint in the reference coordinate system, determine the target position and / or attitude of the first position and / or attitude in the reference coordinate system.

[0137] Specifically, the current position and / or orientation of the distal end of the imaging device in the reference coordinate system can be determined based on the first kinematic model associated with the manipulator assembly and the current joint variables of the joints in the manipulator assembly.

[0138] In some embodiments, see Figure 13 Determine target joint variables of the joints in the manipulator assembly based on the target position and / or attitude, including:

[0139] Step S141': Obtain the configuration parameters of the imaging optical axis of the imaging device.

[0140] In some embodiments, the configuration parameters of the imaging optical axis include the length of the imaging optical axis and the angle of the imaging optical axis relative to the imaging surface, such as a mirror. The configuration parameters of the imaging optical axis can be determined based on acquired attribute information of the imaging device. This attribute information includes the imaging distance range of the imaging device, which includes at least one of a minimum imaging distance, a maximum imaging distance, and an optimal imaging distance between the minimum and maximum imaging distances. The attribute information also includes the type of imaging device, which may be a zero-degree endoscope or a non-zero-degree endoscope.

[0141] In some embodiments, the length of the imaging optical axis can be determined based on the acquired imaging distance range of the imaging device, wherein the length of the imaging optical axis can be understood as the target imaging distance. For example, when the imaging distance range includes a minimum imaging distance, any imaging distance greater than or equal to the minimum imaging distance can be configured as the length of the imaging optical axis. For example, when the imaging distance range includes an optimal imaging distance, the optimal imaging distance can be configured as the length of the imaging optical axis. For example, when the imaging distance range includes a minimum imaging distance and a maximum imaging distance, any imaging distance between the minimum and maximum imaging distances can be configured as the length of the imaging optical axis. In some embodiments, the imaging distance range of the imaging device is related to its focal length. For an imaging device with a fixed-focus image end effector, the imaging distance range is relatively unique, while for an imaging device with a zoom image end effector, the imaging distance range includes different imaging distance ranges corresponding to different focal lengths.

[0142] In some embodiments, the angle of the imaging optical axis relative to the imaging surface can be determined based on the type of imaging device acquired. For example, when the acquired imaging device is a zero-degree endoscope, the angle of the imaging optical axis relative to the imaging surface can be determined to be 90 degrees. As another example, when the acquired imaging device is a +30-degree endoscope, the angle of the imaging optical axis relative to the imaging surface can be determined to be +60 degrees. As yet another example, when the acquired imaging device is a -30-degree endoscope, the angle of the imaging optical axis relative to the imaging surface can be determined to be -60 degrees. In some embodiments, since the first angle between the imaging optical axis and the extension direction of the link, and the second angle between the imaging optical axis and the imaging surface, are usually complementary angles, the second angle can be determined based on the first angle when the first angle is stored in the memory chip. Alternatively, the second angle can be directly stored in the memory chip.

[0143] The attribute information of the imaging device can be stored in the memory chip of the imaging device. When the imaging device is installed in the controller, the attribute information is read by the reading interface in the controller and transmitted to the control device for processing.

[0144] Step S142': Combine the configuration parameters of the imaging optical axis to construct a second kinematic model associated with the manipulator component and the imaging optical axis.

[0145] In some embodiments, the second kinematic model may only relate to the manipulator assembly and the imaging optical axis. In some embodiments, the second kinematic model may relate to at least a portion of the drive arm, including the manipulator assembly, and the imaging optical axis. The aforementioned first kinematic model did not consider the imaging optical axis, while this second kinematic model does, specifically considering the length of the imaging optical axis and its angle relative to the imaging plane. Therefore, the second kinematic model differs from the first kinematic model. More specifically, the construction of the first kinematic model is only related to the physical arm structure, while the construction of the second kinematic model is related not only to the physical arm structure but also to the virtual arm structure, i.e., the imaging optical axis. In the second kinematic model, the structure of the imaging device is extended and tangibly considered, ensuring that when reaching the target imaging center, the actual distal end of the imaging device, such as... Figure 7 The distal endpoint 2193 shown or as Figure 8 The distance between the distal end point 2193' shown and the target imaging center can always be maintained at the length of the imaging optical axis, i.e., the target imaging distance.

[0146] Step S143': Based on the target position and / or attitude, the second kinematic model, and using inverse kinematics, determine the target joint variables of the joints in the manipulator assembly.

[0147] For example, when the second kinematic model is associated only with the manipulator assembly and the imaging optical axis, target joint variables of the joints in the manipulator and imaging device can be determined. As another example, when the second kinematic model is associated with the entire drive arm including the manipulator assembly and the imaging optical axis, target joint variables of the joints in the drive arm can be determined.

[0148] In some embodiments, different second kinematic models can be constructed to achieve different image effects desired by the user. In some embodiments, configuration parameters of the imaging optical axis can be configured. For example, the angle of the imaging optical axis relative to the imaging plane can be configured, and / or the length of the imaging optical axis can be configured.

[0149] For example, a configuration interface can be generated based on an imaging distance range, including one or more selectable target imaging distances; in response to the selection of a target imaging distance through the configuration interface, the selected target imaging distance is used as the length of the imaging optical axis. For instance, when the imaging distance range includes a minimum imaging distance and a maximum imaging distance, multiple target imaging distances can be generated between the minimum and maximum imaging distances for configuration.

[0150] For example, a configuration interface can be generated that includes one or more selectable target angles; in response to the selection of a target angle through the configuration interface, the selected target angle is used as the angle of the imaging optical axis relative to the imaging plane. For instance, multiple target angles between 0 and 90° can be generated for configuration.

[0151] Of course, the length of the imaging optical axis and / or the angle of the imaging optical axis relative to the imaging plane can also be configured in other ways, such as through voice recognition.

[0152] In some embodiments, it is assumed that the rotation center of the imaging device is changed. Since the rotation of the imaging device in the rotation degree of freedom (i.e., roll degree of freedom) does not change the target imaging distance, but the rotation in the pitch degree of freedom and / or yaw degree of freedom will change the target imaging distance, the method of the above embodiments is particularly applicable to the rotation of the imaging device in the pitch degree of freedom and / or yaw degree of freedom.

[0153] The aforementioned first target rotation vector can be input in various ways. For example, it can be input through the operating unit located on the main control panel. Alternatively, it can be input through the touchscreen on the main control panel.

[0154] Typically, imaging instruments and surgical instruments are controlled via a manipulator. When a surgeon is operating surgical instruments with both hands, if they need to control the imaging instrument to adjust their field of vision, they must pause control of at least one surgical instrument before switching to control the imaging instrument. However, during surgery, there is usually a high frequency of switching between imaging and surgical instruments, making this switching operation complex and inefficient. Moreover, even if one hand can operate a surgical instrument while the other simultaneously operates the imaging instrument, coordination problems exist, resulting in a poor user experience and failing to meet the surgeon's needs. Therefore, this application provides an alternative input device that allows control of the imaging instrument without the surgeon's hand input. In other embodiments, the input from the input device can also control the surgical instrument as needed, requiring only the establishment of a master-slave mapping relationship between the input device and the surgical instrument. Notably, since the control of the surgical instrument does not involve imaging distance issues, it is only necessary to control the distal end of the surgical instrument to move according to the target motion information input by the input device. This target motion information can include not only motion information in the degrees of freedom of posture but also motion information in the degrees of freedom of position.

[0155] In some embodiments, the input device includes a voice recognition component, allowing the doctor to generate specific commands by uttering specific sounds, which are then processed by the voice recognition component. These specific commands correspond to a first target rotation vector. For example, the doctor can issue commands such as "left," "right," "up," "down," "upper left," "upper right," "lower left," "lower right," "forward," and "backward," generating incremental motion vectors corresponding to the respective degrees of freedom. For example, the doctor can issue commands such as "5 degrees to the left," "5 degrees to the right," "1 cm forward," and "1 cm backward," generating incremental motion vectors corresponding to the respective degrees of freedom. Furthermore, the rotation vector within the incremental motion vectors can be used as the aforementioned first target rotation vector.

[0156] In some embodiments, the input device includes a sensing component for sensing the movement of a user's head, which includes any motion in space, including rotation and translation. The movement of the user's head sensed by the sensing component is an incremental motion vector corresponding to the degree of freedom, and the rotation vector in the incremental motion vector can be used as the aforementioned first target rotation vector.

[0157] In some embodiments, the sensing component includes one or more sets of sensors, each set of sensors including one or more sensors. For example, the sensing component includes a first set of sensors for sensing the movement of the user's head in a first attitude degree of freedom, such as yaw, where yaw corresponds to left-right rotation of the user's head. For example, the sensing component includes a second set of sensors for sensing the movement of the user's head in a second attitude degree of freedom, such as pitch, where pitch corresponds to up-down rotation of the user's head. For example, the sensing component includes a third set of sensors for sensing the movement of the user's head in a first position degree of freedom, such as depth, where the first position degree of freedom is forward-backward movement of the user's head.

[0158] For example, the sensors in the first, second, and third groups of sensors can be of the same type, or they can be of completely different types or partially different types. These types of sensors can be selected from force sensors, deformation sensors, and / or distance sensors, etc. For example, the force sensor can be further selected from pressure sensors, torque sensors, etc. For example, the distance sensor can be further selected from optical sensors such as infrared sensors, etc. The movement of the user's head can be monitored by sensing the force applied by the user's head and / or the distance moved.

[0159] In one embodiment, the sensors in the first, second, and third groups of sensors can all be force sensors.

[0160] For the aforementioned first and / or second degree of freedom of attitude, the corresponding set of sensors (the first and / or second set of sensors) includes at least two force sensors, such as pressure sensors, disposed at both ends of the corresponding attitude direction. The direction of movement of the user's head can be determined by the force difference, such as the pressure difference, sensed by the force sensors at both ends of the corresponding set of sensors. For example, in the yaw degree of freedom corresponding to the left and right rotation of the user's head, if the pressure on the left-end force sensor is greater than the pressure on the right-end force sensor, the direction of movement of the user's head is determined to be to the left. Furthermore, the cumulative time of the force difference can be recorded, and the rotation vector of the first and / or second degree of freedom of attitude can be determined based on the direction of movement and the cumulative time.

[0161] The control device compares the force difference with a force difference threshold. When the force difference exceeds the threshold, it indicates that the user intends to adjust the imaging device, and the direction of the user's head movement is determined based on the force difference. Conversely, when the force difference is less than the threshold, it indicates that the user does not intend to adjust the imaging device, and the direction of the user's head movement does not need to be determined. By judging the force difference against the force difference threshold, the sensitivity of the imaging device and other responses can be reduced, thereby preventing accidental touches. Furthermore, the control device records the cumulative time for the force difference to exceed the force difference threshold and determines the rotation vector for that first and / or second degree of freedom based on the direction of movement, speed of movement, and cumulative time.

[0162] In one example, the motion speed can be configured as a unit speed without considering other factors.

[0163] In one example, the movement speed of the imaging device can also be determined based on the magnitude of the force difference. For instance, when the force difference exceeds a force difference threshold, the movement speed can be determined by the ratio of the force difference to the threshold, combined with a unit speed. For example, the movement speed can be obtained by multiplying the ratio by the unit speed, resulting in a relatively linear movement speed. Alternatively, the movement speed can be determined by the degree of difference between the force difference and the force difference threshold, combined with a unit speed. For example, when the difference is within a first level, the movement speed is determined by the sum of the unit speed and a first acceleration rate; when the difference is within a second level, the movement speed is determined by the sum of the unit speed and a second acceleration rate, resulting in a stepped movement speed. Preferably, the movement speed has a maximum value to ensure safety. When the movement speed reaches its maximum value through linear adjustment, stepped adjustment, or other methods, the maximum value can be configured.

[0164] For the aforementioned first degree of freedom of position, the corresponding set of sensors (the third set of sensors) includes at least one force sensor disposed at one end of that degree of freedom of position. For example, the force sensor can be a single sensor, which is typically disposed on the front of the user's head, i.e., on the side of the face. During normal operation, the user's head is usually in contact with or at least acts on the force sensor. When the force sensor senses a force (e.g., the force is not zero), if the force is between a first force threshold and a second force threshold less than the first force threshold, it indicates that the user is normally in contact with the force sensor and does not intend to adjust the imaging device. When the force sensor senses a force greater than the first force threshold or less than the second threshold, it indicates that the user intends to adjust the imaging device. Wherein, if the force is greater than the first force threshold, it is determined that the direction of movement of the user's head is in the forward direction, and if the force is less than the second force threshold, it is determined that the direction of movement of the user's head is in the reverse direction. For example, when the direction of movement of the user's head is in the reverse direction, the force is between 0 and the second force threshold. For example, a first threshold can be configured to 6N and a second threshold to 4N, providing comfortable pressure sensitivity for user operation. When the force sensed by the force sensor is between 4N and 6N (inclusive), no intention to adjust the imaging device is expected or generated, i.e., there is no need to determine the direction of user head movement. When the force sensed by the force sensor is greater than 6N, it is determined that the user's head is moving in the forward direction; when the force sensed by the force sensor is less than 4N, it is determined that the user's head is moving in the reverse direction. The forward movement of the user's head is associated with the insertion of the imaging device along its axis in the depth direction, and the forward movement of the user's head is associated with the withdrawal of the imaging device along its axis in the depth direction. Further, the control device records the cumulative time when the force is greater than the first force threshold or less than the second force threshold, and can determine the motion vector along the axis of the imaging device in the first position based on the movement direction, movement speed, and cumulative time.

[0165] In the feed direction of the imaging device, i.e., the axial direction, in one example, the motion speed can be configured to a unit speed for achieving the desired effect, without considering other factors. In another example, the motion speed can be dynamically determined based on the unit speed and the relationship between the force and a first threshold and / or a second threshold. For example, when the force is greater than the first threshold, the motion speed of the imaging device is determined based on the ratio of the force to the first threshold, combined with the unit speed; for instance, the motion speed is obtained by multiplying the ratio by the unit speed, resulting in a relatively linear motion speed. Similarly, when the force is less than the second threshold, the motion speed of the imaging device is determined based on the ratio of the force to the second threshold, combined with the unit speed; for instance, the motion speed is obtained by multiplying the ratio by the unit speed, also resulting in a relatively linear motion speed. Preferably, this motion speed also has a maximum value to ensure safety; when the motion speed is adjusted to reach a maximum value through one or more methods, this maximum value can be configured.

[0166] When the sensors in the first, second, and / or third groups of sensors include deformation sensors, the deformation sensors can be configured with reference to pressure sensors, and the deformation of the deformation sensors essentially originates from the force applied by the user's head. For example, deformation sensors can be set at both ends of the corresponding attitude degree of freedom, and a deformation sensor can be set at the corresponding position degree of freedom. Furthermore, for example, in the attitude degree of freedom, the direction of movement of the user's head can be determined based on the difference in deformation sensed by the deformation sensors at both ends and the difference in deformation threshold. Then, based on the cumulative time recorded when the difference in deformation exceeds the difference in deformation threshold, the motion vector in the corresponding degree of freedom can be determined using the direction of movement, the motion speed, and the cumulative time. This motion speed can also be configured as a unit speed or can be dynamically adjusted linearly or stepwise based on a unit speed and related factors. Simultaneously, a maximum value can be set for the motion speed; when the dynamically adjusted motion speed exceeds the maximum value, the motion speed is determined to be the maximum value.

[0167] For example, the handling of positional degrees of freedom is simpler. When the deformation is between a first deformation threshold and a second deformation threshold less than the first, it indicates the user is normally attached to the deformation sensor and has no intention of adjusting the imaging device. Conversely, when the deformation is greater than the first threshold or less than the second threshold, it indicates the user intends to adjust the imaging device. Specifically, if the deformation is greater than the first threshold, the user's head movement is determined to be in the forward direction; if the deformation is less than the second threshold, the user's head movement is determined to be in the reverse direction. During normal user operation, the sensed deformation is usually not zero. For example, when the user's head movement is in the reverse direction, the deformation is between 0 and the second deformation threshold. The amount of motion can also be determined by relating it to unit speed and cumulative time; the speed should also have a maximum value, which will not be repeated here.

[0168] When the sensors in the first, second, and / or third groups of sensors include distance sensors, the distance sensors can be configured, for example, with reference to pressure sensors. For instance, distance sensors can be placed at both ends of the corresponding attitude degree of freedom, and a distance sensor can be placed at the corresponding position degree of freedom. Furthermore, for example, in the attitude degree of freedom, the direction of movement of the user's head can be determined based on the distance difference sensed by the distance sensors at both ends and a distance difference threshold. Then, based on the cumulative time recorded when the distance difference exceeds the distance difference threshold, the motion vector in the corresponding degree of freedom can be determined using the movement direction, motion speed, and cumulative time. This motion speed can also be configured as a unit speed or dynamically adjusted linearly or stepwise based on a unit speed and related factors. Simultaneously, a maximum value can be set for the motion speed; when the dynamically adjusted motion speed exceeds the maximum value, the maximum value is determined.

[0169] For example, the handling of positional degrees of freedom is simpler. For instance, when the distance measurement is between a first distance threshold and a second distance threshold less than the first distance threshold, it indicates that the user's distance from the distance sensor is normal and they do not intend to adjust the imaging device. Conversely, when the distance measurement is greater than the first distance threshold or less than the second distance threshold, it indicates that the user intends to adjust the imaging device. Specifically, if the distance measurement is greater than the first distance threshold, the user's head movement direction is determined to be forward; if the distance measurement is less than the second distance threshold, the user's head movement direction is determined to be backward. During normal user operation, the sensed distance measurement is usually not zero. For example, when the user's head movement direction is backward, the distance measurement is between 0 and the second distance threshold. The amount of motion can also be determined by relating it to unit speed and cumulative time; the speed of motion should also have a maximum value, which will not be repeated here.

[0170] In the above embodiments, the motion vector can be used as the target movement information of the imaging device to be fed along its axis. After the target movement information is obtained, the control device controls the manipulator to manipulate the imaging device to feed along its axis according to the target movement information.

[0171] In some embodiments, please refer to Figure 16 and Figure 20 The input device includes one or more beacons 61 and one or more detectors for detecting the position of the one or more beacons 61 in space. The beacons 61 may optionally include active or passive beacons, and may optionally include coils, metal sheets, or magnets. The detectors may optionally include detectors that emit magnetic fields, electric fields, infrared radiation, etc. In some embodiments, the beacons 61 may be configured in a wearable device, such as a hat 63, a mask 64, or accessories that are easy for the user to wear on their head, such as glasses, earrings, hair clips, or stickers. By detecting the positional changes of one or more beacons 61 in the wearable devices 63 and / or 65 at different times, such as detecting the positional changes of one, two, three, or four beacons 61, the movement of the user's head can be determined. Wearable devices configured with multiple beacons 61 allow for posture monitoring of almost the entire head of the user, rather than being limited to a localized area such as the face or forehead, thus resulting in higher sensitivity and accuracy, and fewer false triggers caused by unconscious movements such as relaxing the head.

[0172] In some embodiments, when there are three or more beacons, the different beacons 61 can form at least two straight lines, meaning they do not necessarily need to be set on the same straight line. In some embodiments, when there are four or more beacons 61, the different beacons 61 can form at least two planes, meaning they do not necessarily need to be set on the same plane. This is beneficial for achieving attitude positioning.

[0173] Depending on the requirements, the motion information associated with the user's head movement can be configured as control information to control the movement of any part of any medical device, including imaging devices and surgical instruments. For example, the motion information can be configured as control information to control the movement of the imaging device around the distal fixed point, or it can be configured as control information to control the movement of the imaging device around the distal end point. Of course, there are other possibilities as well, which will not be listed here.

[0174] In some embodiments, the detector includes a magnetic field generator for generating a magnetic field within a space. Correspondingly, the beacon includes a magnetic sensor. By sensing changes in the magnetic field strength within the magnetic field, the positional changes of the magnetic sensor at different times within the magnetic field can be detected, thereby determining the movement of the user's head. In some embodiments, when multiple beacons, such as magnetic sensors, are included, identification identifiers can be configured for beacons worn by the same user. Based on the positional changes sensed by the detector, such as the magnetic field generator, for beacons with the same identification identifier, the movement of a specific user's head can be accurately determined.

[0175] In some embodiments, the detector includes a base station capable of receiving and / or transmitting wireless communication signals, such as a transceiver; the beacon includes a beacon that transmits and / or receives wireless communication signals, such as including a transceiver. The transceiver may include one or more transceivers supporting Bluetooth, 2G, 3G, 4G, 5G, infrared, WiFi, Zigbee, etc. The more base stations, the more accurate the positioning. By obtaining the distance between the beacon and different base stations at different times, the change in the beacon's position in space can be determined, and thus the movement of the user's head can be determined.

[0176] In some embodiments, such as Figure 14 As shown, the input device includes an eye-tracking device, primarily used for the automatic control of the pitch and yaw degrees of freedom of medical devices such as imaging instruments. The roll and depth-of-axis movements of the imaging instruments can be controlled by the user's hands or feet, such as using the feet to control foot pedals or the hands to control buttons. The eye-tracking device includes one or more camera devices 303, which are associated with the observation components in the imaging host, for example, positioned around the display unit 302 of the observation components. These cameras track the user's pupils to acquire the area of ​​the surgical field of view that the user's eyes 301 are fixating on. The control device then responds to the acquired fixation area by controlling the imaging instrument to move in the direction associated with that area to expand the field of view.

[0177] For example, Figure 15A schematic diagram of an imaging host screen is shown, where the surgical area 401 can be configured as a rectangular region. A series of peripheral regions can be set around the surgical area, such as above 402, below 403, to the left 404, and to the right 405. When the user's gaze falls on the edge of the surgical area above 402 or below 403, it can be assumed that the user desires an expansion of the surgical field of view upwards or downwards. Accordingly, the surgical robot will adjust the pitch degree of freedom of the imaging instrument, or possibly adjust both the pitch degree of freedom and the axial depth simultaneously. When the user's gaze falls on the edge of the surgical area to the left 404 or to the right 405, it can be assumed that the user desires an expansion of the surgical field of view to the left or right. Accordingly, the surgical robot will adjust the yaw degree of freedom of the endoscope, or possibly adjust both the yaw degree of freedom and the axial depth simultaneously.

[0178] In some embodiments, the imaging device may only move towards a specific area after the duration of the user's gaze continuously fixed on that area reaches a threshold time. For example, if the threshold time is 2 seconds, the device may only move in the corresponding direction after the continuous gaze lasts for 2 seconds, thus preventing accidental triggering of the imaging device adjustment. Of course, the duration and sensitivity of this time can also be parameterized and set accordingly. For example, to facilitate user configuration, a configuration interface can be generated that includes one or more configuration parameters for the duration and / or sensitivity.

[0179] Through the above methods, the user can obtain, for example, the first target rotation vector that the imaging device is expected to rotate around the distal end, and / or the target movement information fed along its axis.

[0180] In some embodiments, such as Figure 16 As shown, the surgical robot also includes an image host 500. The image host includes an observation component 501 and a posture adjustment component 502. The posture adjustment component 502 is used to adjust the posture of the observation component 501, which is used to observe images acquired by the imaging instruments. A control device is coupled to the observation component 501 and the posture adjustment component 502 and is configured to:

[0181] The attitude adjustment component moves according to the first target rotation vector, allowing the observation component to follow the user's head movement within the target orientation's degrees of freedom. This ensures that the observation component automatically adjusts itself when the user's head posture changes. Especially when the imaging device's movement is correlated with this first target rotation vector, the imaging device's movement is always synchronized with the observation component's movement, which in turn is always correlated with the user's head movement. This matched movement, including movements at essentially the same speed, guarantees that the user's eyes always see the desired image as their head moves, and that the user's eyes remain relatively constant relative to the image center.

[0182] In some embodiments, see further reference. Figure 16 The attitude adjustment component 502 includes a base 503, a first pivot 504, and a second pivot 505. The base 503 can be mounted on any fixed or movable object or structure. For example, the base 503 can be mounted on a wall or ceiling. When the base 503 is mounted on a fixed object or structure, the image host 500 has good overall rigidity and is not prone to vibration during movement, thus maintaining the stability of the image for the user to view. Alternatively, the base 503 can be mounted at the distal end of a robotic arm with one or more degrees of freedom, allowing the motion performance of the image host to be extended by utilizing the degrees of freedom of the robotic arm. Please refer to [link to relevant documentation]. Figure 4 The image host 500 can be integrated into the doctor's main control console 103. For example, the image host 500 can be housed in the operating space 510 of the doctor's main control console 103. The base 503 in the posture adjustment component 502 is relatively fixed in the operating space 510. The observation component 501 can move relatively freely in the operating space 510 by means of the motion performance of the first pivot 504 and the second pivot 505 in the posture adjustment component 502.

[0183] Furthermore, the first pivot member 504 is pivotally connected to the base 503 and is configured to rotate relative to the base 503 in a first degree of freedom. The second pivot member 505 is pivotally connected to the first pivot member 504 and is configured to rotate relative to the first pivot member 504 in a second degree of freedom. The observation assembly 501 is fixedly connected to the second pivot member 505. Therefore, the observation assembly 501 can be adjusted in attitude in at least one of the first and second degrees of freedom.

[0184] The image host 500 also includes a first drive mechanism and a second drive mechanism coupled to a control device. The first drive mechanism is configured to drive a first pivot 504 to rotate relative to a base 503 in a first degree of freedom, and the second drive mechanism is configured to drive a second pivot 505 to rotate relative to the first pivot 504 in a second degree of freedom.

[0185] In some embodiments, see Figure 17 The base 503 includes a first curved slide rail 5031, the first pivot 504 includes a second curved slide rail 5041 and a third curved slide rail 5042, and the second pivot 505 includes a fourth curved slide rail 5051. The first curved slide rail 5031 and the second curved slide rail 5041 cooperate to rotate in a first degree of freedom, and the third curved slide rail 5042 and the fourth curved slide rail 5051 cooperate to rotate in a second degree of freedom. The second curved slide rail 5041 and the third curved slide rail 5042 are typically located on opposite sides of the first pivot 504, one side for sliding engagement with the base 503 and the other side for sliding engagement with the second pivot 505.

[0186] Among them, combined Figure 18 Referring to the description, the observation component 501 includes a display unit, which may be a 2D or 3D display unit. The display unit is coupled to the control device and is relatively fixedly disposed on the second pivot 505. The observation component 501 may also include an observation window 5011, also called a binocular window, which is relatively fixedly disposed with the display unit and used to observe the image displayed by the display unit. In some embodiments, a cushion 5012 may be provided above the observation window 5011 for the user's forehead to rest against, thereby improving the user's forehead comfort when viewing the image displayed by the display unit with their eyes close to the observation window 5011. One or more of the above-mentioned sensors may be disposed inside the cushion 5012 or outside the cushion 5012.

[0187] In some embodiments, such as Figure 19 As shown, the base 503' in the attitude adjustment assembly 502' includes a first curved slide rail 5031', and the first pivot 504' includes a second curved slide rail. The first curved slide rail 5031' and the second curved slide rail cooperate to rotate in a first attitude degree of freedom. The attitude adjustment assembly 502' also includes a rotary joint 506'. The second pivot 505' and the first pivot 504' are connected by the rotary joint 506' to rotate in a second attitude degree of freedom.

[0188] The above are as follows Figure 17 Or, as shown in 19, the curved slide rail is an example of an arc-shaped curved slide rail. For structures that achieve rotation via curved slide rails, for example in... Figure 17 In this configuration, for the base 503 and the first pivot 504, the first drive mechanism can be, for example, a planar four-bar linkage, a crank-slider mechanism, etc., to drive the first pivot 504 to rotate relative to the base 503 on the curved slide rail 5031. For structures that achieve rotation through a rotary joint 506', for example in... Figure 19In this context, the implementation of the second drive mechanism is simpler for the second pivot member 505' and the first pivot member 504'. For example, the second drive mechanism can adopt a gear meshing mechanism, a pulley mechanism, etc., to drive the second pivot member 505' to rotate relative to the first pivot member by driving the rotation joint 506' to rotate.

[0189] In some embodiments, see further reference. Figure 18 The second pivot 505 includes a base plate 5051 and sidewalls 5052 extending from both sides of the base plate 5051 away from the base 503. The observation component 501 is disposed on the base plate 5051. When viewing an image, the user's head can be accommodated within the semi-enclosed space formed by the sidewalls 5052 and the base plate 5051, allowing for flexible rotation in the up, down, left, and right directions, as well as forward and backward movement. Sensors and / or detectors for monitoring user head movement can be disposed on the posture adjustment component 502 and / or the observation component 501. For example, a set of sensors 507 for monitoring the user's head rotation in the vertical direction can be disposed on the observation assembly 501, for example, disposed on the substrate 5051 and located on the upper and lower sides of the observation window 5011; a set of sensors for monitoring the user's head rotation in the horizontal direction can be disposed on the second pivot 505, for example, disposed on the two opposite sidewalls 5052 of the second pivot 505; a set of sensors for monitoring the user's head movement in the depth direction can be disposed on the observation assembly 501, for example, integrated inside the cushion 5012. In some embodiments, when the input device uses a detector to monitor the movement of the user's head, the detector can also be disposed in the same or different positions as the multiple sets of sensors such as 507 and 508 described above.

[0190] In some embodiments, the first attitude degree of freedom of the first pivot 504 (504') includes one of the yaw and pitch degrees of freedom associated with the imaging instrument, and the second attitude degree of freedom of the second pivot 505 (505') includes the other of the yaw and pitch degrees of freedom associated with the imaging instrument. For example, the first attitude degree of freedom includes the yaw degree of freedom associated with the imaging instrument, and the second attitude degree of freedom includes the pitch degree of freedom associated with the imaging instrument.

[0191] Furthermore, the yaw degree of freedom of the imaging device is related to one of the user's left-right rotation and up-down rotation movements, and the pitch degree of freedom of the imaging device is related to the other of the user's head rotation. For example, the yaw degree of freedom of the imaging device is related to the user's head rotation, and the pitch degree of freedom of the imaging device is related to the user's head up-down rotation. Such a relationship is intuitive to operate and can provide a better user experience.

[0192] In some embodiments, the axis of rotation of each pivot is substantially the same as the axis of rotation of the associated user head, for example, see [continued] Figure 16 When the first pivot rotates in the yaw degree of freedom, its rotation axis J20 is basically the same as the rotation axis of the user's head (i.e., neck) for left and right rotation. When the second pivot rotates in the pitch degree of freedom, its rotation axis is basically the same as the rotation axis J30 of the user's head (i.e., neck) for up and down rotation. This design ensures that the motion trajectory of the observation component is basically the same as the natural motion trajectory of the user's head, and also guarantees user comfort.

[0193] In some embodiments, the range of motion of the doctor's head in the left and right rotation directions is typically around ±75°. Therefore, the range of motion of the pivot member providing the corresponding degree of posture freedom for this left and right rotation can be configured between -75° and +75°, that is, it can rotate 75° to the left and right respectively with respect to the center position of this degree of posture freedom. Preferably, the more comfortable range of motion for the doctor's head when turning left and right is typically around ±60°. Therefore, the range of motion of the pivot member providing the corresponding degree of posture freedom for this left and right rotation can be configured between -60° and +60°, for example, -45° to +45°. For example, the pivot member is... Figure 17 or Figure 19 The first pivot shown.

[0194] In some embodiments, the range of motion of the doctor's head in the vertical rotation direction is typically around ±45°. Therefore, the range of motion of the pivot member providing the postural degree of freedom corresponding to this vertical rotation can be configured between -45° and +45°, that is, it can rotate 45° upwards and downwards respectively relative to the center position of this postural degree of freedom. Preferably, the more comfortable range of motion for the doctor's head when rotating vertically is typically around ±30°. Therefore, the range of motion of the pivot member providing the postural degree of freedom corresponding to this horizontal rotation can be configured between -30° and +30°, for example, -25° to +25°. For example, the pivot member is... Figure 17 or Figure 19 The second pivot shown.

[0195] In some embodiments, the more specific procedures for a physician to control medical devices, such as imaging devices, include:

[0196] Step 401: Adjust the pose of the image host.

[0197] This includes the user adjusting the height, depth, and tilt angle of the imaging host to achieve a more comfortable surgical posture for observation. Different users may have different operating habits, and the position of the observation components can be automatically adjusted based on the operating habits corresponding to the acquired user identification. During the surgery, whether or not step 401 is performed is determined by the user.

[0198] Step 402: Obtain and record the initial position and / or orientation of the image host.

[0199] The initial position and / or orientation refers to the position and / or orientation of the image host at the moment the input device and the manipulator assembly establish a master-slave mapping relationship. This initial position and / or orientation can be abbreviated as the initial pose, or simply the pose zero position. The initial position and / or orientation includes, as described above, the position and / or orientation of the first pivot relative to the base, and the position and / or orientation of the second pivot relative to the first pivot. For example, the pose zero position only needs to be recorded once when the user first enters the surgical area.

[0200] Step 403: Monitor in real time whether the conditions for starting the surgery are met.

[0201] The conditions for initiating surgery include, but are not limited to, one or more of the following: detecting the user's head approaching the imaging host, receiving a master-slave activation command, or receiving a command to establish a mapping relationship between the input device and the imaging instrument. For example, the user's head approaching the imaging host can be confirmed using sensors as mentioned above, through force, deformation, distance, etc. For example, establishing a mapping relationship between the input device and the imaging instrument can be triggered by pressing a specific button, outputting a specific voice, or stepping on a specific foot pedal. Simultaneously, a mapping relationship between the input device and the imaging instrument is established with the imaging host.

[0202] Step 404: When the conditions for starting surgery are met, determine the movement of the user's head.

[0203] The movement of the user's head includes changes in the position and / or posture of the user's head, including sensing the movement of the user's head using an input device as described in any of the above embodiments. For example, changes in the position and / or posture of the user's head can be determined by determining changes in the position and / or posture of a wearable device configured with a beacon.

[0204] Step 405: Determine whether the user's head movement is intentional.

[0205] The movement of the user's head can be intentional or unintentional. Unintentional movement is generally considered a malfunction and needs to be excluded when controlling the medical device. For example, the initial position and / or posture of the user's head can be established in conjunction with the imaging host and / or imaging device. After the surgery begins, the current position and / or posture of the user's head can be compared with the initial position and / or posture to determine whether the movement of the user's head is intentional. For example, if the change between the current position and / or posture and the initial position and / or posture exceeds a set threshold, it is considered an intentional movement.

[0206] Step 406: When it is determined that the movement of the user's head is intentional, the image host and imaging device are adjusted in response to the movement of the user's head.

[0207] In this way, the movement of the user's head, the image host, and the imaging device is synchronized. Especially when adjusting the imaging device, if the movement of the user's head is related to the rotation of the imaging device around different rotation centers, in order to ensure the consistency of the imaging effect, the scheme described in the above embodiment can be used to compensate for the imaging distance deviation, which will not be repeated here.

[0208] Accordingly, in step 406, the image displayed in the observation component will also change, and the change in the image will be fed back to the user, thereby supporting the user to decide whether further adjustments are needed. If so, the user's head will move further in the desired direction; otherwise, the user's head does not need to move further.

[0209] In some embodiments, the process further includes step 407, whereby, after the surgery begins, the control device monitors in real time whether the surgery interruption conditions are met. These surgical interruption conditions include, but are not limited to, one or more of the following: detecting that the user's head has left the imaging host, the user's hand has left the operating unit, or receiving an instruction to disconnect the mapping relationship between the input device and the imaging instrument.

[0210] The process also includes step 408, which, when the surgical interruption conditions are met, disconnects the mapping relationship between the input device and the imaging instrument and the image host, and locks the position and / or orientation of the imaging instrument and the image host.

[0211] In some embodiments, the process further includes step 409, which involves real-time monitoring to determine if an image host clutch command is received. The image host clutch command can be input via methods such as outputting a specific voice message, pressing a specific button, or stepping on a specific foot pedal. The process also includes step 410, which, upon receiving the image host clutch command, controls the image host to return to its initial position and / or posture (i.e., return to zero) based on a pre-recorded initial position and / or posture. During the process of controlling the image host to return to zero, the imaging device remains locked, and the field of view does not change accordingly. Steps 409 and 410 allow for a wide range of adjustments to the imaging device, facilitating surgical procedures for the user. In other embodiments, if the adjustment of the imaging device reaches a limit, the control device can interrupt the association between the user's head movement and the movement of the imaging device, or it can lock the position and / or posture of the image host.

[0212] In some embodiments, if the user wishes to restart the surgery, steps 404 to 406 above can be repeated.

[0213] In some embodiments, to enable the observation component to move with the user's head, a method can be adopted that eliminates the need for an additional driving mechanism. For example, a head-mounted display device can be provided that, once worn on a user's head, moves naturally with the user's head.

[0214] Furthermore, to achieve control of medical devices, such as imaging devices, as described in any of the above embodiments, an input device for inputting control commands for the medical device, such as the imaging device, can be integrated into the head-mounted display device. This control command is, for example, the aforementioned command to rotate a target rotation vector around a distal end point of the imaging device. In some embodiments, the input device includes one or more sensors for sensing movement of the user's head. For example, these sensors include inertial sensors, specifically one or more of accelerometers, gyroscopes, and inertial measurement units (IMUs), which monitor the movement of the user's head by monitoring the pose and motion of the head-mounted display device. These sensors can be relatively evenly distributed throughout the head-mounted display device, facilitating mutual calibration between multiple sensors to improve measurement accuracy.

[0215] In some embodiments, the head-mounted display device includes a display module and an adjustment mechanism. The display module includes two display units for viewing by the left and right eyes respectively, and the adjustment mechanism is used to adjust the horizontal distance between the two display units to accommodate different users' interpupillary distances.

[0216] In some embodiments, the head-mounted display device may further include one or more sensors for sensing whether a user is wearing the head-mounted display device, such as proximity sensors, deformation sensors, pressure sensors, etc. In some embodiments, the head-mounted display device may further include a communication unit coupled to the surgical system, such as a control device, via wired and / or wireless means.

[0217] In some embodiments, the display unit includes a display screen and a lens group. The distance between the lens group and the display screen is adjustable to accommodate the refractive errors of different users, allowing users with refractive errors to use the head-mounted display normally without wearing glasses or other devices. The display screen can be configured to adjust the image according to the state of the lens group, so that the image, after correction by the lens group, forms a normal image in the human eye.

[0218] In some embodiments, the display unit may further include one or more cameras, which may be visible light cameras or infrared cameras, primarily for eye-tracking functions to determine the area the user is looking at, or simply to determine whether the user is looking at the screen, at least to determine whether the user is wearing the head-mounted display device. Surgical procedures are only permitted to begin after it is determined that the user is wearing the head-mounted display device and other surgical initiation conditions are met. If the surgical initiation conditions are not met, the surgical procedure is interrupted, for example, by disconnecting the mapping relationship between the head-mounted display device and the imaging instruments.

[0219] In some embodiments, the use of this head-mounted display device is suitable for controlling the pitch, yaw, and feed degrees of freedom of the imaging device, and can also be used to control the roll degree of freedom of the imaging device. Specifically, the rotation of the user's head around the longitudinal axis of the neck corresponds to the yaw degree of freedom of the imaging device, the rotation of the user's head around the transverse axis of the neck corresponds to the pitch degree of freedom of the imaging device, and the rotation of the user's head away from the longitudinal axis of the neck corresponds to the roll degree of freedom of the imaging device.

[0220] In one embodiment, this application also provides a computer-readable storage medium storing a computer program configured to be loaded by a processor and executed to implement the control method as described in any of the above embodiments.

[0221] In one embodiment, this application also provides a control device for a surgical robot. For example... Figure 21 As shown, the control device may include: a processor 501, a communications interface 502, a memory 503, and a communications bus 504.

[0222] The processor 501, communication interface 502, and memory 503 communicate with each other through the communication bus 504.

[0223] The communication interface 502 is used to communicate with other network elements such as various sensors, motors, solenoid valves, or other clients or servers.

[0224] The processor 501 is used to execute program 505, which can specifically execute the relevant steps in the above method embodiments.

[0225] Specifically, program 505 may include program code that includes computer operation instructions.

[0226] The processor 505 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), one or more integrated circuits configured to implement the embodiments of this application, or a graphics processing unit (GPU). The control device includes one or more processors, which may be processors of the same type, such as one or more CPUs, or one or more GPUs; or they may be processors of different types, such as one or more CPUs and one or more GPUs.

[0227] Memory 503 is used to store program 505. Memory 503 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0228] Specifically, program 505 can be used to cause processor 501 to execute the control method described in any of the above embodiments.

[0229] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A surgical robot, characterized in that, include: Imaging instruments, used to be inserted into body openings to acquire images; A manipulator for manipulating the imaging instrument to rotate about a telecentric fixed point located at the body opening and to feed it along the axial direction of the imaging instrument. An input device is used to input a first target rotation vector in the target attitude degree of freedom of the imaging instrument rotating around the distal end of the imaging instrument; The control device, coupled to the manipulator and the input device, is configured to: Obtain the first target rotation vector input by the input device; The target position and / or orientation of the imaging instrument in the reference coordinate system are determined based on the first target rotation vector. Obtain the configuration parameters of the imaging optical axis of the imaging instrument; Based on the configuration parameters of the imaging optical axis, a kinematic model relating to the manipulator, the imaging device, and the imaging optical axis is constructed. Based on the target position and / or posture and the kinematic model, determine the target joint variables of the joints in the manipulator and the imaging device; The joints of the manipulator and the imaging device are controlled according to the target joint variables so that the imaging center of the imaging device reaches the target imaging center.

2. The surgical robot according to claim 1, characterized in that, The configuration parameters include the length of the imaging optical axis and / or the angle of the imaging optical axis relative to the imaging surface of the imaging instrument.

3. The surgical robot according to claim 2, characterized in that, The control device is configured to: Obtain the imaging distance range of the imaging device; A configuration interface is generated based on the imaging distance range, including one or more selectable target imaging distances, wherein the target imaging distance is between the minimum and maximum imaging distances of the imaging distance range; In response to the selection of the target imaging distance through the configuration interface, the selected target imaging distance is configured as the length of the imaging optical axis.

4. The surgical robot according to claim 2, characterized in that, The control device is configured to: Generate a configuration interface that includes one or more selectable target angles, wherein the target angles are between 0° and 90°; In response to the selection of the target angle through the configuration interface, the selected target angle is configured as the angle of the imaging optical axis relative to the imaging surface.

5. The surgical robot according to claim 1, characterized in that, Determining the target position and / or orientation of the imaging instrument's target imaging center in the reference coordinate system based on the first target rotation vector includes: Determine the first position and / or orientation of the target imaging center relative to the distal end of the imaging device; Based on the current position and / or attitude of the distal end point in the reference coordinate system, determine the target position and / or attitude of the first position and / or attitude in the reference coordinate system.

6. The surgical robot according to claim 1, characterized in that, The movement of the user's head in the target pose degree of freedom is associated with the acquired first target rotation vector, and the surgical robot further includes: An image host includes an observation component and a posture adjustment component, wherein the posture adjustment component is used to adjust the posture of the observation component, and the observation component is used to observe the image acquired by the imaging device; The control device is coupled to the attitude adjustment assembly and is further configured to: The posture adjustment component is controlled to move according to the first target rotation vector, so that the observation component moves in the target posture degree of freedom following the movement of the user's head.

7. The surgical robot according to claim 6, characterized in that, The observation component is configured to move in sync with the movement of the imaging instrument.

8. The surgical robot according to claim 6, characterized in that, The attitude adjustment component includes: Base; A first pivot member, pivotally connected to the base, is configured to rotate relative to the base in a first orientation degree of freedom; and a second pivot member, pivotally connected to the first pivot member, configured to rotate relative to the first pivot member in a second attitude degree of freedom; The observation component is fixedly connected to the second pivot component.

9. The surgical robot according to claim 8, characterized in that, The base includes a first curved slide rail, the first pivot member includes a second curved slide rail and a third curved slide rail, and the second pivot member includes a fourth curved slide rail. The first and second curved slide rails are slidably engaged to allow rotation in a first degree of freedom, and the third and fourth curved slide rails are slidably engaged to allow rotation in a second degree of freedom; or... The base includes a first curved slide rail, the first pivot member includes a second curved slide rail, the first curved slide rail and the second curved slide rail are slidably engaged to be rotatable in a first degree of freedom of attitude, the attitude adjustment assembly also includes a rotary joint, the second pivot member and the first pivot member are connected through the rotary joint to be rotatable in a second degree of freedom of attitude.

10. The surgical robot according to claim 1, characterized in that, The input device includes one or more base stations capable of receiving and / or transmitting wireless signals, and also includes a wearable device that can be worn on a user's head, the wearable device including one or more beacons capable of transmitting and / or receiving wireless signals, and the first target rotation vector associated with the movement of the user's head can be determined based on sensing the distance between the one or more beacons and the one or more base stations.

Images