A method and related device for laparoscopic posture control by robot gripping

Abstract

This invention discloses a method and related apparatus for controlling the pose of a laparoscopic instrument held by a robot, comprising: calculating the pose matrix of the surgical instrument tip relative to the vision module based on the center coordinates of the beacon relative to the vision module and the distance from the center point of the beacon to the tip of the surgical instrument; converting the pose matrix of the surgical instrument tip relative to the vision module into a pose matrix of the surgical instrument tip relative to the robot base; converting the center coordinates of the laparoscopic trocar relative to the vision module into the center coordinates of the laparoscopic trocar relative to the robot base; calculating the desired pose of the laparoscopic relative to the robot base based on the center coordinates of the laparoscopic trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base; and controlling the robot to adjust the pose of the laparoscopic based on the desired pose of the laparoscopic relative to the robot base. The purpose of this invention is to solve the problem of the difficulty in accurately controlling the pose of a laparoscopic instrument.

Description

Technical Field

[0001] This invention belongs to the field of medical device control technology, specifically relating to a method and related device for controlling the posture of a laparoscopic device held by a robot. Background Technology

[0002] Laparoscopic surgery is an inevitable trend in the future development of surgical methods. Traditional laparoscopic surgery requires at least two surgeons: one to perform the surgical procedure and the other to assist by holding the laparoscope, ensuring the surgeon has a clear view of the lesion. A problem arises because differing or delayed understandings of the surgeon's intentions mean that the level of understanding and coordination between the laparoscopic surgeon and the operating surgeon is crucial to surgical quality. Furthermore, physical contact between the laparoscopic surgeon and the operating surgeon can interfere with the latter's fine motor skills. Additionally, maintaining a fixed posture for extended periods can lead to muscle fatigue and soreness for the laparoscopic surgeon, significantly depleting their energy. Therefore, controlling the laparoscopic position based on the operating surgeon's intraoperative adjustments to the laparoscopic field of view is vital for the development of laparoscopic surgical techniques.

[0003] Patent CN114366313A discloses a control method for a laparoscopic surgical instrument holding robot based on the pose of laparoscopic surgical instruments. This method detects the pose of surgical instruments by installing pose sensors on the surgical instruments. Since the sensors are installed on the surgical instruments, their measurement reference is the surgical instruments. However, since the surgical instruments are manipulated by the doctor, it is difficult to detect the absolute pose of the surgical instruments relative to the world coordinate system. Furthermore, the method of obtaining the pose of surgical instruments uses pose sensor integration, which leads to the defect of sensor error accumulation. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention provides a method and related device for controlling the pose of a laparoscopic instrument held by a robot, the purpose of which is to solve the problem of the difficulty in accurately controlling the pose of the laparoscopic instrument.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] According to a first aspect of the present invention, a method for controlling the pose of a laparoscopic instrument held by a robot is provided, comprising:

[0007] Acquire an image of the laparoscopic surgical work area captured by the vision module. The image of the laparoscopic surgical work area includes a laparoscopic trocar region and a beacon region set on the surgical instruments. The beacon region contains ArUco encoding.

[0008] Calculate the center coordinates of the laparoscopic trocar relative to the visual module based on the laparoscopic trocar region, and calculate the center coordinates of the beacon relative to the visual module based on the beacon region;

[0009] Based on the center coordinates of the beacon relative to the vision module and the set distance from the center point of the beacon to the tip of the surgical instrument, calculate the pose matrix of the tip of the surgical instrument relative to the vision module.

[0010] The pose matrix of the surgical instrument tip relative to the vision module is converted into the pose matrix of the surgical instrument tip relative to the robot base;

[0011] Convert the center coordinates of the laparoscopic trocar relative to the vision module to the center coordinates of the laparoscopic trocar relative to the robot base;

[0012] The desired pose of the laparoscope relative to the robot base is calculated based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base.

[0013] The robot is controlled to adjust the laparoscope's orientation based on the desired orientation of the laparoscope relative to the robot's base.

[0014] In one possible implementation of the first aspect, the laparoscopic surgical work area image further includes a surgical instrument trocar region, and the laparoscopic pose control method further includes:

[0015] Calculate the center coordinates of the surgical instrument trocar relative to the visual module based on the surgical instrument trocar region;

[0016] Calculate the distance between the beacon and the surgical instrument trocar based on the center coordinates of the surgical instrument trocar relative to the center of the visual module and the beacon relative to the center of the visual module;

[0017] The robot is controlled to adjust the depth of insertion of the laparoscope into the trocar based on the change in the distance between the beacon and the surgical instrument trocar.

[0018] Laparoscopic position control methods also include:

[0019] Acquire images of the robot's end effector captured by the vision module;

[0020] Calculate the center coordinates of the robot's end effector relative to the vision module based on the robot's end effector image;

[0021] Determine whether the robot's end effector is within a preset safe area relative to the center coordinates of the vision module. If it exceeds the preset safe area, control the robot to stop working.

[0022] In one possible implementation of the first aspect, the laparoscopic pose control method further includes:

[0023] Within a set time period, analyze whether the ArUco code in the beacon area changes periodically along the same rotation direction. If a periodic change occurs, control the laparoscopic lens to rotate along the same rotation direction until the ArUco code stops changing periodically.

[0024] In one possible implementation of the first aspect, calculating the center coordinates of the beacon relative to the visual module based on the beacon region specifically involves:

[0025] Convert the beacon region into a point cloud;

[0026] The point cloud is fitted based on the geometric features of the beacon to obtain the center point of the geometric object;

[0027] The beacon's attitude is obtained based on the ArUco encoding of the beacon region;

[0028] Based on the center point of the geometry and the beacon's pose, the center coordinates of the beacon relative to the visual module are obtained.

[0029] In one possible implementation of the first aspect, the pose matrix of the surgical instrument tip relative to the visual module is calculated based on the center coordinates of the beacon relative to the visual module and the distance from the set beacon center point to the tip of the surgical instrument, specifically using the following formula:

[0030]

[0031]

[0032]

[0033] In the formula, This represents the pose matrix of the surgical instrument tip relative to the visual module; The orientation of the surgical instrument tip relative to the visual module; The coordinates of the surgical instrument tip relative to the vision module; The pose of the beacon relative to the vision module; and These are the components of the pose matrix of the surgical instrument tip relative to the visual module coordinate system on the X, Y, and Z axes, respectively. M represents the center coordinates of the beacon relative to the vision module; M is the distance from the center point of the beacon to the tip of the surgical instrument.

[0034] In one possible implementation of the first aspect, converting the pose matrix of the surgical instrument tip relative to the vision module into a pose matrix of the surgical instrument tip relative to the robot base specifically involves:

[0035]

[0036] In the formula, The pose matrix of the surgical instrument tip relative to the robot base; This is the transformation matrix from the vision module to the robot base coordinate system obtained through hand-eye calibration; This represents the pose matrix of the surgical instrument tip relative to the visual module; The coordinates of the surgical instrument tip relative to the robot base; The orientation of the surgical instrument tip relative to the robot base;

[0037] The process of converting the center coordinates of the laparoscopic trocar relative to the vision module to the center coordinates of the laparoscopic trocar relative to the robot base specifically involves:

[0038]

[0039] In the formula, Here are the center coordinates of the laparoscopic trocar relative to the robot base; These are the coordinates of the laparoscopic trocar relative to the center of the visual module.

[0040] In one possible implementation of the first aspect, the desired pose of the laparoscope relative to the robot base is calculated based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base, using the following formula:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] In the formula, The pose matrix of the surgical instrument tip relative to the robot base; The coordinates of the surgical instrument tip relative to the robot base; The orientation of the surgical instrument tip relative to the robot base; The desired orientation of the laparoscope relative to the robot base; and These are the components of the laparoscope's attitude matrix relative to the robot's base coordinate system on the X, Y, and Z axes, respectively. Let be the center coordinates of the laparoscopic trocar relative to the robot base; vec{1,0,0} is a unit vector in a fixed direction in the world coordinate system.

[0047] In one possible implementation of the first aspect, the distance between the beacon and the surgical instrument trocar is calculated based on the center coordinates of the surgical instrument trocar relative to the center coordinates of the vision module and the beacon relative to the center coordinates of the vision module, using the following formula:

[0048]

[0049] In the formula, Dist is the distance between the beacon and the surgical instrument trocar; The coordinates of the surgical instrument trocar relative to the center of the vision module; These are the coordinates of the beacon relative to the center of the visual module.

[0050] Furthermore, the visual module is positioned on the left and right sides of the surgeon's operating space in the laparoscopic surgery area;

[0051] The beacon is a cube, and the four sides of the cube are provided with ArUco codes representing different ID values.

[0052] It should be noted that the preset safe area is defined by the surgeon by dragging the robot before the operation.

[0053] According to a second aspect of the present invention, a robot-held laparoscopic posture control device is provided, comprising:

[0054] The acquisition module is used to acquire an image of the laparoscopic surgical work area collected by the vision module. The image of the laparoscopic surgical work area includes a laparoscopic trocar region and a beacon region set on the surgical instruments. The beacon region contains ArUco encoding.

[0055] The first calculation module is used to calculate the center coordinates of the laparoscopic trocar relative to the visual module based on the laparoscopic trocar region, and to calculate the center coordinates of the beacon relative to the visual module based on the beacon region.

[0056] The second calculation module is used to calculate the pose matrix of the surgical instrument tip relative to the vision module based on the center coordinates of the beacon relative to the vision module and the set distance between the beacon center point and the tip of the surgical instrument.

[0057] The first conversion module is used to convert the pose matrix of the surgical instrument tip relative to the vision module into the pose matrix of the surgical instrument tip relative to the robot base.

[0058] The second conversion module is used to convert the center coordinates of the laparoscopic trocar relative to the vision module into the center coordinates of the laparoscopic trocar relative to the robot base;

[0059] The third calculation module is used to calculate the desired posture of the laparoscope relative to the robot base based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base.

[0060] The control module is used to control the robot to adjust the laparoscope's posture according to the desired posture of the laparoscope relative to the robot base.

[0061] According to a third aspect of the present invention, an apparatus is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the aforementioned laparoscopic pose control method for robot gripping.

[0062] Compared with existing technologies, this invention has at least the following advantages: It detects the pose of surgical instruments relative to the coordinate system of the vision module through three-dimensional vision measurement, uses only the vision module to capture images of the beacon under test, establishes the relative positional relationship between the beacon and the vision module, and thus obtains the three-dimensional information of the beacon. This process is performed without interfering with the surgical operation and is not limited by the scene, making it highly feasible. Furthermore, since the pose of the laparoscopy and the pose of the surgical instruments are not in a fixed offset relationship, expressing the pose of the laparoscopy through the pose of the surgical instruments becomes crucial for laparoscopic control. This invention uses a visual method to identify the pose of the surgical instruments relative to the vision module. The pose of the laparoscopy can be obtained through robot forward kinematics calculation. The relative pose of the tip of the surgical instrument and the laparoscopy can be obtained through spatial coordinate transformation of the vision system and the robot. The pose of the tip of the surgical instrument guides the robot to adjust the pose of the laparoscopy. Because the pose of the tip of the surgical instrument is entirely under the control of the surgeon, the robot can perform precise laparoscopic pose control accordingly. Furthermore, this invention acquires the pose of the surgical instruments relative to the robot base in real time throughout the entire surgical procedure, and the results of each two visual inspections are independent of each other, avoiding the problem of uncertainty error accumulation caused by random external interference.

[0063] Furthermore, this invention calculates the center coordinates of the surgical instrument trocar relative to the vision module based on the trocar region; calculates the distance between the beacon and the surgical instrument trocar based on the center coordinates of the surgical instrument trocar and the beacon relative to the vision module; and controls the robot to adjust the depth of the laparoscopic insertion into the laparoscopic trocar based on the change in the distance between the beacon and the surgical instrument trocar. In this way, the decoupling of laparoscopic depth control and posture control is achieved, improving the degree of freedom of operation and enabling the operator to convey their intentions more accurately.

[0064] Furthermore, this invention analyzes whether the ArUco code of the beacon area undergoes periodic changes along the same rotation direction within a set time period. If periodic changes occur, the laparoscopic lens is controlled to rotate along the same rotation direction until the ArUco code stops changing periodically. This method enables adjustment of the laparoscopic lens orientation, increases the operational dimension, and allows for individual control of the lens without changing the laparoscope's position, further improving operational comfort.

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the specific embodiments of the present invention, the drawings used in the description of the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 This is a flowchart of a robot-held laparoscopic posture control method according to the present invention;

[0068] Figure 2 A schematic diagram of pose control for a laparoscopic device held by a robot.

[0069] Figure 3 In the image, (a) is a three-dimensional view of an ArUco coded beacon placed on a surgical instrument, and (b) is a planar unfolded view of the three-dimensional ArUco coded beacon.

[0070] In the diagram: 1-Laparoscope; 2-Surgical instruments; 3-Laparoscope trocar; 4-Surgical instrument trocar; 5-Beacon. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Combination Figure 1 , Figure 2 and Figure 3 As shown in the figure, this application provides a method for controlling the pose of a laparoscopic instrument held by a robot, which specifically includes the following steps:

[0073] S101. Acquire an image of the laparoscopic surgical work area collected by the vision module. The laparoscopic surgical work area image includes a laparoscopic trocar region and a beacon region set on the surgical instruments. The beacon region contains ArUco encoding.

[0074] For example, the vision module is a depth camera with distance perception capability, that is, a depth camera that can detect spatial distance information during shooting.

[0075] Preferably, two cameras are provided in the laparoscopic surgery work area. The cameras are fixed on the left and right sides of the surgeon's operating space by mounting brackets, and the field of view of each camera completely covers the surgeon's operating space on one side.

[0076] Preferably, beacon 5 is a cube with four sides printed with ArUco code marks representing different ID values, the top and bottom of the cube are blank, and it is directly mounted on the rod of surgical instrument 2 during use.

[0077] It's important to note that ArUco encoding is a type of QR code-based visual marker. The markers are square with alternating black and white stripes along the edges, forming a binary code. ArUco markers are characterized by: ① the ability to recognize markers at different angles and distances, and the ability to recognize multiple markers simultaneously. ② The black border of the marker facilitates rapid detection in an image, while the internal binary code is used for marker recognition and error detection and correction. ③ The encoding method is based on Hamming codes, enabling error detection and correction. Furthermore, the ArUco marker encoding method can be modified and extended as needed to adapt to different application scenarios. ArUco markers are widely used to increase the amount of information when mapping from the 2D world to the 3D world, facilitating the discovery of projection relationships between the two worlds, thereby enabling applications such as pose estimation and camera correction. In the OpenCV library, ArUco marker detection and pose estimation can be performed by creating and recognizing ArUco markers using specific predefined dictionaries.

[0078] S102. Calculate the center coordinates of the laparoscopic trocar relative to the visual module based on the laparoscopic trocar region, and calculate the center coordinates of the beacon relative to the visual module based on the beacon region.

[0079] Preferably, the center coordinates of the beacon relative to the visual module are calculated based on the beacon area, as follows:

[0080] First, the beacon region is converted into a point cloud. Then, the point cloud is fitted with the geometric features of the beacon to obtain the center point of the geometric body. Next, the beacon pose is obtained based on the ArUco encoding of the beacon region. Finally, the center coordinates of the beacon relative to the visual module are obtained based on the center point of the geometric body and the beacon pose.

[0081] In other words, the beacon has a distinct geometric feature (a regular hexahedron). By fitting the geometric feature, the positioning result of ArUco encoding can be corrected, enabling real-time high-precision positioning of features on the instrument during surgery, and obtaining the attitude determined relative to the world coordinate system.

[0082] S103. Based on the center coordinates of the beacon relative to the vision module and the set distance from the beacon center point to the tip of the surgical instrument, calculate the pose matrix of the tip of the surgical instrument relative to the vision module. The specific formula is as follows:

[0083]

[0084]

[0085]

[0086] In the formula, This represents the pose matrix of the surgical instrument tip relative to the visual module; The orientation of the surgical instrument tip relative to the visual module; The coordinates of the surgical instrument tip relative to the vision module; The pose of the beacon relative to the vision module; and These are the components of the pose matrix of the surgical instrument tip relative to the visual module coordinate system on the X, Y, and Z axes, respectively. M represents the center coordinates of the beacon relative to the vision module; M is the distance from the center point of the beacon to the tip of the surgical instrument.

[0087] It should be understood that the distance between the center point of the beacon and the tip of the surgical instrument refers to the actual distance between the center point of the beacon 5 installed on the surgical instrument 2 and the tip of the surgical instrument 2. This distance is fixed after the beacon 5 is installed and can be directly measured.

[0088] S104. Convert the pose matrix of the surgical instrument tip relative to the vision module into the pose matrix of the surgical instrument tip relative to the robot base, specifically:

[0089]

[0090] In the formula, The pose matrix of the surgical instrument tip relative to the robot base; This is the transformation matrix from the vision module to the robot base coordinate system obtained through hand-eye calibration; This represents the pose matrix of the surgical instrument tip relative to the visual module; The coordinates of the surgical instrument tip relative to the robot base; The orientation of the tip of the surgical instrument relative to the robot base.

[0091] It should be understood that when the system starts working, the first step is to perform hand-eye calibration between the robot and the camera, and to solve the relationship between the robot's base coordinate system and the module coordinate system. The specific calibration method will not be elaborated here.

[0092] S105. Convert the center coordinates of the laparoscopic trocar relative to the vision module to the center coordinates of the laparoscopic trocar relative to the robot base, specifically:

[0093]

[0094] In the formula, Here are the center coordinates of the laparoscopic trocar relative to the robot base; These are the coordinates of the laparoscopic trocar relative to the center of the visual module.

[0095] S106. Based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base, calculate the desired pose of the laparoscope relative to the robot base. The specific formula is as follows:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] In the formula, The pose matrix of the surgical instrument tip relative to the robot base; The coordinates of the surgical instrument tip relative to the robot base; The orientation of the surgical instrument tip relative to the robot base; The desired orientation of the laparoscope relative to the robot base; and These are the components of the laparoscope's attitude matrix relative to the robot's base coordinate system on the X, Y, and Z axes, respectively. Let be the center coordinates of the laparoscopic trocar relative to the robot base; vec{1,0,0} is a unit vector in a fixed direction in the world coordinate system.

[0102] S107. Based on the desired posture of the laparoscope relative to the robot base, control the robot to adjust the posture of the laparoscope.

[0103] The controller plans the robot's path based on real-time feedback of the desired posture of the laparoscope relative to the robot base. The robot operates the laparoscope, ensuring its field of view fully covers the end of the surgical instruments. In other words, a robot motion trajectory is generated, causing the robot to slowly rotate the laparoscope 1 around a fixed point on the laparoscope trocar 3 until it reaches the operator's desired posture—the adjusted laparoscope posture. For example, the generated trajectory is not executed immediately; it waits for the surgeon to press the trigger button, generating a trigger signal that initiates the robot's movement. The robot stops adjusting its posture once the desired posture is reached or the trigger signal stops.

[0104] It should be noted that the robot consists of a robot with more than 5 degrees of freedom and a mobile base. The robot is mounted on the mobile base.

[0105] In one embodiment, the laparoscopic surgical work area image also includes a surgical instrument trocar area, and the laparoscopic pose control method further includes the following steps:

[0106] S108. Calculate the center coordinates of the surgical instrument trocar relative to the visual module based on the surgical instrument trocar region.

[0107] S109. Calculate the distance between the beacon and the surgical instrument trocar based on the center coordinates of the surgical instrument trocar relative to the visual module and the center coordinates of the beacon relative to the visual module. The specific formula is as follows:

[0108]

[0109] In the formula, Dist is the distance between the beacon and the surgical instrument trocar; The coordinates of the surgical instrument trocar relative to the center of the vision module; These are the coordinates of the beacon relative to the center of the visual module.

[0110] S110. Based on the change in distance between the beacon and the surgical instrument trocar, control the robot to adjust the depth of insertion of the laparoscope into the laparoscope trocar.

[0111] In other words, the surgeon controls the insertion depth of the laparoscope 1 by changing the depth of the surgical instrument 2 inserted into the abdominal cavity.

[0112] Preferably, when the system determines that the surgeon's manipulation of the surgical instruments has caused a significant change in depth—that is, when the change in distance between the beacon and the surgical instrument trocar exceeds a set threshold within a certain time period—the controller determines that the current situation is a depth adjustment request from the surgeon. At this time, the controller generates a robot motion trajectory, causing the robot to slowly adjust its depth along the current orientation to achieve the depth desired by the surgeon. The depth adjustment is related to the time the surgeon issues the depth adjustment command. The generated trajectory is not executed immediately; it waits for the surgeon to press the trigger button, generating a trigger signal, at which point the robot begins to move. When the trigger signal stops, the robot stops adjusting its depth.

[0113] In one embodiment, the laparoscopic pose control method further includes the following steps:

[0114] S111. Within a set time period, analyze whether the ArUco code of the beacon area changes periodically along the same rotation direction. If a periodic change occurs, control the laparoscopic lens to rotate along the same rotation direction until the ArUco code stops changing periodically.

[0115] In other words, the surgeon controls the rotation of the laparoscope lens by rotating the surgical instruments along their fixed axes. When the system detects significant instrument rotation caused by the surgeon's manipulation of the instruments—that is, when the ArUco codes detected by the camera switch rapidly and sequentially within a certain time period—the controller determines that the current situation is a request from the surgeon to adjust the direction of the laparoscope. At this time, the controller sends a command to rotate the laparoscope lens clockwise or counterclockwise according to the surgeon's rotation command to achieve the desired angle. The angle adjustment is related to the time when the surgeon issues the lens angle adjustment command. The control command is not executed immediately; it waits for the surgeon to press the trigger button. A trigger signal is generated, and the laparoscope lens begins to rotate. When the trigger signal stops, the laparoscope lens stops adjusting its angle.

[0116] For example, in the above embodiment, there are two conditions for the robot to automatically adjust its position during the entire operation: 1) the vision module detects that the surgeon has a clear intention to adjust their posture; 2) the trigger switch is pressed. By using these two conditions, the robot is strictly prevented from making any potentially dangerous movements, ensuring the safety of both the patient and the surgeon.

[0117] For example, the relative positions of the laparoscope and surgical instruments, as well as the desired positions and generated motion trajectories identified by the control system, are displayed in real time through simulation. This assists the surgeon in determining whether the current operation of the laparoscope meets expectations. If it does, the robot is allowed to make adjustments; if not, the surgical instruments are readjusted and new motion commands are issued. Through these methods, the surgeon can control the laparoscope, thereby achieving independent operation of laparoscopic surgery.

[0118] Preferably, the laparoscopic position control method also includes:

[0119] Acquire images of the robot's end effector captured by the vision module;

[0120] Calculate the center coordinates of the robot's end effector relative to the vision module based on the robot's end effector image;

[0121] The system determines whether the robot's end effector is within a preset safe zone relative to the center coordinates of the vision module. If it exceeds the preset safe zone, the robot is controlled to stop working and an alarm is issued to ensure the safety of the surgical procedure.

[0122] In one embodiment of the present invention, a laparoscopic posture control device for robot gripping is provided, comprising:

[0123] The acquisition module is used to acquire images of the laparoscopic surgical work area collected by the vision module. The laparoscopic surgical work area image includes a laparoscopic trocar region and a beacon region set on the surgical instruments. The beacon region contains ArUco encoding.

[0124] The first calculation module is used to calculate the center coordinates of the laparoscopic trocar relative to the visual module based on the laparoscopic trocar region, and to calculate the center coordinates of the beacon relative to the visual module based on the beacon region.

[0125] The second calculation module is used to calculate the pose matrix of the surgical instrument tip relative to the vision module based on the center coordinates of the beacon relative to the vision module and the set distance between the beacon center point and the tip of the surgical instrument.

[0126] The first conversion module is used to convert the pose matrix of the surgical instrument tip relative to the vision module into the pose matrix of the surgical instrument tip relative to the robot base.

[0127] The second conversion module is used to convert the center coordinates of the laparoscopic trocar relative to the vision module into the center coordinates of the laparoscopic trocar relative to the robot base.

[0128] The third calculation module is used to calculate the desired posture of the laparoscope relative to the robot base based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base.

[0129] The control module is used to control the robot to adjust the laparoscope's posture according to the desired posture of the laparoscope relative to the robot base.

[0130] The specific implementation methods of the above modules can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.

[0131] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to implement the operation of a laparoscopic posture control method for robot gripping.

[0132] In one embodiment of the present invention, a method for controlling the pose of a laparoscopic device held by a robot, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data.

[0133] The computer storage medium can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs)), optical storage (e.g., CDs, DVDs, BDs, HVDs), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In the description of this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0139] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robotic gripper for controlling the pose of a laparoscopic instrument, characterized in that, include: The acquisition module is used to acquire an image of the laparoscopic surgical work area collected by the vision module. The image of the laparoscopic surgical work area includes a laparoscopic trocar region and a beacon region set on the surgical instruments. The beacon region contains ArUco encoding. The first calculation module is used to calculate the center coordinates of the laparoscopic trocar relative to the visual module based on the laparoscopic trocar region, and to calculate the center coordinates of the beacon relative to the visual module based on the beacon region. The second calculation module is used to calculate the pose matrix of the surgical instrument tip relative to the vision module based on the center coordinates of the beacon relative to the vision module and the set distance between the beacon center point and the tip of the surgical instrument. The first conversion module is used to convert the pose matrix of the surgical instrument tip relative to the vision module into the pose matrix of the surgical instrument tip relative to the robot base. The second conversion module is used to convert the center coordinates of the laparoscopic trocar relative to the vision module into the center coordinates of the laparoscopic trocar relative to the robot base; The third calculation module is used to calculate the desired posture of the laparoscope relative to the robot base based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base. The control module is used to control the robot to adjust the laparoscope's posture according to the desired posture of the laparoscope relative to the robot base.

2. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, The laparoscopic surgical work area image also includes the surgical instrument trocar area; Calculate the center coordinates of the surgical instrument trocar relative to the visual module based on the surgical instrument trocar region; Calculate the distance between the beacon and the surgical instrument trocar based on the center coordinates of the surgical instrument trocar relative to the center of the visual module and the beacon relative to the center of the visual module; The robot is controlled to adjust the depth of insertion of the laparoscope into the trocar based on the change in the distance between the beacon and the surgical instrument trocar. Also includes: Acquire images of the robot's end effector captured by the vision module; Calculate the center coordinates of the robot's end effector relative to the vision module based on the robot's end effector image; Determine whether the robot's end effector is within a preset safe area relative to the center coordinates of the vision module. If it exceeds the preset safe area, control the robot to stop working.

3. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, Within a set time period, analyze whether the ArUco code in the beacon area changes periodically along the same rotation direction. If a periodic change occurs, control the laparoscopic lens to rotate along the same rotation direction until the ArUco code stops changing periodically.

4. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, The calculation of the beacon's center coordinates relative to the visual module based on the beacon region specifically involves: Convert the beacon region into a point cloud; The point cloud is fitted based on the geometric features of the beacon to obtain the center point of the geometric object; The beacon's attitude is obtained based on the ArUco encoding of the beacon region; Based on the center point of the geometry and the beacon's pose, the center coordinates of the beacon relative to the visual module are obtained.

5. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, The pose matrix of the surgical instrument tip relative to the visual module is calculated based on the center coordinates of the beacon relative to the visual module and the distance from the beacon center point to the tip of the surgical instrument. The specific formula is as follows: In the formula, This represents the pose matrix of the surgical instrument tip relative to the visual module; The orientation of the surgical instrument tip relative to the visual module; The coordinates of the surgical instrument tip relative to the vision module; The pose of the beacon relative to the vision module; , and These are the components of the pose matrix of the surgical instrument tip relative to the visual module coordinate system on the X, Y, and Z axes, respectively. The coordinates of the beacon relative to the center of the visual module; The distance from the center point of the set beacon to the tip of the surgical instrument.

6. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, The process of converting the pose matrix of the surgical instrument tip relative to the vision module into the pose matrix of the surgical instrument tip relative to the robot base specifically involves: In the formula, The pose matrix of the surgical instrument tip relative to the robot base; This is the transformation matrix from the vision module to the robot base coordinate system obtained through hand-eye calibration; This represents the pose matrix of the surgical instrument tip relative to the visual module; The coordinates of the surgical instrument tip relative to the robot base; The orientation of the surgical instrument tip relative to the robot base; The process of converting the center coordinates of the laparoscopic trocar relative to the vision module to the center coordinates of the laparoscopic trocar relative to the robot base specifically involves: In the formula, Here are the center coordinates of the laparoscopic trocar relative to the robot base; These are the coordinates of the laparoscopic trocar relative to the center of the visual module.

7. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, The desired pose of the laparoscope relative to the robot base is calculated based on the center coordinates of the laparoscope trocar relative to the robot base and the pose matrix of the surgical instrument tip relative to the robot base. The specific formula is as follows: In the formula, The pose matrix of the surgical instrument tip relative to the robot base; The coordinates of the surgical instrument tip relative to the robot base; The orientation of the surgical instrument tip relative to the robot base; The desired orientation of the laparoscope relative to the robot base; These are the components of the laparoscope's attitude matrix relative to the robot's base coordinate system on the X, Y, and Z axes, respectively. Here are the center coordinates of the laparoscopic trocar relative to the robot base; It is a unit vector with a fixed direction in the world coordinate system.

8. The robot-held laparoscopic posture control device according to claim 2, characterized in that, The distance between the beacon and the surgical instrument trocar is calculated based on the center coordinates of the surgical instrument trocar relative to the center coordinates of the beacon relative to the center coordinates of the visual module. The specific formula is as follows: In the formula, The distance between the beacon and the surgical instrument trocar; The coordinates of the surgical instrument trocar relative to the center of the vision module; These are the coordinates of the beacon relative to the center of the visual module.

9. The laparoscopic posture control device for robot gripping according to claim 1, characterized in that, The visual modules are positioned on the left and right sides of the surgeon's operating space in the laparoscopic surgery area. The beacon is a cube, and the four sides of the cube are provided with ArUco codes representing different ID values.

Images