An AI surgery robot based on spatial coordinate information obtained through antennal perception

Abstract

The application provides an AI surgery robot based on spatial coordinate information obtained through antennae sensing, installs the antennae on a mechanical arm, and controls and positions the surgical instrument, the antennae and a depth camera by the three mechanically arms with fixed relative positions, so that a unified coordinate system and mutual conversion can be formed. The spatial position of the telescopic antennae sensing distance and the mechanical arm end is corrected based on the depth information of the depth camera in a certain area. The application provides a real measurement method for the depth sensing of the surgery process without increasing the complexity of the surgery, and eliminates the situation that the surgery risk is increased due to the difference of the surgery environment; meanwhile, the problem that the accurate depth information cannot be obtained in the changing surgery environment is solved, so that the AI surgery robot has more accurate visual sensing, completely gets rid of the scheme of evaluating the depth information by using various algorithms, and thus the AI surgery robot can independently and accurately complete the surgery target.

Description

Technical Field

[0001] This invention belongs to the field of medical AI surgical robot technology, specifically relating to an AI surgical robot that acquires spatial coordinate information based on tactile perception. Background Technology

[0002] With the development of medical technology, more and more assistive medical devices are being introduced into hospitals to help doctors perform surgical procedures. AI surgical robots, as assistive medical devices, are intelligent, flexible, and easy to operate. Because AI surgical robots need to display their position on medical imaging equipment, they require extremely high positioning accuracy. Existing positioning modules such as laser, millimeter wave, infrared, and ultrasound, due to the surgical environment and accuracy requirements, do not provide sufficient depth data for use in medical and AI vision surgery fields.

[0003] Currently, combining depth cameras with other algorithms holds promise for improving the positioning accuracy of AI surgical robots for spatial point cloud coordinates. However, within the narrow surgical space, the principle of binocular cameras cannot solve the problem of obtaining precise depth information for targets.

[0004] AI surgical robots require real-time presentation of the depth information and spatial dot matrix data of the surgical object to form accurate visual data for the robot's surgical motion planning and execution. The error between the visual data and the real space must be less than 0.1 millimeters. Current positioning solutions and products cannot meet the requirements of robot vision systems in real surgical environments due to environmental limitations such as changes in surgical lighting, imaging distance and spatial layout, as well as the complex tissue structure of the surgical object. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides an AI surgical robot that acquires spatial coordinate information based on tactile perception. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides an AI surgical robot that acquires spatial coordinate information based on tactile perception, comprising: a depth camera, a retractable tactile antenna with a scale, surgical instruments, three robotic arms, a control center, and a surgical operating table;

[0007] The three robotic arms are fixed at a fixed position on the operating table. All three robotic arms are controlled by the control center, which controls the depth camera, the retractable tentacles with scales, and the surgical instruments to move toward the target area.

[0008] The depth camera, under the direct control of the control center, acquires spatial dot matrix data and image frame data within a certain angle and range, and transmits them to the control center; the retractable antenna, under the direct control of the control center, extends and retracts, and acquires extension and retraction data, which is then transmitted to the control center; the control center uses the depth information from the extension and retraction data and the image frame data to calibrate the spatial dot matrix data and determine the target area; the surgical instruments, under the direct control of the control center, perform surgical actions at the target area.

[0009] Optionally, the control center is mounted on a control center hardware platform, which carries a real-time operating system. A software platform runs on the real-time operating system, and the software platform completes AI data training, recognition, and inference. The control center hardware platform communicates with the depth camera, the retractable antenna, the surgical instruments, and the three robotic arms through physical interfaces. The control center hardware platform includes an AI server and a data center.

[0010] Optionally, the retractable antenna consists of three antennae, each at a certain angle in space. The angles between the antennae and the antennae themselves are adjusted by the robotic arm and the control center. The antennae move by adjusting the angles of the antennae and the rotation of the three antennae, and the extension and retraction data of the stimulated antennae are collected.

[0011] Optionally, the control center controls the maximum extendable distance and maximum resistance of each of the tentacles; when the tentacles encounter an obstacle, it detects whether the collision force is greater than the maximum resistance of the tentacles. If so, the tentacles automatically stop and collect extension and retraction data; if not, the extension and retraction length is increased and the extension and retraction continues.

[0012] Optionally, each of the tentacles is a retractable tentacle, and the extended portion of each tentacle is made of transparent rubber material with a diameter of 0.01 mm. The base of the tentacle is made of a spring, and the extension is stopped by resistance when the tentacle contacts human organ tissue. The scale on the retractable tentacle shows the extension length.

[0013] Optionally, the control center is used for:

[0014] When the spatial dot matrix data, image frame data and extension data transmitted by the retractable antenna are received from the depth camera, the spatial dot matrix data is recalibrated according to the image frame data, the three-dimensional position of the robotic arm end effector and the extension data to obtain calibration dot matrix data.

[0015] The spatial point data transmitted by the depth camera is the data estimated by the depth camera.

[0016] The calibration dot matrix data is used to determine the actual area between the surgical instruments and the surgical site, and the actual area is determined as the target area.

[0017] Optionally, the step of recalibrating the spatial dot matrix data based on image frame data, the three-dimensional position of the robotic arm's end effector, and the extension data to obtain calibrated dot matrix data includes:

[0018] Determine the scale information of the ruler on the antennae from the image frame data;

[0019] The extension length is determined from the extension data, and the extension length is calibrated.

[0020] Using the telescopic length, scale information, and three-dimensional position of the robotic arm's end effector, the coordinate offset of the target area is calculated.

[0021] The spatial point matrix data is calibrated using the coordinate offset to obtain calibration point matrix data.

[0022] Optionally, calibrating the telescopic length includes:

[0023] The depth camera captures scale information and antenna shape when the antennae bend;

[0024] The curvature is calculated using the scale information and the shape of the antennae;

[0025] The extension length is calibrated using the curvature.

[0026] Optionally, determining the actual area of ​​the surgical instrument and surgical site using the calibration dot matrix data includes:

[0027] The image frame data and the calibration dot matrix data are synchronously input into the AI ​​recognition algorithm module to obtain the part classification result;

[0028] Determine the true three-dimensional location of the part based on the part classification results;

[0029] The actual three-dimensional location is determined as the actual region.

[0030] Beneficial effects:

[0031] This invention provides an AI surgical robot that acquires spatial coordinate information based on antennal perception. An antenna is mounted on a robotic arm, and surgical instruments, the antenna, and a depth camera are controlled and positioned by three relatively fixed robotic arms, forming a unified coordinate system and allowing for mutual conversion. A transparent, scaled, extendable antenna is used to sense distance and the spatial position of the robotic arm's end effector to correct the depth information of a specific area based on the depth camera. This invention provides a realistic means of measuring depth perception during surgery without increasing surgical complexity, eliminating various situations where surgical risks increase due to differences in the surgical environment. It also solves the problem of not being able to obtain accurate depth information in changing surgical environments, enabling the AI ​​surgical robot to have more precise visual perception. This completely eliminates the need for relying on various algorithms to evaluate depth information, making it possible for the AI ​​surgical robot to independently and accurately complete surgical objectives.

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the structure of an AI surgical robot that acquires spatial coordinate information based on antennal perception, provided by the present invention.

[0034] Figure 2 This is a schematic diagram of the control center hardware platform provided by the present invention;

[0035] Figure 3 This is a schematic diagram illustrating the operating principle of the AI ​​surgical robot provided by the present invention. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0037] refer to Figure 1 The present invention provides an AI surgical robot based on antenna perception to obtain spatial coordinate information, including: a depth camera, a retractable antenna with a scale, surgical instruments, three robotic arms, a control center, and a surgical operating table;

[0038] The three robotic arms are fixed at a fixed position on the operating table. All three robotic arms are controlled by the control center, which controls the depth camera, the retractable tentacles with scales, and the surgical instruments to move toward the target area.

[0039] It is worth noting that the robotic arm of this invention mainly performs the tasks of moving the depth camera, positioning the retractable tentacles, moving the surgical instruments, and designing the motion. The robotic arm base, which fixes the depth camera, tentacles, and surgical instruments on the surgical platform, can obtain the relative positions of the three components through coordinate transformation.

[0040] The depth camera is directly controlled by the control center to collect spatial dot matrix data and image frame data within a certain angle and range and transmits them to the control center.

[0041] The depth camera outputs image frame data from a video stream and visual information in three-dimensional space, i.e., spatial point data. The closer the spatial point data and image frame data are to the critical range and angle, the greater the deviation. In this invention, the depth camera sends one frame of image data every 30ms. The control center receives the image frames output by the depth camera and summarizes the reported depth information to obtain spatial point data.

[0042] The retractable tentacles are directly controlled by the control center to extend and retract, and the extension and retraction data is collected and transmitted to the control center. The control center uses the extension and retraction data and the depth information of the image frame data to calibrate the spatial dot matrix data and determine the target area. The surgical instruments are directly controlled by the control center to complete the surgical action at the target area.

[0043] refer to Figure 2 The control center is mounted on a control center hardware platform, which carries a real-time operating system. A software platform runs on the real-time operating system, and the software platform completes AI data training, recognition, and inference. The control center hardware platform communicates with the depth camera, the retractable antenna, the surgical instruments, and the three robotic arms through physical interfaces. The control center hardware platform includes an AI server and a data center.

[0044] The real-time operating system of this invention can schedule computing resources, and the AI ​​server and data center provide powerful computing capabilities for the AI ​​surgical robot. The software platform provides a large amount of data for training models, data, and complex algorithms, accelerating the training and inference process of the AI ​​model.

[0045] In one specific embodiment, the retractable antenna of the present invention consists of three antennae, each antennae having a certain angle in space. The angle between the antennae and the antennae themselves are adjusted by the robotic arm and the control center. The antennae are moved by adjusting the angle of the antennae and the rotation of the three antennae themselves, and the extension and retraction data of the stimulated antennae are collected.

[0046] Each tentacle is retractable, and the extended portion of each tentacle is made of transparent rubber material with a diameter of 0.01 mm. A scale on the retractable tentacle indicates the extension length. A tiny spring at its base pushes the tentacle forward. When the tentacle contacts human organs or other instruments, it stops extending due to resistance. The tentacle itself can output angle and length information, while a depth camera collects the scale on the tentacle to calculate the length of the tentacle. The true coordinates of the tentacle tip are then calculated and corrected to achieve depth visual information with an error of less than 0.1 mm.

[0047] The control center controls the maximum extension distance and maximum resistance of each tentacle in the target area; when the tentacle encounters an obstacle, it detects whether the collision force is greater than the maximum resistance of the tentacle. If so, the tentacle automatically stops and collects extension and contraction data; if not, it increases the extension and contraction length and continues to extend and contract.

[0048] It is worth noting that the control center of this invention can control the maximum extension distance and maximum resistance of each tentacle in real time. When encountering an obstacle, if the collision force exceeds the maximum resistance of the tentacle, the tentacle will automatically stop.

[0049] In one specific embodiment of the present invention, the control center is used for:

[0050] When the spatial dot matrix data, image frame data and extension data transmitted by the retractable antenna are received from the depth camera, the spatial dot matrix data is recalibrated according to the image frame data, the three-dimensional position of the robotic arm end effector and the extension data to obtain calibration dot matrix data.

[0051] The spatial point data transmitted by the depth camera is the data estimated by the depth camera.

[0052] This invention determines the scale information of the scale on the antenna from the image frame data; determines the extension length from the extension data and calibrates the extension length; calculates the coordinate offset of the target area using the extension length, scale information, and three-dimensional position of the robotic arm end effector; and calibrates the spatial dot matrix data using the coordinate offset to obtain calibration dot matrix data.

[0053] When the angular information changes, it is reported to the control center in real time. The control center performs a unified coordinate transformation on the spatial position information of the antennal tip and compares it with the reported coordinates from the depth camera. When the difference exceeds a certain threshold within a continuous time period, the depth information from the depth camera is recalibrated. Using the corrected spatial dot matrix data, the control center readjusts the surgical control commands and sends them to the surgical instrument robotic arm module.

[0054] The calibration dot matrix data is used to determine the actual area between the surgical instruments and the surgical site, and the actual area is determined as the target area.

[0055] It's worth noting that: the extension distance of the tentacle itself can be acquired in real time; the three-dimensional position of the robotic arm's end effector can be acquired in real time via the robotic arm; and the camera can acquire the scale information of the tentacle reaching the target position in real time. By calculating the coordinate offset of these three data points in space, the three-dimensional spatial depth information of the target surface based on the user's coordinate system can be obtained. This information is then used to correct the three-dimensional spatial visual information output by the camera, i.e., to calibrate the dot matrix data. For example, if the information at the end effector of the robotic arm is x, y, z, and the distance the tentacle extends in the direction of angle θ is a, b, c, the final three-dimensional coordinates of the target position can be calculated using the formula: X, Y, Z.

[0056] In one specific embodiment of the present invention, calibrating the telescopic length includes:

[0057] The depth camera captures scale information and antenna shape when the antennae bend;

[0058] The curvature is calculated using the scale information and the shape of the antennae;

[0059] The extension length is calibrated using the curvature.

[0060] In particular, when the antennae bend or otherwise change under certain surgical conditions, the X, Y, and Z coordinates of the final target position are further corrected by visually observing the scale information on the antennae and the shape of the antennae.

[0061] In one specific embodiment of the present invention, determining the true area of ​​the surgical instrument and the surgical site using the calibration dot matrix data includes:

[0062] The image frame data and the calibration dot matrix data are synchronously input into the AI ​​recognition algorithm module to obtain the part classification result;

[0063] This invention outputs three-dimensional spatial data of the entire target area by simultaneously outputting data from multiple antennae, generating data multiple times during the surgical process, and reflecting changes in the relative position of the surgical site. The output data is marked in units of 0.1 millimeters. Based on the site classification results output by the AI ​​recognition algorithm module, this invention filters images according to the classification results and image quality, outputting image frames that meet the required clarity and recognition result indicators (trust level, accuracy, etc.).

[0064] Determine the true three-dimensional location of the part based on the part classification results;

[0065] The actual three-dimensional location is determined as the actual region.

[0066] refer to Figure 3 , Figure 3 The working principle of the AI ​​surgical robot provided by this invention is as follows:

[0067] 1. The AI ​​recognition algorithm module identifies the specific body part, and the video stream of the AI ​​recognition algorithm module is used as a visual input source in the form of image frames, with one image frame input every 30ms.

[0068] 2. Surgical targets of surgical robots include: A. Organs such as the liver, gallbladder, bile duct area, and intestines in gallbladder surgery. B. Surgical instruments such as electrocautery hooks, ultrasonic scalpels, and speculum forceps. C. Tentacles.

[0069] 3. The control center outputs depth information; the control center of this invention contains a spatial position calculation module, which uses the output of the depth camera to calculate the depth information (i.e., z coordinate) at the corresponding position in the two-dimensional image (x / y coordinates).

[0070] 4. Initial output of the surgical robot: First, it is necessary to extract the site information required during the operation. The format includes: site name, position (including the x, y, z coordinates of the center point and the contour), and pose.

[0071] 5. Filter the output of the current image frame and determine whether there is antenna information. If not, the initial output of the surgical site in the previous step is taken as the trustworthy final output.

[0072] 6. Filter the output of the current image frame. If antenna information is present, check again whether the antenna has collided. If no collision has occurred, the initial output of the surgical site is taken as the reliable final output. Note: The criterion for determining a collision is that the change in the length of the initial output antenna of the surgical site over 20 consecutive image frames is less than 1mm.

[0073] 7. Calculate the coordinates of the antennal tip:

[0074] A. The current image frame contains antenna information and a collision has occurred. The antenna's own length information is reported in real time, along with the angle information θ and the relative elongation data a, b, and c.

[0075] B. At the same time, the antenna recognition algorithm calculates the length of the antenna based on the scale information of the antenna in the image frame;

[0076] C. Determine the antenna curvature (or non-curvature) based on the antenna length;

[0077] D. Calculate the three-dimensional spatial data of the target area based on the antenna length and angle, and the position of the robotic arm end.

[0078] 8. Based on the actual position (coordinates) of the antenna tip fed back by the depth camera, compare it with the calculation result in step 7. If any of the x, y, or z coordinates exceeds a certain threshold (which can be adjusted according to the parameters of the depth camera), recalculate the position and pose of the part in the image frame based on the difference (the difference between the target area and the real area in D), and use it as the final output.

[0079] 9. Based on the requirements of the current surgical scenario, the control center controls the tentacle robotic arm to reach the target area, which can correct the spatial information of a certain area, so as to achieve more accurate visual positioning of the AI ​​surgical robot.

[0080] This invention provides an AI surgical robot that acquires spatial coordinate information based on antennal perception. An antenna is mounted on a robotic arm, and surgical instruments, the antenna, and a depth camera are controlled and positioned by three relatively fixed robotic arms, forming a unified coordinate system and allowing for mutual conversion. A transparent, scaled, extendable antenna is used to sense distance and the spatial position of the robotic arm's end effector to correct the depth information of a specific area based on the depth camera. This invention provides a realistic means of measuring depth perception during surgery without increasing surgical complexity, eliminating various situations where surgical risks increase due to differences in the surgical environment. It also solves the problem of not being able to obtain accurate depth information in changing surgical environments, enabling the AI ​​surgical robot to have more precise visual perception. This completely eliminates the need for relying on various algorithms to evaluate depth information, making it possible for the AI ​​surgical robot to independently and accurately complete surgical objectives.

[0081] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0083] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An AI surgical robot that acquires spatial coordinate information based on antennal perception, characterized in that, include: Depth camera, retractable antenna with scale, surgical instruments, three robotic arms, control center, operating table; The three robotic arms are fixed at a fixed position on the operating table. All three robotic arms are controlled by the control center, which controls the depth camera, the retractable tentacles with scales, and the surgical instruments to move toward the target area. The depth camera, under the direct control of the control center, acquires spatial dot matrix data and image frame data within a certain angle and range, and transmits them to the control center; the retractable antenna, under the direct control of the control center, extends and retracts, and acquires extension and retraction data, which is then transmitted to the control center; the control center uses the depth information from the extension and retraction data and the image frame data to calibrate the spatial dot matrix data and determine the target area; the surgical instruments, under the direct control of the control center, perform surgical actions at the target area.

2. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 1, characterized in that, The control center is mounted on a control center hardware platform, which carries a real-time operating system. A software platform runs on the real-time operating system, and the software platform completes AI data training, recognition, and inference. The control center hardware platform communicates with the depth camera, the retractable antenna, the surgical instruments, and the three robotic arms through physical interfaces. The control center hardware platform includes an AI server and a data center.

3. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 1, characterized in that, The retractable tentacles consist of three tentacles, each at a certain angle in space. The angles between the tentacles and the tentacles themselves are adjusted by the robotic arm and the control center. The tentacles move by adjusting the angles and the rotation of the three tentacles, and the extension and retraction data of the stimulating tentacles are collected.

4. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 3, characterized in that, The control center controls the maximum extendable distance and maximum resistance of each tentacle; when the tentacle encounters an obstacle, it detects whether the collision force is greater than the maximum resistance of the tentacle. If so, the tentacle automatically stops and collects extension and retraction data; if not, it increases the extension and retraction length and continues to extend and retract.

5. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 3, characterized in that, Each of the tentacles is retractable, and the extended portion of each tentacle is made of transparent rubber material with a diameter of 0.01 mm. The base of the tentacle is spring-loaded, and the extension is stopped by resistance when the tentacle comes into contact with human organs or tissues. The scale on the retractable tentacle indicates the extension length.

6. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 1, characterized in that, The control center is used for: When the spatial dot matrix data, image frame data and extension data transmitted by the retractable antenna are received from the depth camera, the spatial dot matrix data is recalibrated according to the image frame data, the three-dimensional position of the robotic arm end effector and the extension data to obtain calibration dot matrix data. The spatial point data transmitted by the depth camera is the data estimated by the depth camera. The calibration dot matrix data is used to determine the actual area between the surgical instruments and the surgical site, and the actual area is determined as the target area.

7. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 6, characterized in that, The step of recalibrating the spatial dot matrix data based on image frame data, the three-dimensional position of the robotic arm's end effector, and the extension data to obtain calibrated dot matrix data includes: Determine the scale information of the ruler on the antennae from the image frame data; The extension length is determined from the extension data, and the extension length is calibrated. Using the telescopic length, scale information, and three-dimensional position of the robotic arm's end effector, the coordinate offset of the target area is calculated. The spatial point matrix data is calibrated using the coordinate offset to obtain calibration point matrix data.

8. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 7, characterized in that, The calibration of the telescopic length includes: The depth camera captures scale information and antenna shape when the antennae bend; The curvature is calculated using the scale information and the shape of the antennae; The extension length is calibrated using the curvature.

9. The AI ​​surgical robot based on antennal perception to acquire spatial coordinate information according to claim 6, characterized in that, The process of determining the actual area of ​​the surgical instruments and surgical site using the calibration dot matrix data includes: The image frame data and the calibration dot matrix data are synchronously input into the AI ​​recognition algorithm module to obtain the part classification result; Determine the true three-dimensional location of the part based on the part classification results; The actual three-dimensional location is determined as the actual region.

Images