Surgical robot system, surgical robot drilling guidance method and device

Abstract

This invention relates to a surgical robot system, a surgical robot perforation guidance method and apparatus. The system acquires first image data of the lesion area, establishes a three-dimensional model of the lesion area based on the first image data, acquires second image data of the scanning device's working environment, and determines the perforation position of the surgical arm in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data. Throughout the process, the surgeon can intuitively observe the perforation position of the surgical arm and the lesion area through the visual device, thereby achieving accurate surgical robot perforation guidance.

Description

Technical Field

[0001] This application relates to the field of robot-assisted surgery technology, and in particular to a surgical robot system, a surgical robot punching guidance method, device, storage medium, and computer program product. Background Technology

[0002] With the development of science and technology, minimally invasive surgical robots are increasingly being used in minimally invasive surgeries. The design philosophy of surgical robots is to perform complex surgical procedures with precision using a minimally invasive approach. Surgical instruments enter the body cavity through specific openings on the skin to approach the lesion, and the surgeon controls the surgical robot to complete the operation. The openings on the patient's skin for the surgical instruments to pass through are usually planned preoperatively by the surgeon based on the lesion and surgical procedure, and the surgeon makes the holes based on experience.

[0003] However, it is difficult for doctors' experience to be combined with the configuration and workspace of minimally invasive surgical robots, which makes it impossible for traditional surgical robot systems to achieve accurate drilling guidance. Summary of the Invention

[0004] Therefore, it is necessary to address the technical problem that existing surgical robot systems cannot achieve accurate drilling guidance by providing a surgical robot system, surgical robot drilling guidance method, device, storage medium, and computer program product that can achieve accurate drilling guidance.

[0005] In a first aspect, this application provides a surgical robot system, including a scope-holding arm, and:

[0006] The scanning unit is used to acquire the first image data of the lesion area;

[0007] The modeling unit is used to build a three-dimensional model of the lesion area based on the first image data;

[0008] A vision unit is used to acquire second image data of the working environment of the scanning device, wherein the scanning device is a scanning device that scans and generates first image data of the lesion area;

[0009] The processing unit is used to determine the position of the first hole in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data, wherein the first hole is the hole of the lens arm.

[0010] In one embodiment, the processing unit determines the position of the first aperture in the visual device coordinate system based on the three-dimensional model of the lesion region and the second image data, including:

[0011] Obtain the first pose relationship between the scanning device and the lesion region;

[0012] Obtain the second pose relationship between the scanning device and the vision device;

[0013] Based on the first pose relationship and the second pose relationship, the position information of the three-dimensional model of the lesion area in the visual device coordinate system is obtained;

[0014] Based on the position information of the three-dimensional model of the lesion area in the coordinate system of the visual device, the position of the first hole in the coordinate system of the visual device is determined.

[0015] In one embodiment, the processing unit is further configured to obtain the pre-drilling position of the lesion region; the step of determining the position of the first hole in the visual device coordinate system based on the position information of the three-dimensional model of the lesion region in the visual device coordinate system includes:

[0016] Based on the position of the lesion area in the coordinate system of the visual device according to the three-dimensional model, the pre-drilling position is corrected to obtain the position of the first hole in the coordinate system of the visual device.

[0017] In one embodiment, the surgical robot system further includes a robotic arm, and the processing unit is further configured to obtain the position of a second hole based on the position of the first hole, wherein the position of the second hole is a hole of the robotic arm.

[0018] In one embodiment, obtaining the position of the second hole based on the position of the first hole includes:

[0019] Obtain the preset distance between the first hole position and the second hole position;

[0020] The position of the second hole is obtained based on the preset distance and the position of the first hole.

[0021] In one embodiment, obtaining the position of the second hole based on the preset distance and the position of the first hole includes:

[0022] The end-effector position is determined based on the preset distance, the position of the first hole, and the lesion area;

[0023] The inverse kinematics of the arm are solved based on the end-effector pose of the arm.

[0024] If the inverse kinematics has a solution, then the contact point between the line connecting the end pose of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

[0025] In one embodiment, determining the end-effector pose based on the preset distance, the position of the first aperture, and the lesion area includes:

[0026] The axis of the arm cannula is determined based on the preset distance, the position of the first hole, and the lesion area.

[0027] Based on the axis of the arm sleeve, the midpoint of the joint stroke is determined by selecting the degree of freedom of rotation of the arm sleeve around itself.

[0028] The end-effector position is determined based on the midpoint of the joint stroke.

[0029] In one embodiment, the number of preset distances is multiple; obtaining the position of the second hole based on the preset distance and the position of the first hole includes:

[0030] Based on multiple preset distances and the position of the first hole, determine the positions of multiple initial second holes;

[0031] Calculate the arm spacing between the lens-holding arm and the mechanical arm in multiple initial second hole positions;

[0032] The hole with the largest arm-to-arm distance between the lens-holding arm and the mechanical arm is determined as the second hole.

[0033] In one embodiment, calculating the arm spacing between the lens-holding arm and the mechanical arm in a plurality of initial second aperture positions includes:

[0034] Based on the end-effector pose of the arm, inverse kinematics solution is performed on the arm to obtain the target pose of the arm.

[0035] The arm spacing between the lens-holding arm and the mechanical arm in multiple initial second aperture positions is calculated based on the target pose of the mechanical arm and the pose of the lens-holding arm.

[0036] Secondly, this application also provides a surgical robot drilling guidance method, the method comprising:

[0037] Acquire first-image data of the lesion area;

[0038] Based on the first image data, a three-dimensional model of the lesion area is established;

[0039] Acquire second image data of the working environment of the scanning device, wherein the scanning device is the scanning device that generates the first image data of the lesion area;

[0040] Based on the three-dimensional model of the lesion area and the second image data, the position of the first aperture in the visual device coordinate system is determined, and the first aperture is the aperture of the lens arm.

[0041] In one embodiment, determining the position of the first aperture in the visual device coordinate system based on the three-dimensional model of the lesion region and the second image data includes:

[0042] Obtain the first pose relationship between the scanning device and the lesion region;

[0043] Obtain the second pose relationship between the scanning device and the vision device;

[0044] Based on the first pose relationship and the second pose relationship, the position information of the three-dimensional model of the lesion area in the visual device coordinate system is obtained;

[0045] Based on the position information of the three-dimensional model of the lesion area in the coordinate system of the visual device, the position of the first hole in the coordinate system of the visual device is determined.

[0046] In one embodiment, the above-described surgical robot punching guidance method further includes:

[0047] Obtain the preset distance between the first hole and the second hole, where the second hole is the hole of the holding arm;

[0048] The position of the end effector arm is determined based on the preset distance and the lesion area;

[0049] The inverse kinematics of the arm are solved based on the end-effector pose of the arm.

[0050] If the inverse kinematics has a solution, then the contact point between the line connecting the end pose of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

[0051] Thirdly, this application also provides a surgical robot drilling guidance device, including a scanning component, a vision component, and a control component;

[0052] The scanning component generates first image data of the lesion area and sends the first image data to the control component;

[0053] The control component uses the surgical robot drilling guidance method described above to generate the position of the first hole in the vision component.

[0054] In one embodiment, the scanning assembly includes an ultrasound probe with a visual target disposed thereon; the visual target emits a cursor signal to illuminate the lesion area;

[0055] The vision component receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

[0056] In one embodiment, the visual component includes:

[0057] The camera element acquires the cursor signal on the lesion area and sends the position information of the cursor signal to the control component;

[0058] An angle adjustment element carries the camera element and is connected to the control component, wherein the control component controls the angle adjustment element to adjust the image tracking angle of the camera element;

[0059] A mixed reality element displays the hole position in the visual component coordinate system pushed by the control component.

[0060] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0061] Acquire first-image data of the lesion area;

[0062] Based on the first image data, a three-dimensional model of the lesion area is established;

[0063] Acquire second image data of the working environment of the scanning device, wherein the scanning device is the scanning device that generates the first image data of the lesion area;

[0064] Based on the three-dimensional model of the lesion area and the second image data, the position of the first aperture in the visual device coordinate system is determined, and the first aperture is the aperture of the lens arm.

[0065] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0066] Acquire first-image data of the lesion area;

[0067] Based on the first image data, a three-dimensional model of the lesion area is established;

[0068] Acquire second image data of the working environment of the scanning device, wherein the scanning device is the scanning device that generates the first image data of the lesion area;

[0069] Based on the three-dimensional model of the lesion area and the second image data, the position of the first aperture in the visual device coordinate system is determined, and the first aperture is the aperture of the lens arm.

[0070] The aforementioned surgical robot system, surgical robot perforation guidance method, device, storage medium, and computer program product acquire first image data of the lesion area, establish a three-dimensional model of the lesion area based on the first image data, acquire second image data of the scanning device's working environment, and determine the perforation position of the surgical arm in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data. Throughout the process, the surgeon can intuitively observe the perforation position of the surgical arm and the lesion area through the visual device, thereby achieving accurate surgical robot perforation guidance. Attached Figure Description

[0071] Figure 1 This is a schematic diagram illustrating an application scenario of the surgical robot system of this application in one embodiment;

[0072] Figure 2 This is a structural block diagram of a surgical robot system in one embodiment;

[0073] Figure 3 This is a schematic diagram illustrating the relative relationships between the scanning device, the visual device, and the lesion area in one embodiment.

[0074] Figure 4 This is a schematic diagram showing the relative structural positions of the surgical robot arm and the endoscope arm in one embodiment of the surgical robot system.

[0075] Figure 5 This is a schematic diagram illustrating the relative positional relationship between the lesion point and the endoscope arm in one embodiment.

[0076] Figure 6 A flowchart illustrating the process of determining the pre-drilling location in a specific application example;

[0077] Figure 7 This is a flowchart illustrating a surgical robot drilling guidance method in one embodiment;

[0078] Figure 8 This is a schematic diagram of the perforation guidance device for a surgical robot in one embodiment;

[0079] Figure 9 This is a schematic diagram of the structure of the scanning component and the vision component in one embodiment. Detailed Implementation

[0080] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0081] In one embodiment, such as Figure 1As shown, a surgical robot system is provided, including a scope-holding arm. This embodiment illustrates the use of this surgical robot system to assist doctors in guiding the drilling location. In this embodiment, the surgical robot system also includes a scanning unit, a modeling unit, a vision unit, and a processing unit, which can be integrated into one or more main control consoles. In practical applications, the surgical robot system also includes a scanning device and a vision device. One or more main control consoles integrating the above-mentioned system units are respectively connected to the scanning device and the vision device. The patient lies flat on the operating table, the scanning device acquires the patient's medical images, and sends the acquired medical images to the aforementioned computer device. The main control console processes the data to obtain the hole position of the scope-holding arm in the coordinate system of the vision device, and sends the data to the vision device. The doctor wears the vision device, which displays the hole position of the scope-holding arm. In this way, the entire surgical robot system can achieve accurate drilling guidance.

[0082] Specifically, such as Figure 2 As shown, in one embodiment, the surgical robot system includes:

[0083] The scanning unit 200 is used to acquire the first image data of the lesion area.

[0084] The scanning device scans the object to obtain medical image data. The scanning unit 200 analyzes this medical image data to obtain the first image data of the lesion area. Specifically, the object to be scanned refers to the object to be drilled by the surgical robot. Taking a patient as an example, the object to be drilled is the surgical patient, and the medical image of the patient is scanned by the scanning device. The lesion area refers to the area on the body of the object to be drilled by the surgical robot where a lesion has occurred. For example, it could be the location of a lesion in the kidney or gallbladder in the abdominal cavity of the object to be scanned. Furthermore, the scanning unit 200 can first extract the image of a preset scanning area from the medical image data sent by the scanning device. The preset scanning area refers to the area corresponding to the area to be operated on. Specifically, it is a scanning area pre-defined on the object to be scanned, such as the abdomen, chest, or back of the object to be scanned, for example, through the image data of the patient's abdomen.

[0085] Modeling unit 400 is used to build a three-dimensional model of the lesion area based on the first image data.

[0086] Using the first image data as the basis, a virtual 3D model was constructed, and the corresponding lesion areas in the patient's body were identified on the virtual 3D model. This facilitates the calculation and planning of drilling locations for the lesion areas in subsequent processing within the virtual 3D model.

[0087] The vision unit 600 is used to acquire second image data of the working environment of the scanning device, which is a scanning device that scans and generates first image data of the lesion area.

[0088] The operating environment of a scanning device refers to the scene environment in which the scanning device scans a patient to generate medical image data. Specifically, as mentioned above, the scanning device scans the patient to generate medical images, and simultaneously, image data is acquired during the scanning process to accurately determine the phase pose relationship between the scanning device and the visual device. Furthermore, this facilitates accurate determination of the relative pose relationship between the visual device and the scanning device, enabling the accurate representation of the arm's aperture position in the visual device.

[0089] The processing unit 800 is used to determine the position of the first aperture in the coordinate system of the visual device based on the three-dimensional model of the lesion area and the second image data. The first aperture is the aperture of the lens arm.

[0090] Visual devices assist doctors in visually observing their surroundings and displaying specific drilling locations on them for easy structural viewing. For example, a visual device could be a mixed reality (MR) helmet worn by a doctor. Specifically, the hole positions of the endoscope-holding arm in a surgical robot system can be determined based on a 3D model of the lesion area. At this point, the hole positions are constructed in the coordinate system of the scanning device. To accurately map these hole positions to the visual device coordinate system, the relative pose relationship between the scanning and visual devices must be further determined, i.e., the point mapping relationship between the scanning and visual device coordinate systems must be clarified. Based on this pose relationship, the hole positions of the endoscope-holding arm determined from the 3D model of the lesion area are mapped to the visual device coordinate system to determine the hole positions of the endoscope-holding arm in the visual device coordinate system.

[0091] Furthermore, when determining the hole positions of the surgical arm based on a 3D model of the lesion area, the drilling positions are pre-calculated in a virtual 3D model based on the location of the lesion in the patient's body and the configuration of the surgical robot system. For example, if the lesion is located on an organ within the abdominal cavity, the drilling position corresponds to the surface of the abdomen. After determining the hole positions of the surgical arm in the coordinate system of the visual device, the data can be pushed to the visual device so that the doctor can see the accurate hole positions of the surgical arm on the display interface of the recognition device after wearing the visual device. This allows the doctor to intuitively observe the drilling positions and thus achieve accurate drilling guidance.

[0092] The aforementioned surgical robot system acquires first image data of the lesion area and establishes a three-dimensional model of the lesion area based on the first image data; it also acquires second image data of the scanning device's working environment and determines the hole position of the surgical arm in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data. Throughout the process, the surgeon can intuitively observe the drilling position of the surgical arm and the lesion area through the visual device, thus achieving accurate surgical robot drilling guidance.

[0093] In one embodiment, the processing unit 800 determines the position of the first aperture in the visual device coordinate system based on the three-dimensional model of the lesion region and the second image data by: acquiring a first pose relationship between the scanning device and the lesion region; acquiring a second pose relationship between the scanning device and the visual device; obtaining the position information of the three-dimensional model of the lesion region in the visual device coordinate system based on the first pose relationship and the second pose relationship; and determining the position of the first aperture in the visual device coordinate system based on the position information of the three-dimensional model of the lesion region in the visual device coordinate system.

[0094] A scanning device refers to a device used to scan a patient and generate images of a preset scanning area. For example, it can be any of an ultrasound imaging device, an X-ray device, or a magnetic resonance imaging device (MRI device). In this embodiment, the scanning device is an ultrasound imaging device. The first pose relationship between the scanning device and the corresponding lesion area of ​​the scanned object refers to the pose relationship between the scanning device and the lesion area of ​​the scanned object. For example, it can be the relationship between the relative position and orientation of the ultrasound probe of the ultrasound imaging device and the lesion area of ​​the patient.

[0095] The relative pose relationship between the scanning device and the vision device refers to the relationship between the relative position and orientation of the structure in the scanning device used to scan the pre-defined scanning area on the patient's body and the vision device. For example, this could be the relationship between the position and orientation of the ultrasound probe in an ultrasound imaging device and the MR helmet. The second pose relationship between the scanning device and the vision device refers to the relationship between their relative positions and orientations. For example, this could be the relationship between the position and orientation of the ultrasound probe in an ultrasound imaging device and the MR helmet. By combining the first and second pose relationships, the position in the 3D model of the lesion area can be mapped to the vision device coordinate system. Therefore, the position of the endoscope arm aperture determined in the 3D model of the lesion area can be mapped to the vision device coordinate system, obtaining the aperture position of the endoscope arm in the vision device coordinate system.

[0096] For example, refer to Figure 3It can be understood that the processing unit 800 can obtain a first pose relationship based on the relative pose relationship between the ultrasound probe on the ultrasound imaging equipment and the corresponding lesion area of ​​the patient. The first pose relationship is the relative pose relationship between the probe and the lesion area. The processing unit 800 can also obtain a second pose relationship based on the relative pose relationship between the probe on the ultrasound imaging equipment and the MR helmet. The second pose relationship is the pose relationship between the probe and the MR helmet. Based on the relationship between the first pose relationship and the second pose relationship, the processing unit 800 can map the three-dimensional model of the lesion area to the coordinate system of the visual device, and then determine the position of the aperture of the endoscope arm in the coordinate system of the visual device based on the position of the aperture of the endoscope arm determined in the three-dimensional model of the lesion area.

[0097] In this embodiment, the pose relationship between the lesion region and the MR helmet is obtained by transforming the pose relationship between the lesion region, the scanning device, and the vision device, thereby converting the lesion region's location information into location information in the vision device's coordinate system. It should be noted that pose includes both positional information and orientation information; therefore, pose relationships include relationships between positions and relationships between orientations.

[0098] It should be noted that in this embodiment, the vision device may include a depth camera and a gimbal. The depth camera can detect three-dimensional information of the environment, and the gimbal can adjust the orientation of the depth camera. For example, the angle of the depth camera can be adjusted by rotating the gimbal, thereby adjusting the position of the depth camera's field of view. Since the depth camera can only detect three-dimensional information of the surrounding environment and cannot detect intracavitary information of the patient, this embodiment obtains the position information of the probe in the scanning device through the depth camera. The position information of the lesion area can be obtained through the probe. Then, the pose relationship between the depth camera and the lesion area can be calculated based on the probe in the scanning device.

[0099] In one embodiment, the processing unit 800 is further configured to obtain the pre-drilling position of the lesion area; determining the position of the first hole in the visual device coordinate system based on the position information of the three-dimensional model of the lesion area in the visual device coordinate system includes: correcting the pre-drilling position based on the position of the three-dimensional model of the lesion area in the visual device coordinate system to obtain the position of the first hole in the visual device coordinate system.

[0100] The pre-drilling position in the lesion area refers to the drilling position of the endoscope arm determined in advance based on the 3D model of the lesion area before surgery. In practice, the drilling position is initially determined by an examination before surgery. However, during the actual surgery, to further improve the accuracy of the surgery, the pre-determined drilling position is corrected. After the correction is completed in the 3D model of the lesion area, the position of the first hole in the visual device coordinate system is obtained based on the position of the 3D model of the lesion area in the visual device coordinate system.

[0101] In one embodiment, the surgical robot system further includes a robotic arm, and the processing unit is further configured to obtain the position of a second hole based on the position of the first hole, the position of the second hole being the hole of the robotic arm.

[0102] like Figure 4 As shown, the surgical robot system includes an endoscope-holding arm 420 and a surgical instrument-holding arm 410. The endoscope-holding arm 420 is fixedly connected to an endoscope 422 and moves the endoscope 422. The surgical instrument-holding arm 410 is fixedly connected to a surgical instrument 412 and moves the surgical instrument 412. After determining the port positions of the endoscope-holding arm 420, the port positions of the surgical instrument-holding arm 410 need to be further determined based on the port positions of the endoscope-holding arm 420. Specifically, since the endoscope-holding arm 420 and the surgical instrument-holding arm 410 generally maintain a relative distance, the port positions of the surgical instrument-holding arm 410 can be determined based on this relative distance after the port positions of the endoscope-holding arm 420 are obtained.

[0103] In one embodiment, obtaining the position of the second hole based on the position of the first hole includes: obtaining a preset distance between the first hole and the second hole; and obtaining the position of the second hole based on the preset distance and the position of the first hole.

[0104] The preset distance is a pre-determined distance, which can be obtained based on prior experience. Specifically, the minimum and maximum drilling distances between the robotic arm and the endoscope arm can be determined based on prior experience. Then, a suitable preset distance is determined from the minimum and maximum drilling distances based on the movement trajectory and size parameters of the robotic arm and the endoscope arm, as well as the patient's size parameters. More specifically, in a certain application scenario, the minimum drilling distance between the robotic arm and the endoscope arm can be 6 cm, and the maximum drilling distance can be 10 cm.

[0105] In one embodiment, obtaining the position of the second hole based on the preset distance and the position of the first hole includes: determining the end-effector pose of the robotic arm based on the preset distance, the position of the first hole, and the lesion area; performing inverse kinematics solution on the robotic arm based on the end-effector pose; if there is a solution for the inverse kinematics, then determining the contact point between the line connecting the end-effector pose of the robotic arm and the lesion area and the body surface as the position of the second hole.

[0106] like Figure 5 As shown, after determining the lesion area, i.e., the lesion point, the surgical instrument fixed by the robotic arm needs to point towards the lesion point. Since the preset distance fixes the distance between the robotic arm and the endoscope arm, the end-effector pose of the robotic arm can be determined based on the position of the first aperture. Based on the end-effector pose, inverse kinematics is performed on the robotic arm according to this pose. Specifically, inverse kinematics calculation refers to calculating the solution that satisfies the constraint of maximizing the arm-to-arm distance between the distal mechanisms of the robotic arm and the preset distance requirement between the robotic arm and the endoscope arm during movement. If the inverse kinematics has a solution, it indicates that a suitable target joint angle of the robotic arm can be calculated, and thus the arm-to-arm distance value of the distal mechanisms can be obtained. With a solution in the inverse kinematics, the line connecting the end-effector pose of the robotic arm and the lesion area is further marked. The contact point between this line and the patient's surface is the aperture of the robotic arm.

[0107] In one embodiment, determining the end-effector pose of the robotic arm based on a preset distance, the position of the first hole, and the lesion area includes: determining the axis of the robotic arm cannula based on the preset distance, the position of the first hole, and the lesion area; selecting the degree of freedom of rotation of the robotic arm cannula around itself based on the axis of the robotic arm cannula to determine the midpoint of the joint stroke; and determining the end-effector pose of the robotic arm based on the midpoint of the joint stroke.

[0108] Continue to refer to Figure 5 Considering that if the preoperative positioning steps can ensure that the axis of the surgical arm (trocar cannula) points to the lesion, i.e., the distal instrument points to the lesion, the process of adjusting the distal mechanism of the surgical arm can be simplified. Under this concept, in this embodiment, after the preset distance between the surgical arm and the endoscope arm, the endoscope arm aperture position, and the axis of the surgical arm cannula are determined, the cannula is rotated around its own degree of freedom to select the midpoint of the joint stroke. At this time, the distal position of the surgical arm can be determined.

[0109] In one embodiment, there are multiple preset distances; obtaining the position of the second hole based on the preset distance and the position of the first hole includes: determining the positions of multiple initial second holes based on the multiple preset distances and the positions of the first holes; calculating the arm spacing between the lens-holding arm and the mechanical arm among the multiple initial second holes; and determining the hole with the largest arm spacing between the lens-holding arm and the mechanical arm as the second hole.

[0110] The preset distance between the robotic arm and the lens-holding arm can be set to multiple values ​​as needed. With multiple preset distances, multiple candidate second hole positions can be determined based on these distances and the position of the first hole, resulting in multiple initial second hole positions. In this case, the hole position with the largest arm-to-arm distance between the lens-holding arm and the robotic arm needs to be selected from these initial second hole positions as the final second hole position. Specifically, the largest arm-to-arm distance between the lens-holding arm and the robotic arm means the lowest probability of collision between them; the second hole position determined in this case is the optimal second hole position. Furthermore, the multiple preset distances can be iteratively calculated within the number of iterations k, using ΔP as the increment of the preset distance. The arm-to-arm distance corresponding to different preset distances is then selected as the second hole position, with the maximum arm-to-arm distance between the lens-holding arm and the robotic arm.

[0111] In one embodiment, calculating the arm spacing between the lens-holding arm and the mechanical arm in multiple initial second aperture positions includes: performing inverse kinematics on the mechanical arm based on the end pose of the mechanical arm to obtain the target pose of the mechanical arm; and calculating the arm spacing between the lens-holding arm and the mechanical arm in multiple initial second aperture positions based on the target pose of the mechanical arm and the pose of the lens-holding arm.

[0112] Based on the end-effector pose, inverse kinematics of the arm is solved. The target joint angles of the arm can then be calculated using this inverse kinematics, thus obtaining the target pose of the arm. To further determine the arm-to-arm distance between the lens-holding arm and the arm-holding device, the arm-to-arm distance needs to be calculated separately for multiple initial second aperture positions, based on the target pose of the arm-holding device, the pose of the lens-holding arm, and different initial second aperture positions.

[0113] To further explain in detail the technical principles and working process of the surgical robot system of this application, the process of determining the preset drilling position will be described below.

[0114] In one embodiment, the preset drilling position is obtained by: determining the surgical area based on the location information of the lesion area; determining the candidate drilling area based on the location and size of the surgical area and the structural parameters of the robotic arm in the surgical robot; selecting feasible drilling areas from the candidate drilling areas based on the first image information; and determining the preset drilling position in the feasible drilling area based on the structural parameters of the robotic arm in the surgical robot.

[0115] The location information of the lesion area refers to the specific location of the lesion point within the patient's body. For example, the lesion point could be the gallbladder, stomach, or other parts of the patient's abdominal cavity. Based on the specific location of the lesion point, the approximate surgical area can be determined, that is, the approximate location and size of the surgical area. Using the approximate location and size of the surgical area, combined with the length of the robotic arms in the surgical robot and the spacing between the robotic arms, candidate drilling areas can be determined. Then, areas suitable for drilling, i.e., feasible drilling areas, are selected from the candidate drilling areas. Based on the length of the robotic arms in the surgical robot and the spacing between the robotic arms, preset drilling positions are determined within the feasible drilling areas. The robotic arms include an endoscope-holding arm for holding the endoscope and a surgical instrument-holding arm for holding surgical instruments. Typically, there is one endoscope-holding arm and three surgical instrument-holding arms. This processing flow improves the accuracy of the preset drilling positions, thereby reducing surgical risks.

[0116] In one embodiment, selecting feasible punching regions from the candidate punching regions based on the first image information includes:

[0117] Based on the first image information, risky drilling locations are identified, including drilling locations that pass through bones, blood vessels, or nerves; risky drilling locations are eliminated from the candidate drilling area, and feasible drilling areas are selected.

[0118] Because the human body contains structures such as blood vessels, nerves, and bones, drilling holes in areas with these structures poses surgical risks. Therefore, these locations need to be eliminated. In this embodiment, locations with blood vessels, nerves, or bones are eliminated based on the first image information to determine the feasible drilling area. This reduces the drilling risk, thereby ensuring the reliability and safety of the surgery.

[0119] In one embodiment, determining the location of the pre-drilling hole in the lesion area within the feasible drilling area, based on the structural parameters of the robotic arm, includes:

[0120] Based on the structural parameters of the robotic arm, the length parameters of the endoscope-holding arm are obtained; according to the length parameters of the endoscope-holding arm and the preset depth-of-field constraints, the feasible drilling area of ​​the endoscope-holding arm is selected from the feasible drilling area; the drilling points of the endoscope-holding arm and the robotic arm are selected from the feasible drilling area of ​​the endoscope-holding arm; the drilling points of the endoscope-holding arm and the robotic arm are collected to obtain the pre-drilling position of the lesion area.

[0121] Since the precision requirements for the drilling position of the endoscope arm are lower than those for the surgical arm, the drilling position of the endoscope arm can be determined first. In this application, the preset depth of field of the endoscope is such that when the endoscope is placed in the abdominal cavity, all objects within the depth of field can be clearly imaged. This ensures the drilling precision of the endoscope arm, thereby ensuring that the specific location of the lesion can be clearly obtained through the endoscope during surgery, thus ensuring the smooth progress of the surgical procedure. Furthermore, the point closest to the lesion area within the feasible drilling area of ​​the endoscope arm can be selected as the drilling point for the endoscope arm (endoscope cannula). By selecting the point closest to the lesion area as the drilling point for the endoscope arm, the condition of the lesion can be well observed after the endoscope is inserted into the abdominal cavity.

[0122] In one embodiment, such as Figure 6 As shown, this application also provides another method for determining the drilling position of the robotic arm, which is as follows:

[0123] First image data of the lesion area is acquired, and the surgical area is determined based on the location information of the lesion area. Candidate perforation areas are determined based on the location and size of the surgical area and the structural parameters of the robotic arm in the surgical robot. Risk perforation locations are identified based on the first image information, including perforation locations that pass through bone, blood vessels, or nerves. Risk perforation locations are removed from the candidate perforation areas, and feasible perforation areas are selected. The length parameter of the endoscope-holding arm is obtained based on the structural parameters of the robotic arm in the surgical robot. Feasible perforation areas for the endoscope-holding arm are selected from the feasible perforation areas based on the length parameter of the endoscope-holding arm and preset depth-of-field constraints. Perforation points for the endoscope-holding arm and the robotic arm are selected within the feasible perforation areas for the endoscope-holding arm. The perforation points for the endoscope-holding arm and the robotic arm in the surgical robot are aggregated to obtain preset perforation positions. The perforation positions in the visual device coordinate system are obtained based on the converted lesion area location information and the preset perforation positions in the lesion area. The perforation positions in the visual device coordinate system are pushed to the visual device.

[0124] Based on the same inventive concept, such as Figure 7 As shown in the embodiments of this application, a surgical robot drilling guidance method is also provided, the method comprising:

[0125] S200: Acquire first image data of the lesion area;

[0126] S400: Based on the first image data, establish a three-dimensional model of the lesion area;

[0127] S600: Acquires second image data of the working environment of the scanning device, where the scanning device is the scanning device that generates first image data of the lesion area;

[0128] S800: Based on the three-dimensional model of the lesion area and the second image data, determine the position of the first aperture in the coordinate system of the visual device. The first aperture is the aperture of the arm holding the scope.

[0129] In one embodiment, determining the position of the first aperture in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data includes:

[0130] Obtain the first pose relationship between the scanning device and the lesion area; obtain the second pose relationship between the scanning device and the vision device; based on the first pose relationship and the second pose relationship, obtain the position information of the 3D model of the lesion area in the coordinate system of the vision device; based on the position information of the 3D model of the lesion area in the coordinate system of the vision device, determine the position of the first aperture in the coordinate system of the vision device.

[0131] In one embodiment, the above-mentioned surgical robot drilling guidance method further includes: obtaining a preset distance between a first hole position and a second hole position, wherein the second hole position is the hole position of the surgical arm; determining the end-effector pose of the surgical arm based on the preset distance and the lesion area; performing inverse kinematics solution on the surgical arm based on the end-effector pose; and if there is a solution for the inverse kinematics, determining the contact point between the line connecting the end-effector pose of the surgical arm and the lesion area and the body surface as the position of the second hole position.

[0132] The solution provided by the surgical robot punching guidance method in the above embodiments is similar to the solution described in the surgical robot system. Therefore, the specific limitations in one or more surgical robot punching guidance method embodiments can be found in the limitations in the surgical robot system above, and will not be repeated here.

[0133] Based on the same inventive concept, such as Figure 8 The embodiments of this application also provide a surgical robot drilling guidance device, the entire device specifically including a scanning component 820, a vision component 840 and a control component 860;

[0134] The scanning component 820 generates first image data of the lesion area and sends the first image data to the control component 860; the control component 860 uses the surgical robot drilling guidance method described above to generate the position of the first hole in the vision component 840.

[0135] In one embodiment, such as Figure 8 , Figure 9 As shown, the scanning component 820 includes an ultrasound probe 822, on which a visual target is provided; the visual target emits a cursor signal to illuminate the lesion area; the visual component 840 receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

[0136] The scanning component 820 includes an ultrasound probe 822, on which a visual target is provided; the visual target emits a cursor signal to illuminate the lesion area of ​​the scanned object; the visual component 840 receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

[0137] It should be noted that the visual target is a structure capable of emitting a cursor signal. Specifically, the optical signal can be an infrared cursor or light emitted by a light-emitting diode, etc. In this embodiment, the scanning component 820 is described as an ultrasound device, and in this embodiment, the lesion area includes the location of the lesion within the abdominal cavity and the corresponding area on the patient's body surface. The cursor signal emitted by the visual target illuminates the body surface corresponding to the lesion location of the scanned object. Specifically, when scanning the patient with the ultrasound device, the ultrasound device's probe rests against the patient's body surface, and at this time, the cursor signal emitted by the visual target on the probe illuminates the body surface. Since the depth camera can detect the three-dimensional information of the environment, it can receive the cursor signal and send the position information of the received cursor signal to the processing component.

[0138] In this embodiment, a visual target capable of emitting a cursor signal is set on the ultrasound probe 822. The cursor signal is received by the vision component 840, and the position information of the cursor signal is sent to the processing component to obtain the relative pose relationship between the scanning component 820 and the vision component 840. The pose relationship between the ultrasound probe 822 and the lesion area detected by the ultrasound probe 822 can be obtained.

[0139] In one embodiment, the control component is also used to control the scanning component 820 to emit a cursor signal to illuminate the lesion area of ​​the scanned object.

[0140] A cursor signal refers to a visible light marker signal, such as a bright red spot or visible light with a specific pattern. "Cursor signal illuminating the lesion area of ​​the scanned object" means that the cursor signal emitted by the scanning component 820 illuminates the body surface corresponding to the lesion area of ​​the scanned object. For example, when scanning a patient's abdomen, the red or patterned cursor emitted by the ultrasound target on the ultrasound probe 822 can illuminate the body surface corresponding to the patient's abdomen.

[0141] The visual component 840 is controlled to continuously track the cursor signal on the lesion area, ensuring that the cursor signal remains within the visual component 840's field of view. Controlling the visual component 840 to continuously track the cursor signal on the lesion area means controlling the visual component 840's field of view to ensure that the cursor signal is always within its field of view. For example, the visual component 840's field of view can be adjusted by controlling its movement or rotation, thereby ensuring that the cursor signal remains within its field of view.

[0142] Only when the cursor signal remains within the field of view of the vision component 840, which is also the field of view of the depth camera, can the vision component 840 consistently detect the specific position of the ultrasound probe 822 on the body surface. This ensures that the processing component can always obtain the pose relationship between the ultrasound probe 822 and the vision component 840. Therefore, by using the pose relationship between the ultrasound probe 822 and the lesion area, the pose relationship between the vision component 840 and the lesion area can be obtained. Thus, in this embodiment, the processing component controls the vision component 840 to continuously track the cursor signal on the lesion area, ensuring that the cursor signal remains within the field of view of the vision component 840. This allows the processing component to consistently obtain the pose relationship between the vision component 840 and the lesion area, thereby ensuring the smooth operation of the perforation guidance and guaranteeing the reliability of the surgical robot system.

[0143] In one embodiment, such as Figure 9 As shown, the vision component 840 includes a camera element 842, an angle adjustment element 844, and a mixed reality element 846. The camera element 842 receives a cursor signal on the lesion area and sends the position information of the cursor signal to the processing component. The angle adjustment element 844 is connected to the camera element 842 and is communicatively connected to the processing component. The processing component controls the angle adjustment element 844 to adjust the image tracking angle of the camera element 842. The mixed reality element 846 is communicatively connected to the processing component and displays the punching position in the coordinate system of the vision component 840 pushed by the processing component.

[0144] It should be noted that the camera element 842 can be the depth camera mentioned above, the angle adjustment element 844 can be the gimbal mentioned above, and the mixed reality element 846 can present the punch position information to the wearer of the MR helmet through mixed reality. The camera element 842, i.e., the depth camera, tracks the cursor signal emitted by the visual target on the ultrasonic probe 822 and sends the detected cursor signal's position information to the processing component. By connecting the camera element 842 to the angle adjustment element 844, the camera element 842 can change its position as the angle adjustment element 844 rotates, thereby adjusting the angle and position of the field of view. Furthermore, by communicating with the processing component, the angle adjustment element 844 can be controlled by the processing component, thereby adjusting the image tracking angle of the camera element 842.

[0145] For example, the gimbal has two degrees of freedom: pitch and yaw. When the doctor puts on the MR helmet, the doctor adjusts the head posture so that the cursor signal emitted by the visual target in the ultrasound probe 822 is within the field of view of the depth camera in the MR helmet. This process is the initialization process of the visual component 840. After initialization, the gimbal adjustment function of the depth camera is activated, and the depth camera can detect the cursor signal emitted by the visual target. After the processing component acquires the image of the cursor signal sent by the depth camera, it can identify the position and orientation of the cursor signal and control the depth camera to track the cursor signal in real time. This ensures that the processing component can always obtain the pose relationship between the visual component 840 and the ultrasound target, thereby ensuring the reliability of the drilling guidance operation.

[0146] In one embodiment, the processing component is also used to control the vision component 840 to track the cursor signal on the lesion area; if the cursor signal is not within the field of view of the vision component 840, the image tracking angle of the vision component 840 is adjusted.

[0147] Image tracking angle refers to the field of view angle of vision component 840. For example, when vision component 840 is used in an MR helmet, the depth camera on the MR helmet is used to detect the cursor signal. When the cursor signal is outside the lens of the depth camera, the cursor signal can be placed within the field of view of vision component 840 by adjusting the angle of the depth camera or adjusting the angle of the MR helmet.

[0148] If the image tracking angle exceeds the preset angle limit, a notification message indicating that the image is out of the field of view will be pushed.

[0149] The preset angle limit refers to the maximum or minimum value set in advance for adjusting the image tracking angle. When this value is exceeded, further adjustment of the image tracking angle will be impossible. The "out of field of view" warning message is a signal sent to the user to facilitate adjustment of the image tracking angle. For example, when the depth camera's gimbal reaches its travel limit, it will be unable to move further. In this case, the doctor needs to be reminded to adjust the angle of the MR helmet so that the cursor signal is within the depth camera's lens.

[0150] Understandably, the vision component 840 tracks the cursor signal in real time and sends the acquired image information to the processing component. The processing component identifies whether the image information contains a cursor signal. When a cursor signal is present in the image information and is located in the central area of ​​the visual field, the processing component only needs to control the vision component 840 to track the cursor signal in real time. When the processing component detects a cursor signal in the image information but it is not located in the central area of ​​the visual field, it calculates the deviation between the current cursor signal and the boundary of the central area and controls the angle adjustment element 844 to move, thereby moving the camera element 842 and adjusting the image tracking angle of the camera element 842 so that the cursor signal is located in the central area of ​​the visual field. When the angle adjustment element 844 reaches its limit and cannot move further during the movement controlled by the processing component, it indicates that the image tracking angle has exceeded the preset limit. At this point, the image tracking angle cannot be adjusted further. The processing component will push a notification message indicating that the cursor signal is outside the visual field to the vision component 840 to remind the doctor to adjust the direction of the vision component 840 worn on the head, thereby adjusting the field of vision of the vision component 840 so that the cursor signal is located in the central area of ​​the visual field. Specifically, the adjustment direction can be determined by the intersection of the cursor signal and the central area, and the adjustment distance of the angle adjustment element 844 can be determined by the distance between the cursor signal and the center of the field of view, thereby determining the adjustment direction and adjustment distance of the angle adjustment element 844.

[0151] For example, when the cursor signal is on the left edge of the central area, the angle adjustment element needs to be moved to the left to bring the cursor signal into the center of the field of view. If the angle adjustment element cannot be moved to the left, it means that the image tracking angle exceeds the preset angle limit. At this time, the processing component pushes a prompt message to the vision component, i.e., the MR helmet, so that the doctor can adjust the angle of his head to bring the cursor signal into the center of the field of view. In this way, it can be effectively ensured that the vision component can always track the cursor signal, thereby ensuring the smooth progress of the punching guidance work.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0153] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0154] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A surgical robot system, comprising a scope-holding arm, characterized in that, include: The scanning unit is used to acquire the first image data of the lesion area; The modeling unit is used to build a three-dimensional model of the lesion area based on the first image data; A vision unit is used to acquire second image data of the working environment of the scanning device, wherein the scanning device is a scanning device that scans and generates first image data of the lesion area; The processing unit is used to determine the position of the first hole in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data, wherein the first hole is the hole of the lens holder arm; The surgical robot system also includes a holding arm, and the processing unit is further configured to obtain the position of a second hole based on the position of the first hole, wherein the position of the second hole is the hole of the holding arm; The step of obtaining the position of the second hole based on the position of the first hole includes: Obtain the preset distance between the first hole position and the second hole position; The position of the second hole is obtained based on the preset distance and the position of the first hole. The number of preset distances is multiple; obtaining the position of the second hole based on the preset distance and the position of the first hole includes: Based on multiple preset distances and the position of the first hole, determine the positions of multiple initial second holes; Calculate the arm spacing between the lens-holding arm and the mechanical arm in multiple initial second hole positions; The hole with the largest arm-to-arm distance between the lens-holding arm and the mechanical arm is determined as the second hole.

2. The surgical robot system according to claim 1, characterized in that, The processing unit determines the position of the first aperture in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data, including: Obtain the first pose relationship between the scanning device and the lesion region; Obtain the second pose relationship between the scanning device and the vision device; Based on the first pose relationship and the second pose relationship, the position information of the three-dimensional model of the lesion area in the visual device coordinate system is obtained; Based on the position information of the three-dimensional model of the lesion area in the coordinate system of the visual device, the position of the first hole in the coordinate system of the visual device is determined.

3. The surgical robot system according to claim 2, characterized in that, The processing unit is also used to obtain the pre-drilling position of the lesion area; Determining the position of the first aperture in the visual device coordinate system based on the position information of the three-dimensional model of the lesion region in the visual device coordinate system includes: Based on the position of the lesion area in the coordinate system of the visual device according to the three-dimensional model, the pre-drilling position is corrected to obtain the position of the first hole in the coordinate system of the visual device.

4. The surgical robot system according to claim 1, characterized in that, The step of obtaining the position of the second hole based on the preset distance and the position of the first hole includes: The end-effector position is determined based on the preset distance, the position of the first hole, and the lesion area; The inverse kinematics of the arm are solved based on the end-effector pose of the arm. If the inverse kinematics has a solution, then the contact point between the line connecting the end pose of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

5. The surgical robot system according to claim 4, characterized in that, Determining the end effector position of the robotic arm based on the preset distance, the position of the first aperture, and the lesion area includes: The axis of the arm cannula is determined based on the preset distance, the position of the first hole, and the lesion area. Based on the axis of the arm sleeve, the midpoint of the joint stroke is determined by selecting the degree of freedom of rotation of the arm sleeve around itself. The end-effector position is determined based on the midpoint of the joint stroke.

6. The surgical robot system according to claim 1, characterized in that, The calculation of the arm spacing between the lens-holding arm and the mechanical arm in the multiple initial second aperture positions includes: Based on the end-effector pose of the arm, inverse kinematics solution is performed on the arm to obtain the target pose of the arm. The arm spacing between the lens-holding arm and the mechanical arm in multiple initial second aperture positions is calculated based on the target pose of the mechanical arm and the pose of the lens-holding arm.

7. A surgical robot drilling guidance method, characterized in that, The method includes: Acquire first-image data of the lesion area; Based on the first image data, a three-dimensional model of the lesion area is established; Acquire second image data of the working environment of the scanning device, wherein the scanning device is the scanning device that generates the first image data of the lesion area; Based on the three-dimensional model of the lesion area and the second image data, the position of the first aperture in the visual device coordinate system is determined, and the first aperture is the aperture of the lens arm; Obtain the preset distance between the first hole and the second hole, where the second hole is the hole of the holding arm; The position of the second hole is obtained based on the preset distance and the position of the first hole. The number of preset distances is multiple; obtaining the position of the second hole based on the preset distance and the position of the first hole includes: Based on multiple preset distances and the position of the first hole, determine the positions of multiple initial second holes; Calculate the arm spacing between the lens-holding arm and the mechanical arm in multiple initial second hole positions; The hole with the largest arm-to-arm distance between the lens-holding arm and the mechanical arm is determined as the second hole.

8. The surgical robot drilling guidance method according to claim 7, characterized in that, Determining the position of the first aperture in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data includes: Obtain the first pose relationship between the scanning device and the lesion region; Obtain the second pose relationship between the scanning device and the vision device; Based on the first pose relationship and the second pose relationship, the position information of the three-dimensional model of the lesion area in the visual device coordinate system is obtained; Based on the position information of the three-dimensional model of the lesion area in the coordinate system of the visual device, the position of the first hole in the coordinate system of the visual device is determined.

9. The surgical robot drilling guidance method according to claim 7, characterized in that, Also includes: The position of the end effector arm is determined based on the preset distance and the lesion area; The inverse kinematics of the arm are solved based on the end-effector pose of the arm. If the inverse kinematics has a solution, then the contact point between the line connecting the end pose of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

10. A surgical robot punching guide device, characterized in that, Includes scanning components, vision components, and control components; The scanning component generates first image data of the lesion area and sends the first image data to the control component; The control component uses the surgical robot drilling guidance method as described in any one of claims 7 to 9 to generate the position of the first hole in the vision component.

11. The surgical robot punching guide device according to claim 10, characterized in that, The scanning component includes an ultrasound probe, on which a visual target is provided; the visual target emits a cursor signal to illuminate the lesion area. The vision component receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

12. The surgical robot punching guide device according to claim 11, characterized in that, The visual component includes: The camera element acquires the cursor signal on the lesion area and sends the position information of the cursor signal to the control component; An angle adjustment element carries the camera element and is connected to the control component, wherein the control component controls the angle adjustment element to adjust the image tracking angle of the camera element; A mixed reality element displays the hole position in the visual component coordinate system pushed by the control component.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 7 to 9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 7 to 9.

Images