A surgical robot system and surgical navigation method

Abstract

This invention provides a surgical robot system and surgical navigation method. The system includes a workstation, a robotic arm, and a structured light module. The workstation includes a computing control center, a display device, and an input device. The structured light module includes a projection component and a camera component, and is connected to the robotic arm. The robotic arm adjusts its posture according to control commands from the workstation. The computing control center is configured to perform the following: acquire a 3D point cloud of the target object through the structured light module; register the 3D point cloud and a medical model, and generate a projected image corresponding to the current pose of the projection component; send the projected image to the structured light module so that the structured light module projects the image onto the target object through the projection component. This invention can project the information to be observed onto the surface of the target object, increasing information density. Users do not need to switch their gaze between the area of ​​interest and the screen, facilitating intuitive surgical planning and navigation.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a surgical robot system and surgical navigation method. Background Technology

[0002] When doctors develop appropriate treatment plans for patients or perform surgical procedures, they need to determine the location of certain tissue structures or the location of certain surgical planning components.

[0003] A common practice in existing technologies is to directly observe a medical image model on a screen and determine the location of relevant content by comparing it with a real patient. The drawbacks of this method are: 1. Doctors can only adjust the medical image model on the screen for observation, limiting their field of vision and hindering multi-person discussions; 2. The visual effect of the medical image model observed on the screen differs from that of a real patient in actual space, making it less intuitive; 3. The medical image model does not directly correspond to a real patient, making it difficult for doctors to accurately locate surgical-related content on a real patient.

[0004] Existing technology also includes "computer + projector" projection systems, where doctors project information from medical image models onto a real space. However, the matching accuracy between the projected information and the real space is poor, requiring doctors to repeatedly adjust the projector to optimize the projection effect. The drawbacks of this method are: 1. The projection process is complex and requires repeated manual adjustments; 2. The projection accuracy is low; 3. The projector affects the surgical space.

[0005] To address the aforementioned deficiencies in the prior art, this invention proposes a surgical robot system and surgical navigation method to solve or at least partially solve the aforementioned deficiencies in the prior art. Summary of the Invention

[0006] This invention provides a surgical robot system and surgical navigation method to solve the defects in the prior art where doctors cannot intuitively observe the three-dimensional model and need to repeatedly switch their gaze between the surgical site and the screen, which can easily lead to discontinuous surgical operations.

[0007] This invention provides a surgical robot system, comprising: a workstation, a robotic arm, and a structured light module;

[0008] The workstation includes a computing control center, a display device, and input devices;

[0009] The structured light module includes a projection component and a camera component, and the structured light module is connected to the robotic arm;

[0010] The robotic arm adjusts its posture according to the control commands from the workstation;

[0011] The computing control center is configured to perform the following:

[0012] S11. Obtain the three-dimensional point cloud of the target object through the structured light module;

[0013] S12. Register based on the three-dimensional point cloud and the medical model, and generate a projection image corresponding to the current pose of the projection component;

[0014] S13. Send the projected image to the structured light module so that the structured light module projects the image onto the target object through the projection component.

[0015] According to a surgical robot system provided by the present invention, the structured light module is detachably mounted to the robotic arm via a flange;

[0016] The system also includes a guidance module, which is installed on the flange after the structured light module is removed from the flange, for guiding the movement trajectory of surgical instruments.

[0017] According to a surgical robot system provided by the present invention, the system further includes a guidance module, which is integrated or detachably mounted to the robotic arm for guiding the movement trajectory of surgical instruments.

[0018] According to a surgical robot system provided by the present invention, S11 includes:

[0019] Control the robotic arm to move to the first pose;

[0020] The structured light module is controlled to acquire the three-dimensional point cloud in the first pose.

[0021] According to a surgical robot system provided by the present invention, S11 includes:

[0022] Control the robotic arm to move to at least two positions;

[0023] The structured light module is controlled to acquire point clouds at at least two poses;

[0024] Based on the robotic arm posture matrix corresponding to each posture, the point clouds collected from each posture are fused to obtain the three-dimensional point cloud.

[0025] According to a surgical robot system provided by the present invention, S12 includes:

[0026] The three-dimensional point cloud is registered with the medical model to obtain the transformation relationship between the point cloud coordinate system and the model coordinate system;

[0027] Based on the transformation relationship between the point cloud coordinate system and the model coordinate system, the projection component and the medical model are unified to the same coordinate system;

[0028] In the same coordinate system, a projected image corresponding to the model to be projected is generated in the current pose of the projection component; wherein, the medical model includes the model to be projected.

[0029] According to a surgical robot system provided by the present invention, S12 includes:

[0030] The 3D point cloud in the robotic arm coordinate system is registered with the medical model to obtain the transformation relationship between the robotic arm coordinate system and the model coordinate system;

[0031] Based on the transformation relationship between the robotic arm coordinate system and the model coordinate system, the medical model is transformed to the same coordinate system;

[0032] After each adjustment of the structured light module pose, based on the current pose of the projection component in the robotic arm coordinate system and the medical model in the same coordinate system, a projection image corresponding to the model to be projected is generated in the current pose of the projection component; wherein, the medical model includes the model to be projected.

[0033] According to a surgical robot system provided by the present invention, generating the projected image corresponding to the model to be projected in the current pose of the projection component includes:

[0034] The current pose of the projection component and the model to be projected are input to the image capture model to obtain the projected image; wherein the model to be projected is a patch model.

[0035] According to a surgical robot system provided by the present invention, the image acquisition model is a callable interface generated based on a three-dimensional model processing framework.

[0036] According to a surgical robot system provided by the present invention, generating the projected image corresponding to the model to be projected in the current pose of the projection component includes:

[0037] Based on the current pose of the projection component, the model to be projected is transformed into the coordinate system of the projection component; wherein, the model to be projected is a point cloud model;

[0038] The projected image is generated based on the intrinsic parameters of the projection component and the model to be projected in the coordinate system of the projection component.

[0039] According to a surgical robot system provided by the present invention, generating the projected image based on the intrinsic parameters of the projection component and the model to be projected in the coordinate system of the projection component includes:

[0040] The model to be projected in the coordinate system of the projection component is normalized to obtain a first image;

[0041] The first image is distorted according to the distortion coefficients of the projection component to obtain the second image;

[0042] The second image is mapped to the projection image plane according to the intrinsic parameters of the projection component, and the projection image is obtained through processing.

[0043] According to a surgical robot system provided by the present invention, generating the projected image corresponding to the model to be projected in the current pose of the projection component includes:

[0044] The structured light point cloud of the target object is acquired under the current pose of the projection component; wherein the structured light point cloud of the target object is acquired by the structured light module.

[0045] Based on the positions of the medical model and the projection component, a first point cloud corresponding to the projection model on the surface of the target object is generated; wherein, the projection model is a point cloud model.

[0046] Based on the first landing point cloud, a second landing point cloud corresponding to the region where the first landing point cloud is located is determined from the structured light point cloud of the target object;

[0047] The projection image is generated based on the second landing point cloud and the acquisition process of the structured light point cloud of the target object.

[0048] According to a surgical robot system provided by the present invention, generating a first point cloud of the projection model on the surface of a target object based on the positions of the medical model and the projection component includes:

[0049] If the model to be projected belongs to the surface point cloud of the target object, the model to be projected is taken as the first landing point cloud;

[0050] When the model to be projected contains non-surface points of the target object, the projection component is abstracted as a point and connected to each point in the model to be projected. The intersection of each line with the surface of the target object constitutes the second landing point cloud.

[0051] According to a surgical robot system provided by the present invention, the step of determining a second landing point cloud corresponding to the region where the first landing point cloud is located from the structured light point cloud of the target object based on the first landing point cloud includes:

[0052] A search tree is established for the structured light point cloud of the target object. For each point in the first landing point cloud, the nearest neighbor point is searched in the search tree, and the nearest neighbor points constitute the second landing point cloud.

[0053] According to a surgical robot system provided by the present invention, the step of generating the image to be projected based on the second landing point cloud and the acquisition process of the structured light point cloud of the target object includes:

[0054] Generate a blank two-dimensional image with the same resolution as the two-dimensional pattern projected during the structured light point cloud acquisition process of the target object;

[0055] For each point in the second landing point cloud, the coordinates of the corresponding two-dimensional point in the second landing point cloud are determined based on the correspondence between the pixels in the two-dimensional pattern projected during the structured light point cloud acquisition of the target object and the generated three-dimensional points. These coordinates are then added to the blank two-dimensional image to finally obtain the image to be projected.

[0056] According to a surgical robot system provided by the present invention, in step S12, after registration is completed, a verification step is also performed:

[0057] The projection component projects a pre-stored image onto the target object to display a projection mark on the target object;

[0058] Adjust the pose of the structured light module so that the projected mark reaches the first feature position of the target object;

[0059] The structured light module is used to re-acquire the 3D point cloud of the target object;

[0060] The location of the projection mark in the medical model is determined based on the re-acquired 3D point cloud, the pre-stored image, and the registration relationship, and then displayed through a display device.

[0061] According to a surgical robot system provided by the present invention, determining the position of the projection marker in the medical model based on the re-acquired 3D point cloud, the pre-stored image, and the registration relationship includes:

[0062] Based on the correspondence between the pixels in the two-dimensional pattern projected during the point cloud acquisition process of the re-acquired three-dimensional point cloud and the generated three-dimensional points, the three-dimensional points corresponding to the markers in the pre-stored image are determined.

[0063] Based on the transformation relationship between the registered robotic arm coordinate system and the model coordinate system, as well as the current robotic arm posture, the three-dimensional points are transformed into the model coordinate system.

[0064] The present invention also provides a surgical navigation method, comprising:

[0065] S11. Obtain the three-dimensional point cloud of the target object through the structured light module; the structured light module includes a projection component and a camera component;

[0066] S12. Generate a projection image based on the three-dimensional point cloud and the medical model;

[0067] S13. Send the projected image to the structured light module so that the structured light module projects the image onto the target object through the projection component.

[0068] The present invention also provides a surgical robot system, comprising: a workstation, a first robotic arm, a second robotic arm, a structured light module, and a guidance module;

[0069] The structured light module is installed or integrated on the first robotic arm and is used to collect three-dimensional point clouds and project positioning auxiliary information onto the target object through the projection component of the structured light module.

[0070] The guidance module is installed or integrated on the second robotic arm and is used to guide the movement trajectory of surgical instruments.

[0071] The workstation controls the posture of the first robotic arm and the second robotic arm, and generates a projection image containing the positioning assistance information based on the structured light point cloud and sends it to the structured light module.

[0072] The surgical robot system and surgical navigation method provided by this invention have at least one of the following beneficial effects:

[0073] 1. It can project the information (internal or surface) that needs to be observed onto the surface of the target object, increasing the information density. Users do not need to switch their gaze between the area of ​​interest and the screen, which facilitates intuitive discussion, teaching, surgical planning, surgical navigation, etc., and provides a wider field of view.

[0074] 2. The structured light module has the dual function of collecting point clouds and projecting point clouds. The processing module processes the data to generate the projected image. The entire projection system has a simple structure and saves costs.

[0075] 3. Patient registration based on structured light modules offers high registration accuracy and more precise projection;

[0076] 4. By combining the model point cloud and the 3D point cloud collected by the structured light module in the same coordinate system, a projection image is generated. There is no need to use a tracking device to detect whether the projection landing point meets the expectations, and there is no need to spend time adjusting the projector angle and correcting the projection effect, which improves the projection efficiency and projection accuracy.

[0077] 5. Provides multiple ways to generate "projected images" to further meet different projection accuracy and visual effect requirements;

[0078] 6. The location of the projection device can be easily moved according to actual space requirements;

[0079] 7. Registration accuracy was easily verified without adding any extra equipment. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0081] Figure 1 This is one of the structural schematic diagrams of a surgical robot system provided by the present invention;

[0082] Figure 2 This is one of the structural schematic diagrams of a guidance module for a surgical robot system provided by the present invention;

[0083] Figure 3 This is a schematic diagram illustrating the process of a surgical robot system generating a first landing point cloud, as provided by the present invention.

[0084] Figure 4 This is a schematic diagram of the process of acquiring point clouds based on structured light in a surgical robot system provided by the present invention;

[0085] Figure 5 This is a flowchart illustrating a surgical navigation method provided by the present invention;

[0086] Figure 6 This is one of the structural schematic diagrams of another surgical robot system provided by the present invention;

[0087] Figure 7 This is one of the schematic diagrams illustrating the projection effect of a point cloud projection method provided by the present invention;

[0088] Figure 8 This is the second schematic diagram of the projection effect of a point cloud projection method provided by the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0090] The following is combined Figures 1-8This invention describes a surgical robot system and surgical navigation method. The surgical robot system provided by this invention can be used alone or in conjunction with equipment such as microscopes, endoscopes, and exoscopes. The surgical robot system projects the information to be observed (internal or surface) onto the surface of the target object, providing positioning assistance for preoperative planning and craniotomy operations, or projecting relevant information during surgery to assist in adjusting the field of view of the microscope / endoscope / exterior endoscope and guide the surgeon's operation.

[0091] Figure 1 This is one of the structural schematic diagrams of a surgical robot system provided by the present invention, such as... Figure 1 As shown, the system includes a workstation 100, a robotic arm 200, and a structured light module 300;

[0092] Workstation 100 includes a computing control center 101, a display device 102, and an input device 103;

[0093] The structured light module 300 includes a projection component 301 and a camera component 302, and the structured light module 300 is connected to the robotic arm 200. Figure 1 (Connection status not shown in the image);

[0094] The robotic arm 200 adjusts its posture according to the control commands from the workstation 100;

[0095] The computing control center 101 is configured to perform the following:

[0096] S11. Obtain the 3D point cloud of the target object through the structured light module 300;

[0097] S12. Register based on the 3D point cloud and medical model, and generate a projection image corresponding to the current pose of the projection component;

[0098] S13. Send the projected image to the structured light module 300 so that the structured light module 300 projects the image onto the target object through the projection component 301.

[0099] Specifically, workstation 100 includes a computing control center 101, a display device 102, and an input device 103. The computing control center 101 is communicatively connected to both the display device 102 and the input device 103. Alternatively, the computing control center 101 can also communicate with other hospital equipment, such as magnetic resonance imaging (MRI) equipment, X-ray computed tomography (CT) equipment, or a database. The display device 102 displays the 3D images and software control interface generated by the computing control center 102. There can be more than one display device, and it can be other existing devices, such as LCD monitors, laptops, tablets, smartphones, etc. The display device 102 can also use a touchscreen, thus having both display and input functions. The input device 103 can be any input accessory, such as a foot switch, touchscreen, joystick, trackball, wireless mouse, mouse, keyboard, voice input port, or a combination thereof, allowing the user to input commands into the computing control center 101. If the display device 102 has input functionality, the input device 103 can be omitted (i.e., the display device 102 and input device 103 are integrated). Workstation 100 may also include other components such as wheels, fixtures, and handles to ensure easy movement / fixing of workstation 100 by the user. Robotic arm 200 is any robotic arm with at least six degrees of freedom, such as seven, eight, nine, or ten degrees of freedom. Workstation 100 can control the movement of robotic arm 200 and adjust its position and orientation so that structured light module 300 can acquire 3D point clouds at appropriate positions / angles or project images onto a target object at appropriate positions / angles.

[0100] The structure light module 300 is connected to the robotic arm 200. Specifically, the structure light module 300 can be integrated into the robotic arm 200, meaning the structure light module 300 and the robotic arm 200 are one unit. Alternatively, the structure light module 300 can be detachably installed onto the robotic arm 200. Figure 1 (Only the state before installation of the structured light module 300 is shown). The robotic arm 200 can move according to the control commands of the workstation 100 to move the structured light module 300 to the target position, so as to collect three-dimensional point clouds at the target position / angle, or project images onto the target object at the target position.

[0101] The computing control center 101 is configured to perform the following:

[0102] S11. Obtain the 3D point cloud of the target object through the structured light module 300;

[0103] Specifically, the computing control center 101 sends control commands to the structured light module 300, causing the structured light module 300 to collect data from the target object. The computing control center receives the three-dimensional point cloud of the target object from the structured light module 300, or receives the data transmitted by the structured light module 300 and processes it to generate a three-dimensional point cloud.

[0104] The target object here can be a real person, such as a patient, or a physical demonstration model, such as a mannequin used for teaching. Structured light point cloud acquisition is an active point cloud acquisition technology with advantages such as non-contact operation, large field of view, high precision (large amount of point cloud data), and good real-time performance. The structured light module 300 includes a projection component 301 and a camera component 302. The projection component 301 and the camera component 302 have a fixed relative positional relationship, which has been pre-calibrated. During the point cloud acquisition process, the structured light module 300 projects an coded two-dimensional pattern onto the target object through the projection component 301. The pattern will be deformed due to the inconsistent height of the object's surface. The camera component 302 captures the deformed pattern, and then decodes the pattern to obtain the correspondence between the object's surface and the projected pattern points. Based on the principle of triangulation, the three-dimensional point cloud of the target object's surface can then be calculated.

[0105] In one feasible approach, the structured light module 300 acquires a two-dimensional image of the target object and sends it to the computing control center 101. The computing control center 101 then decodes and calculates the image using the coded two-dimensional pattern projected onto the target object to generate a three-dimensional point cloud. In another feasible approach, the structured light module 300 has its own processing unit. It acquires two-dimensional image data of the target object and, using the coded two-dimensional pattern projected onto the target object, automatically processes and generates a three-dimensional point cloud. The computing control center 101 then directly receives and acquires this three-dimensional point cloud from the structured light module 300.

[0106] S12. Register based on the 3D point cloud and medical model, and generate a projection image corresponding to the current pose of the projection component;

[0107] Specifically, the position of the target object (in the point cloud coordinate system) can be determined based on the 3D point cloud. The 3D point cloud also includes the surface information of the target object, which can be matched with the medical model to establish a mapping relationship between the point cloud space and the model space (i.e., to achieve patient registration). The medical model is a 3D model built based on the medical image of the target object. The medical model contains content to be projected, such as a certain tissue structure of the target object (tumor, blood vessel, brain region, etc.), or surgical planning information in the medical model (needle insertion point, needle insertion direction, incision line, etc.). According to the aforementioned mapping relationship, the content to be projected can be mapped to the point cloud space, thereby generating a projected image corresponding to the content to be projected under the current pose (viewpoint) of the projection component 301.

[0108] S13. Send the projected image to the structured light module 300 so that the structured light module 300 projects the image onto the target object through the projection component 301.

[0109] Specifically, the computing control center 101 sends the projected image to the structured light module 300, which then projects the image onto the target object via the projection component 301. After projection, the target object can display the relevant content to be projected. For example, the marked and segmented brain regions can be projected onto the patient's head to facilitate the planning of puncture points that avoid important brain regions; blood vessels can be projected onto the exposed brain tissue after craniotomy to avoid blood vessel areas during surgery and reduce the risk of bleeding; and the incision line can be projected onto the skin surface for auxiliary positioning, etc., providing intuitive positioning information guidance for surgical planning or execution.

[0110] Figure 7 This is one of the schematic diagrams illustrating the projection effect of the point cloud projection method of the present invention. Figure 7 This illustrates the projection effect of projecting the location of the tumor area onto the patient's head; Figure 8 This is the second schematic diagram of the projection effect of a point cloud projection method provided by the present invention. Figure 8 This illustration shows the projection effect of a tumor and surrounding tissue onto a patient's head (different colors can be rendered for different tissues). Figure 7 , Figure 8 (This is a grayscale image; color effects are missing.)

[0111] The surgical robot system in this embodiment projects the information to be observed (internal or surface) onto the surface of the target object, increasing information density. Users do not need to switch their gaze between the area of ​​interest and the screen, facilitating intuitive discussions, teaching, surgical planning, and surgical navigation, and providing a wider field of view. Furthermore, the structured light module has a dual function: collecting point cloud data for registration and projecting relevant content onto the target object. The entire system has a simple structure, and during operation, there is no need to use tracking devices to check if the projection point matches expectations, nor is there a need to repeatedly adjust the projector angle or correct the projection effect, thus improving surgical efficiency.

[0112] In one feasible approach, the computing control center 101 may include a memory and a processor. The memory stores computer program instructions, and the processor loads the computer program instructions stored in the memory to execute the aforementioned configuration. Those skilled in the art also know that, besides implementing the computing control center 101 by loading a computer program using a general computer architecture, it is entirely possible to fabricate separate integrated circuit modules for each step of its execution, with each integrated circuit module serving as a component of the hardware component to implement the computing control center 101. Alternatively, integrated circuit modules can be fabricated based on some of the steps of its execution and combined with software modules implementing other steps to implement the computing control center 101. Therefore, the computing control center 101 can also be implemented as a hardware component, or as a combination of software modules and hardware components.

[0113] Based on the above embodiments, in one embodiment, the projection component 301 includes a light source, a lens group, a digital micromirror element, and a control module, and the imaging component 302 is a camera or a video camera.

[0114] Based on any of the above embodiments, in one embodiment, the structured light module 300 is integrated on the robotic arm 200, or is detachably mounted to the robotic arm 200 via a flange.

[0115] Specifically, the structure light module 300 can be integrated into the robotic arm 200, meaning that the structure light module 300 and the robotic arm 200 are one unit. The structure light module 300 can also be detachably installed onto the robotic arm 200 via a flange. This flange can be a flange specifically designed for installing the structure light module 300, or it can be a flange that is compatible with other accessories. For example, the structure light module, grippers, guide modules, etc., can be installed on the flange. After the structure light module 300 has collected the point cloud of the target object, it can be removed from the flange and replaced with grippers to hold surgical instruments, or replaced with a guide module to guide the movement trajectory of the surgical instruments.

[0116] Still refer to Figure 1 Based on any of the above embodiments, in one embodiment, the surgical robot system further includes a guidance module 400. The guidance module 400 can be integrated onto the robotic arm 200 or detachably mounted onto the robotic arm 200. The guidance module 400 can guide the movement trajectory of surgical instruments, such as guide wires, drills, electrodes, etc. Specifically, refer to... Figure 2 The guide module 400 includes an instrument channel 401 adapted to the shape of the surgical instruments. After the robotic arm 200 drives the guide module 400 to the target position, the movement trajectory of the surgical instruments is regulated through the instrument channel 401.

[0117] The surgical robot system in this embodiment is equipped with a guidance module, which can provide positioning and guidance for surgical instruments.

[0118] Based on any of the above embodiments, in one embodiment, the structured light module is detachably mounted to the robotic arm via a flange; the system also includes a guide module, which is installed on the flange after the structured light module is detached from the flange, for guiding the movement trajectory of surgical instruments.

[0119] In other words, the structured light module 300 and the guidance module 400 in the surgical robot system share a flange. After one module has finished using the flange, the other module can be installed in its place. For example, the structured light module 300 can be installed on the robotic arm 200 via the flange. After the structured light module 300 has finished collecting point cloud data / projecting the content to be displayed, the structured light module 300 can be removed and replaced with the guidance module 400 to provide positioning guidance for the surgical instruments.

[0120] Based on any of the above embodiments, in one embodiment, the robotic arm 200 includes a first flange and a second flange. The structured light module 300 is detachably mounted to the first flange, and the guide module 400 is detachably mounted to the second flange. That is, the structured light module 300 and the robotic arm can be mounted to different flanges. During use, while providing visual information guidance to the doctor through projection via the structured light module 300, the guide module 400 also provides positioning guidance for surgical instruments, facilitating the doctor's operation. Furthermore, the doctor can cross-reference and correct based on the positioning provided by the two methods, improving surgical accuracy.

[0121] Based on any of the above embodiments, in one embodiment, S11 includes:

[0122] Control the robotic arm 200 to move to the first position;

[0123] The structured light module 300 is controlled to acquire a three-dimensional point cloud in the first pose.

[0124] Specifically, the process of controlling the robotic arm 200 to move to the first pose can be as follows: the computational control center 101 outputs corresponding control commands based on user input instructions (e.g., coordinates, direction of movement), causing the robotic arm 200 to move to the first pose; for example, the user drags the robotic arm 200 in follow mode, and the computational control center 101 controls the robotic arm 200 to move to the first pose in accordance with the user's dragging action; for example, the computational control center 101 identifies the (rough) position of the target object based on the two-dimensional / three-dimensional data initially collected by the structured light module, plans the first pose based on the position of the target object, and then controls the robotic arm 200 to move to the first pose. Furthermore, the process of planning the first pose can be, for example, determining a suitable first pose based on the range and angle of the acquisition or projection of the structured light module 300 with the target object as the center, so as to facilitate the acquisition of the region of interest on the target object.

[0125] In this embodiment, the robotic arm is controlled to move the structured light module to the first pose, which facilitates the structured light module to collect the point cloud of the target object in the first pose, thereby improving the point cloud quality and thus improving the registration accuracy.

[0126] Based on any of the above embodiments, in one embodiment, S11 includes:

[0127] Control the robotic arm 200 to move to at least two positions;

[0128] The structured light module 300 is controlled to acquire point clouds in at least two poses;

[0129] Based on the robotic arm posture matrix corresponding to each posture, the point clouds collected from each posture are fused to obtain a 3D point cloud.

[0130] Specifically, the robotic arm 200 drives the structured light module 300 to move to multiple poses (at least two) in sequence, and controls the structured light module 300 to collect point clouds in the above multiple poses respectively. The point cloud collected in each pose is in the structured light coordinate system under that pose. It is also necessary to transform the point cloud into the robotic arm coordinate system according to the robotic arm posture matrix corresponding to the pose. Then, the point clouds corresponding to each pose in the robotic arm coordinate system are fused to obtain the three-dimensional point cloud of the target object.

[0131] In this embodiment, by acquiring point clouds of the target object from multiple poses, the three-dimensional point cloud of the target object becomes more comprehensive, reducing blind spots in the point cloud and improving registration accuracy. Correspondingly, the accuracy of projection onto the target object is also improved.

[0132] Based on any of the above embodiments, in one embodiment, S12 includes:

[0133] The 3D point cloud is registered with the medical model to obtain the transformation relationship between the point cloud coordinate system and the model coordinate system;

[0134] Based on the transformation relationship between the point cloud coordinate system and the model coordinate system, the projection component 301 and the medical model are unified to the same coordinate system;

[0135] In the same coordinate system, a projected image corresponding to the model to be projected is generated under the current pose of the projection component 301; wherein, the medical model contains the model to be projected.

[0136] Specifically, 3D point clouds contain the surface features of the target object, and medical models also contain the surface features of the target object. By registering the 3D point cloud with the medical model, the transformation relationship between the point cloud coordinate system and the model coordinate system can be obtained. For the point cloud coordinate system, the 3D point cloud can be a point cloud in the structured light coordinate system (e.g., a 3D point cloud acquired in a single pose). In this case, the point cloud coordinate system refers to the structured light coordinate system, and the above-mentioned transformation relationship is the transformation relationship between the structured light coordinate system and the model coordinate system. The 3D point cloud can also be a point cloud in the robotic arm coordinate system. In this case, the point cloud coordinate system refers to the robotic arm coordinate system. The point cloud can be a point cloud in a single pose or a point cloud acquired in multiple poses and fused in the robotic arm coordinate system. Correspondingly, the above-mentioned transformation relationship is the transformation relationship between the robotic arm coordinate system and the model coordinate system.

[0137] The current position of the structured light module 300 in the point cloud coordinate system can be determined, and the current pose of its projection component 301 can also be determined. By combining the transformation relationship between the point cloud coordinate system and the model coordinate system, the projection component 301 and the medical model can be unified to the same coordinate system, such as unified to the model coordinate system, unified to the point cloud coordinate system, or further transformed to the robotic arm coordinate system, the hospital bed coordinate system, the operating table coordinate system, etc.

[0138] Within the same coordinate system, the medical model can be "observed" from the current pose (viewpoint) of the projection component 301, resulting in a projected image of the projection component 301 at its current pose (viewpoint). The medical model includes a model to be projected, which is a data model corresponding to the aforementioned content to be projected, such as data models for tumors, blood vessels, brain regions, needle insertion points, needle insertion directions, and incision lines. Correspondingly, the generated projected image also contains the content to be projected. The process of generating the projected image takes into account the "viewpoint" of the projection component 301. Therefore, the projection component 301 can directly project this image to display the content to be projected at the corresponding position on the surface of the target object, providing information guidance for the doctor. Furthermore, no complex projection correction process is required after projection.

[0139] Based on any of the above embodiments, in one embodiment, S12 includes:

[0140] The 3D point cloud in the robotic arm coordinate system is registered with the medical model to obtain the transformation relationship between the robotic arm coordinate system and the model coordinate system;

[0141] Based on the transformation relationship between the robotic arm coordinate system and the model coordinate system, the medical model is transformed to the same coordinate system;

[0142] After each adjustment of the structured light module to the projection pose, a projection image corresponding to the model to be projected is generated in the current pose of the projection component 301 based on the current pose of the projection component 301 and the medical model in the same coordinate system; wherein, the medical model contains the model to be projected.

[0143] Specifically, the 3D point cloud in the robotic arm coordinate system is registered with the medical model in the model coordinate system to obtain the transformation relationship between the robotic arm coordinate system and the model coordinate system (i.e., to achieve patient registration). Based on this transformation relationship, the medical model can be transformed into the same coordinate system, such as the robotic arm coordinate system, the model coordinate system, or the bed coordinate system. The computational control center 101 can flexibly adjust the projection pose of the structured light module 300 by controlling the robotic arm 200. It can be understood that the projection pose does not need to be the same as the pose when the structured light module 300 acquires the point cloud; the projection pose can be flexibly adjusted.

[0144] The computational control center 101 can determine the current position of the structured light module 300 based on the projection pose of the robotic arm 200, and thus determine the current pose of the projection component 301 in the robotic arm coordinate system. After each adjustment to a projection pose, the model to be projected in the medical model is used as the "observation object" in the same coordinate system, and the current pose of the projection component 301 is used to determine the "viewing angle," thereby generating a projection image corresponding to the model to be projected.

[0145] The surgical robot system in this embodiment can flexibly adjust its position for projection after patient registration, reducing navigation blind spots. It can also flexibly adjust the projection position, freeing up operating space for doctors and improving surgical efficiency.

[0146] Furthermore, after registration is completed, the location of the target object can be identified, and positional constraints can be added to the structured light module based on the location of the target object, so that the structured light module projects within a range where the distance to the target object is less than a preset distance threshold; the posture of the target object can also be identified, and directional constraints can be added to the structured light module in combination with the surgical planning content, so that the structured light module projects within a set directional range of the target object, such as projecting onto the top of the head, or projecting to avoid locations where craniotomy is not possible.

[0147] Based on any of the above embodiments, in one embodiment, generating a projected image corresponding to the model to be projected in the current pose of the projection component 301 includes:

[0148] The current pose of the projection component 301 and the model to be projected are input into the image capture model to obtain the projected image.

[0149] Specifically, the model to be projected is a patch model, which consists of a series of patches, each composed of polygons. The curved shape of a patch is defined by a set of vertices of the polygons. Patch models can construct "smooth" models with very little data, and the surfaces are renderable geometric objects. Rendering the surfaces (adding color, texture, etc.) can make the viewing effect more realistic. The image capture model is a tool model for processing the 3D model, capable of outputting 2D images based on the 3D model and defined parameters. Specifically, the image capture model can acquire 2D images (i.e., projected images) of the model to be projected under the current pose (viewpoint) of the projection component 301.

[0150] Based on any of the above embodiments, in one embodiment, the image capture model is a callable interface generated based on a 3D model processing framework.

[0151] Specifically, the image capture model is used to process 3D models and generate 2D pixel images (projected images) in the required pose. The image capture model can also perform lighting and shadow processing to make the generated projected image more realistic. The image capture model can be a callable interface generated based on a 3D model processing framework, such as VTK, Blender, Cinemachine, etc. By setting the image capture model as a callable interface, data dependencies between different programs can be reduced, improving application running efficiency.

[0152] Based on any of the above embodiments, in one embodiment, generating a projected image corresponding to the model to be projected in the current pose of the projection component 301 includes:

[0153] Based on the current pose of the projection component 301, the model to be projected is transformed into the coordinate system of the projection component 301;

[0154] Based on the intrinsic parameters of the projection component 301 and the model to be projected in the coordinate system of the projection component, a projected image is generated.

[0155] Specifically, in this embodiment, the model to be projected is a point cloud model, which can be obtained by extracting point clouds from the content to be projected in the medical model (brain regions, blood vessels, tumors, puncture paths, etc.). Furthermore, the model to be projected can also be a point cloud extracted from the surface of the content to be projected, such as the surface point cloud of a tumor area or the surface point cloud of a blood vessel. Extracting surface point clouds can reduce the amount of data processing while ensuring projection quality and improving data processing efficiency. It is understood that point-like content to be projected can be directly used as the model to be projected, such as needle insertion points or bone screw implantation points; linear content to be projected still needs to have point clouds extracted, such as point clouds extracted from anatomical lines.

[0156] The model to be projected is point cloud data described by coordinates. The model to be projected in the same coordinate system is determined, and the current pose of the projection component 301 is determined. Based on this, the model to be projected (point cloud) is transformed to the coordinate system of the projection component 301, and then the projection image is generated by combining the intrinsic parameters of the projection component 301 with the camera imaging model.

[0157] In this embodiment, the point cloud model to be projected is used as the processing object. The projected image is generated simply and efficiently by coordinate transformation and combining the intrinsic parameters of the projection component 301.

[0158] Based on any of the above embodiments, in one embodiment, generating a projected image according to the intrinsic parameters of the projection component 301 and the model to be projected in the coordinate system of the projection component 301 includes:

[0159] The model to be projected in the coordinate system of the projection component is normalized to obtain the first image;

[0160] The first image is distorted according to the distortion coefficients of the projection component 301 to obtain the second image;

[0161] The second image is mapped to the projection image plane according to the intrinsic parameters of the projection component 301, and the projection image is obtained through processing.

[0162] Specifically, the model to be projected in the coordinate system of the projection component is normalized along the principal axis (Z-axis) of the projection component 301 to convert the point cloud into a two-dimensional image, thus obtaining the first image mentioned above. Furthermore, the normalization process can be implemented according to the following formula:

[0163]

[0164] in, These are the coordinates of a point in the model to be projected within the coordinate system of the projection component. These are the coordinates of the corresponding point in the first image.

[0165] Since distortion occurs during the projection process of the projection component 301, further distortion transformation of the first image can eliminate the distortion caused by the projection component 301 during projection onto the target object. Preferably, the distortion transformation process can be implemented according to the following formula:

[0166] x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 );

[0167] y′ ＝ y(1+k1r 2 +k2r4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 );

[0168] Where x and y are the coordinates of a point in the first image, x′ and y′ are the coordinates of a point in the second image obtained after distortion transformation, and r 2 =x 2 +y 2 k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters. These distortion parameters are obtained by pre-calibrating the projection component 301.

[0169] After obtaining the second image, the second image is converted to the projection image plane by combining the focal length of the projector component 301 to obtain the corresponding physical image. Then, the physical image is converted into a pixel image by combining the resolution of the projection component 301. The pixel image is easy for the projection component 301 to process and project based on.

[0170] In this embodiment, the projection image of the projection component under the current pose (viewpoint) is obtained simply and efficiently by transforming and processing the coordinates of the model to be projected (point cloud), and the processing speed is relatively fast.

[0171] Based on any of the above embodiments, in one embodiment, generating a projected image corresponding to the model to be projected in the current pose of the projection component 301 includes:

[0172] The structured light point cloud of the target object is acquired under the current pose of the projection component 301; wherein, the structured light point cloud of the target object is acquired by the structured light module.

[0173] Based on the positions of the medical model and the projection component 301, a point cloud of the first landing point of the model to be projected on the surface of the target object is generated; wherein, the model to be projected is a point cloud model;

[0174] Based on the first landing point cloud, determine the second landing point cloud corresponding to the area where the first landing point cloud is located from the structured light point cloud of the target object;

[0175] Based on the second landing point cloud, combined with the acquisition process of the structured light point cloud of the target object, an image to be projected is generated.

[0176] Specifically, the computing control center 101 can control the structured light module to acquire the structured light point cloud of the target object in the current projection pose (the current projection pose of the structured light module corresponds one-to-one with the current pose of the projection component 301). It is understood that the 3D point cloud of the target object has already been acquired for registration with the medical model (achieving patient registration). If the current projection pose of the structured light module is the same as the pose of the previously acquired 3D point cloud of the target object, then the structured light point cloud can be directly retrieved and reused without having to acquire it again.

[0177] Reference Figure 3 In the same coordinate system mentioned above, the model to be projected in the medical model is "observed" at the corresponding position of the projection component 301, and the first landing point cloud corresponding to the model to be projected on the surface of the target object is determined. Based on the first landing point cloud, a second landing point cloud overlapping with the area where the first landing point cloud is located is determined in the aforementioned structured light point cloud. It can be understood that the second landing point cloud is a part of the structured light point cloud.

[0178] Figure 4 This is a schematic diagram of the point cloud acquisition process based on structured light, referencing... Figure 4 The projection component 301 projects a two-dimensional coded image A onto the surface of the target object. Then, the camera component 302 captures an image of the target object to obtain a two-dimensional image B. Based on the two-dimensional point p' in image A, the two-dimensional point p'' in image B, and the calibration relationship between the projection component 301 and the camera component 302, the three-dimensional point p in the point cloud C can be calculated. During this acquisition process, there is a correspondence between the two-dimensional point p', the two-dimensional point p'', and the three-dimensional point p. For each point in the second landing point point cloud, its corresponding two-dimensional point in the projected image is determined according to the above correspondence, and finally, the projected image can be determined.

[0179] In this embodiment, the landing point cloud is accurately determined, and the projected image is generated efficiently and accurately. The process of generating the projected image under the current pose (viewpoint) of the projection component does not require complex calculations based on the projector parameters (extrinsic and intrinsic parameters), which improves data processing efficiency and results in higher projection accuracy and more accurate positioning.

[0180] Based on any of the above embodiments, in one embodiment, generating a first landing point cloud of the model to be projected on the surface of the target object according to the position of the medical model and the projection component 301 includes:

[0181] If the model to be projected is a surface point cloud of the target object, the model to be projected will be used as the first landing point point cloud.

[0182] When the model to be projected contains non-surface points of the target object, the projection component 301 is abstracted as a point and connected to each point in the model to be projected. The intersection of each connection with the surface of the target object constitutes the second landing point cloud.

[0183] Specifically, determining the first landing point cloud is the process of "projecting" the model to be projected onto the surface of the target object. If the model to be projected is itself a surface point, such as an incision line or a needle insertion point, it can be directly used as the first landing point cloud. If the model to be projected contains internal points, such as tumor point clouds or blood vessel point clouds, then each point in the model needs to be connected to the abstract point (such as the center or optical center) of the projection component 301, and the intersection of each connection with the surface of the target object is used as the second landing point cloud.

[0184] In this embodiment, the landing point cloud is accurately determined, and the projected image is generated efficiently and accurately. The projected image under the current pose (viewpoint) of the projection component can be generated without complex calculations based on the projector parameters (extrinsic and intrinsic parameters), which improves data processing efficiency and provides higher projection accuracy and more accurate positioning.

[0185] Based on any of the above embodiments, in one embodiment, determining a second landing point cloud corresponding to the region where the first landing point cloud is located from the structured light point cloud of the target object includes:

[0186] A search tree is established for the structured light point cloud of the target object. For each point in the first landing point cloud, the nearest neighbor point is searched in the search tree. The nearest neighbor points constitute the second landing point cloud.

[0187] Specifically, a search tree (such as KD tree, KDB tree, BSP tree, octree, etc.) is established for the structured light point cloud of the target object to improve the search efficiency of 3D points. For each point in the first landing point cloud, the nearest neighbor point is searched in the search tree. The nearest neighbor points corresponding to each point in the first landing point cloud constitute the second landing point cloud.

[0188] In this embodiment, a search tree is established on the structured light point cloud of the target object, and the nearest neighbor point corresponding to each point in the first landing point point cloud is searched in the search tree. The landing point corresponding to the model to be projected is accurately and efficiently determined from the structured light point cloud of the target object.

[0189] Based on any of the above embodiments, in one embodiment, generating a projection image based on the second landing point cloud and the acquisition process of the structured light point cloud of the target object includes:

[0190] Generate a blank 2D image with the same resolution as the 2D pattern projected during the acquisition of the structured light point cloud of the target object;

[0191] For each point in the second landing point cloud, based on the correspondence between the pixels in the two-dimensional pattern projected during the structured light point cloud acquisition of the target object and the generated three-dimensional points, the coordinates of the corresponding two-dimensional point in the second landing point cloud are determined and added to the blank two-dimensional map, thus obtaining the image to be projected.

[0192] Specifically, a blank 2D image with the same resolution as the 2D pattern projected by structured light is generated to facilitate the addition of pixels of the image to be projected. The second landing point cloud is a part of the structured light point cloud of the target object. For each point in the second landing point cloud, the coordinates of the corresponding 2D point in the second landing point cloud are determined according to the correspondence between the pixels of the projected 2D pattern and the generated 3D points during the point cloud acquisition process, and added to the blank 2D image. The 2D image obtained after all points have been added is the projected image.

[0193] This embodiment generates projected images efficiently and accurately, without requiring complex calculations based on the projection component parameters (extrinsic and intrinsic parameters) to generate a two-dimensional image of the projection component in its current pose (viewpoint), thus improving data processing efficiency.

[0194] Based on any of the above embodiments, in one embodiment, S12 further includes:

[0195] When the point to be projected is highlighted in the projected image; and / or,

[0196] When the projection needs to be of a contour, connect the outer points in the projected image; and / or,

[0197] When the area to be projected is a region, morphological processing is performed on the projected image.

[0198] Specifically, if the projection needs to be of a point, such as a needle insertion point or a bone screw implantation point, the point to be projected in the image to be projected is highlighted, for example, by displaying a cross centered on the point or enlarging the point. If the projection needs to be of an outline, such as a tumor outline or a brain region outline, the outer points in the image to be projected are connected, and the outer outline is projected onto the surface of the target object. If the projection needs to be of an entire region, morphological processing is performed on the image to be projected. Specifically, the first landing point cloud is used to determine the second landing point cloud by searching for the nearest neighbor. This process results in the pixels in the second landing point cloud not falling precisely and continuously in the blank 2D image. There may be missing pixels in the pixel area. Image morphological processing can supplement the missing pixels in the area and further improve the projection effect.

[0199] Based on any of the above embodiments, in one embodiment, in S12, after registration is completed, a verification step is also performed:

[0200] The projector 301 projects a pre-stored image onto the target object to display a projection mark on the target object.

[0201] Adjust the pose of the structured light module 300 so that the projected mark reaches the first feature position of the target object;

[0202] The 3D point cloud of the target object is acquired again using the structured light module 300;

[0203] The location of the projection mark in the medical model is determined based on the re-acquired 3D point cloud, the pre-stored image, and the registration relationship, and then displayed through the display device 102.

[0204] Specifically, after completing the registration by obtaining the transformation relationship between the robotic arm coordinate system and the model coordinate system, the registration accuracy can be further verified to improve surgical safety. During the verification process, the pre-stored image in the workstation 100 is first projected onto the target object (patient) using the projection component 301. For example, a pre-stored image marked with circles, crosses, stars, etc., is projected onto the patient's face. Then, the pose of the structured light module 300 is adjusted by adjusting the robotic arm so that the projected markers reach the first feature position of the target object. For example, the projected circular marker is adjusted to the tip of the nose, or the projected cross marker is adjusted to the corner of the eye. Then, the structured light module 300, after this adjustment, again acquires the 3D point cloud of the target object. The acquired 3D point cloud corresponds to the pixels in the pre-stored image, thus allowing the identification of the corresponding 3D points in the 3D point cloud. These points are then transformed into the model coordinate system based on the registration relationship and displayed in the medical model. The doctor can observe the position of the projected markers in the medical module to confirm whether they are displayed at the corresponding "first feature position," thereby verifying the registration accuracy.

[0205] This embodiment efficiently and accurately verifies registration accuracy without adding additional equipment.

[0206] Based on the above embodiments, in one embodiment, determining the position of the projection marker in the medical model according to the re-acquired 3D point cloud, pre-stored images, and registration relationships includes:

[0207] Based on the correspondence between the pixels in the projected 2D pattern and the generated 3D points during the point cloud acquisition process of the re-acquired 3D point cloud, the 3D points corresponding to the markers in the pre-stored image are determined.

[0208] Based on the transformation relationship between the robot arm coordinate system and the model coordinate system obtained from registration, and the current robot arm posture, the 3D points are transformed into the model coordinate system.

[0209] Specifically, referring to the description in the previous embodiment, during the structured light point cloud acquisition process, there is a correspondence between the pixels in the projected 2D image and the generated 3D points. The pre-stored image is an image that conforms to the projection resolution of the projection component 301. Based on this correspondence, the 3D point corresponding to the pre-stored image in the 3D point cloud acquired again can be determined. Combined with the robotic arm posture matrix, the 3D point can be transformed into the robotic arm coordinate system. Furthermore, by combining the transformation relationship between the registered robotic arm coordinate system and the model coordinate system, the 3D point can be further transformed into the model coordinate system, thus obtaining the position of the projected mark in the medical model.

[0210] In this embodiment, the position of the projection mark in the medical model is conveniently determined by acquiring a 3D point cloud after the pose is adjusted, which makes it convenient for users to compare the actual projection mark to verify the registration accuracy.

[0211] This invention also provides a surgical navigation method, which can be referred to in conjunction with the aforementioned surgical robot system. Figure 5 This is a flowchart illustrating a surgical navigation method provided by the present invention, as shown below. Figure 5 As shown, the method includes:

[0212] S11. Obtain the 3D point cloud of the target object through the structured light module; the structured light module includes a projection component and a camera component;

[0213] S12. Register based on the 3D point cloud and medical model, and generate a projection image corresponding to the current pose of the projection component;

[0214] S13. Send the projected image to the structured light module so that the structured light module can project the image onto the target object through the projection component.

[0215] Specifically, the structured light module includes a projection component and a camera component. The structured light module can acquire the three-dimensional point cloud of the target object based on the structured light principle. The projection component projects an encoded two-dimensional pattern onto the target object. The pattern will be deformed due to the inconsistent height of the object's surface. The camera component captures the deformed pattern and then decodes the pattern to obtain the correspondence between the object's surface and the projected pattern points. Based on the triangulation principle, the three-dimensional point cloud of the target object's surface can then be obtained.

[0216] A 3D point cloud contains visually visible surface information of the target object. A medical model is a 3D model built based on medical images of the target object. This model contains content to be projected, such as a specific tissue structure of the target object (tumor, blood vessel, brain region, etc.), or surgical planning information (needle insertion point, needle insertion direction, incision line, etc.). A mapping relationship between real space and model space can be established based on the 3D point cloud. Based on this mapping, the projector "observes" the model to be projected (tumor, blood vessel, brain region, needle insertion point, needle insertion direction, incision line, etc.) within the medical model at its current pose, generating a projected image. This projected image is then sent to the structured light module, which controls the projector component to project the image onto the target object. After projection, the corresponding content (tumor, blood vessel, brain region, needle insertion point, needle insertion direction, incision line, etc.) is displayed on the surface of the target object. This projected content corresponds to the tissue structure / surgical planning information of the target object, providing doctors with information for location and navigation.

[0217] The surgical navigation method in this embodiment projects the information to be observed (internal or surface) onto the surface of the target object, increasing information density. Users do not need to switch their gaze between the area of ​​interest and the screen, facilitating intuitive discussions, teaching, surgical planning, and surgical navigation, and providing a wider field of view. Furthermore, the structured light module has a dual function: collecting point cloud data for registration and projecting relevant content onto the target object. The entire system has a simple structure, and during operation, there is no need to use tracking devices to check if the projection point matches expectations, nor is there a need for time-consuming adjustments to the projector angle or corrections to the projection effect, thus improving surgical efficiency.

[0218] This invention also provides a surgical robot system, which can be referred to in conjunction with the aforementioned surgical robot system and surgical navigation method. Figure 5 This is a flowchart illustrating a surgical navigation method provided by the present invention, as shown below. Figure 5 As shown, the method includes:

[0219] The present invention also provides a surgical robot system, comprising: a workstation 100, a first robotic arm 500, a second robotic arm 600, a structured light module 300, and a guidance module 400;

[0220] The structured light module 300 is installed or integrated on the first robotic arm 500 to collect structured light point clouds and project positioning auxiliary information onto the target object through the projection component of the structured light module 300.

[0221] The guidance module 400 is installed or integrated on the second robotic arm 600 to guide the movement trajectory of surgical instruments.

[0222] Workstation 100 controls the posture of the first robotic arm 500 and the second robotic arm 600, and generates a projection image containing positioning assistance information based on the structured light point cloud and sends it to the structured light module 300.

[0223] Specifically, the structured light module 300 includes a projection component and a camera component. The structured light module 300 is mounted or integrated onto the first robotic arm 500. The workstation 100 can control and adjust the posture of the first robotic arm 500, moving the structured light module 300 to one or more positions to acquire 3D point clouds of the target object. The workstation 100 can register the acquired 3D point clouds with the medical model of the target object, obtaining a mapping relationship between the model space and the real space, thus completing the patient registration process. Based on the mapping relationship, the workstation 100 also "observes" the model to be projected (tumor, blood vessels, brain regions, needle insertion point, needle insertion direction, incision line, etc.) in the medical model at the corresponding position and viewpoint of the projection component, generating a projection image. This projection image is then sent to the structured light module 300, which controls the projection component to project the image onto the target object. After projection, corresponding positioning assistance information (corresponding to the aforementioned model to be projected) can be displayed on the surface of the target object. This projection-displayed positioning assistance information corresponds to the tissue structure / surgical planning information in the target object, providing doctors with information for positioning and navigation.

[0224] The guidance module 400 is installed or integrated on the second robotic arm 600. The guidance module 400 can guide the movement trajectory of surgical instruments, for example, by guiding the surgical instruments through the surgical instrument channel on the guidance module 400. The workstation 100 has already collected point clouds through the structured light module 300 and completed patient registration. Based on this, it can map the surgical plan onto the real space, determine the target position that the guidance module 400 needs to reach, and control the second robotic arm 600 to adjust its posture so that the guidance module 400 reaches the target position, providing positioning guidance for the surgical instruments and improving the accuracy of the surgical operation.

[0225] The surgical robot system provided by this invention can flexibly collect point clouds from multiple angles using a structured light module driven by a first robotic arm, reducing blind spots in point cloud acquisition and improving registration accuracy. It can also accurately project positioning assistance information onto the target object using the structured light module, providing information guidance for the surgeon. A second robotic arm drives a guidance module to the target position, providing positioning guidance for surgical instruments and improving surgical precision. The high degree of freedom of movement of both the structured light module and the guidance module reduces mutual interference. Furthermore, the surgeon can perform cross-verification and calibration based on the projected positioning assistance information and the actual guidance effect of the guidance module, further improving surgical precision.

[0226] Based on the above embodiments, in one embodiment, the first robotic arm 500 and the second robotic arm 600 are mounted on the same carrier or fixing device. For example, both the first robotic arm 500 and the second robotic arm 600 are fixed to an operating table or hospital bed; or, for example, both the first robotic arm 500 and the second robotic arm 600 are fixed to a movable and lockable carrier (e.g., Figure 1 (The trolley in the middle).

[0227] Based on any of the above embodiments, in one embodiment, the first robotic arm 500 and the second robotic arm 600 are mounted on different carriers or fixing devices. For example, the first robotic arm 500 is mounted on one movable and lockable carrier, and the second robotic arm 600 is mounted on another movable and lockable carrier; or, for example, the first robotic arm 500 is mounted on a movable and lockable carrier, which facilitates the flexible projection of positioning assistance information onto the target object from multiple positions and angles, while the second robotic arm 600 is fixed to an operating table or hospital bed, improving the stability of the robotic arm, reducing the distance to the target object, and facilitating the operation of surgical instruments by the doctor with the assistance of the guidance module.

[0228] Based on any of the above embodiments, in one embodiment, the workstation 100 is integrated into the carrier of the first robotic arm 500 or into the carrier of the second robotic arm 600.

[0229] Specifically, workstation 100 is integrated into the carrier of a robotic arm. Workstation 100 is connected to another robotic arm via wired or wireless means, preferably wirelessly. Of course, the first robotic arm 500 and the second robotic arm 600 can be mounted on the same carrier (see reference). Figure 6 Workstation 100 is integrated into this vehicle.

[0230] Based on any of the above embodiments, in one embodiment, the workstation 100 is an independent device, and the workstation 100 is connected to the first robotic arm 500 and the second robotic arm 600 by wire or wireless means. Preferably, the workstation 100 is connected to the first robotic arm 500 and the second robotic arm 600 by wireless means.

[0231] Based on any of the above embodiments, in one embodiment, the surgical robot system further includes: a display device for displaying three-dimensional images and a software control interface.

[0232] There can be more than one display device, which can be an LCD monitor, a laptop, a tablet, a smartphone, etc.

[0233] Based on any of the above embodiments, in one embodiment, the surgical robot system further includes: an input device for receiving user-inputted instructions. The input device may be, for example, a foot switch, a touchscreen, a joystick, a trackball, a wireless mouse, a keyboard, or a voice input port.

[0234] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical robotic system, characterized by, The system comprises: a workstation, a mechanical arm, and a structured light module; the workstation comprises a computing control center, a display device, and an input device; the structured light module comprises a projection assembly and a camera assembly, and is connected to the mechanical arm; the mechanical arm adjusts its posture according to the control instruction of the workstation; the computing control center is configured to perform the following steps: S11. Obtain a three-dimensional point cloud of a target object by using the structured light module; S12. Register the three-dimensional point cloud and a medical model, and generate a projection image corresponding to the current posture of the projection assembly; the generation of the projection image corresponding to the current posture of the projection assembly comprises: obtain a structured light point cloud of the target object under the current posture of the projection assembly; wherein the structured light point cloud of the target object is obtained by the structured light module; generate a first landing point cloud corresponding to the target object surface of a to-be-projected model according to the medical model and the position of the projection assembly; wherein the medical model contains the to-be-projected model, and the to-be-projected model is a point cloud model; determine a second landing point cloud corresponding to the region where the first landing point cloud is located from the structured light point cloud of the target object according to the first landing point cloud; generate the projection image according to the second landing point cloud and the acquisition process of the structured light point cloud of the target object; S13. Send the projection image to the structured light module, so that the structured light module projects the target object through the projection assembly.

2. The surgical robotic system of claim 1, wherein, The structured light module is detachably mounted to the mechanical arm through a flange; the system further comprises a guiding module, which is mounted to the flange after the structured light module is detached from the flange, and is used to guide the movement trajectory of a surgical instrument.

3. The surgical robotic system of claim 1, wherein, S12 comprises: register the three-dimensional point cloud and the medical model to obtain the conversion relationship between the point cloud coordinate system and the model coordinate system; unify the projection assembly and the medical model to the same coordinate system according to the conversion relationship between the point cloud coordinate system and the model coordinate system; generate the projection image corresponding to the to-be-projected model under the current posture of the projection assembly in the same coordinate system.

4. The surgical robotic system of claim 1, wherein, S12 comprises: register the three-dimensional point cloud in the mechanical arm coordinate system and the medical model to obtain the conversion relationship between the mechanical arm coordinate system and the model coordinate system; convert the medical model to the same coordinate system according to the conversion relationship between the mechanical arm coordinate system and the model coordinate system; after adjusting the structured light module to the projection posture each time, generate the projection image corresponding to the to-be-projected model under the current posture of the projection assembly according to the current posture of the projection assembly and the medical model in the same coordinate system; wherein the medical model contains the to-be-projected model.

5. The surgical robotic system of claim 1, wherein, the generation of the first landing point cloud corresponding to the to-be-projected model on the surface of the target object according to the medical model and the position of the projection assembly comprises: in the case that the to-be-projected model belongs to the surface point cloud of the target object, take the to-be-projected model as the first landing point cloud; In a case where the to-be-projected model contains non-surface points of the target object, the projection component is abstracted as a point, and each point in the to-be-projected model is connected with the point, and an intersection of each connection line and a surface curve of the target object constitutes the second landing point point cloud.

6. The surgical robotic system of claim 1, wherein, The determining, according to the first landing point point cloud, of a second landing point point cloud corresponding to a region where the first landing point point cloud is located from the structured light point cloud of the target object comprises: A search tree is established for the structured light point cloud of the target object, and for each point in the first landing point point cloud, a nearest point in the search tree is searched, and each nearest point constitutes the second landing point point cloud.

7. The surgical robotic system of claim 1, wherein, The generating, according to the second landing point point cloud, of the projection image in combination with a collection process of the structured light point cloud of the target object comprises: A blank two-dimensional graph with the same resolution as a two-dimensional pattern projected in the collection process of the structured light point cloud of the target object is generated; For each point in the second landing point point cloud, a two-dimensional point coordinate corresponding to the point in the second landing point point cloud is determined according to a corresponding relationship between a pixel point in the two-dimensional pattern projected in the collection process of the structured light point cloud of the target object and the generated three-dimensional point, and is added to the blank two-dimensional graph, and finally the projection image is obtained.

8. The surgical robotic system of claim 1, wherein, In the S12, after the registration is completed, a verification step is further performed: An image stored in advance is projected onto the target object by the projection component to display a projection mark on the target object; The pose of the structured light module is adjusted so that the projection mark reaches a first feature position of the target object; The three-dimensional point cloud of the target object is collected again by the structured light module; According to the three-dimensional point cloud collected again, the image stored in advance, and the registration relationship, a position corresponding to the projection mark in the medical model is determined, and is displayed by a display device.

9. The surgical robotic system of claim 8, wherein, The determining, according to the three-dimensional point cloud collected again, the image stored in advance, and the registration relationship, of the position corresponding to the projection mark in the medical model comprises: According to a corresponding relationship between a pixel point in a two-dimensional pattern projected in a point cloud collection process of the three-dimensional point cloud collected again and a generated three-dimensional point, a three-dimensional point corresponding to a mark in the image stored in advance is determined; The three-dimensional point is converted into a model coordinate system according to a conversion relationship between a mechanical arm coordinate system and a model coordinate system obtained through the registration and a current mechanical arm pose.

10. A surgical navigation method characterized by, Comprise: S11, a three-dimensional point cloud of a target object is obtained by a structured light module; the structured light module comprises a projection component and a camera component; S12, registration is performed according to the three-dimensional point cloud and a medical model, and a projection image corresponding to a current pose of the projection component is generated; The generating of the projection image corresponding to the current pose of the projection component comprises: A structured light point cloud of the target object is obtained under the current pose of the projection component; wherein the structured light point cloud of the target object is obtained by the structured light module; According to the medical model and the position of the projection component, a first landing point cloud corresponding to the target object surface of a to-be-projected model is generated; wherein the medical model contains the to-be-projected model, and the to-be-projected model is a point cloud model; According to the first landing point cloud, a second landing point cloud corresponding to the region where the first landing point cloud is located is determined from the structured light point cloud of the target object; According to the second landing point cloud, the projection image is generated in combination with the acquisition process of the structured light point cloud of the target object; S13, the projection image is sent to the structured light module, so that the structured light module projects to the target object through the projection component.

11. A surgical robotic system, characterized by, Comprise: Workstation, first mechanical arm, second mechanical arm, structured light module, guide module; The structured light module is installed or integrated on the first mechanical arm, and is used for acquiring a three-dimensional point cloud and projecting positioning auxiliary information to a target object through a projection component of the structured light module; The guide module is installed or integrated on the second mechanical arm, and is used for guiding the movement trajectory of a surgical instrument; The workstation controls the postures of the first mechanical arm and the second mechanical arm, and generates a projection image containing the positioning auxiliary information according to the three-dimensional point cloud acquired by the structured light module and sends the projection image to the structured light module; The generation of the projection image containing the positioning auxiliary information comprises: Obtaining the structured light point cloud of the target object under the current pose of the projection component; wherein the structured light point cloud of the target object is acquired by the structured light module; According to the medical model and the position of the projection component, a first landing point cloud corresponding to the target object surface of a to-be-projected model is generated; wherein the medical model contains the to-be-projected model, and the to-be-projected model is a point cloud model; According to the first landing point cloud, a second landing point cloud corresponding to the region where the first landing point cloud is located is determined from the structured light point cloud of the target object; According to the second landing point cloud, the projection image is generated in combination with the acquisition process of the structured light point cloud of the target object.

Images