Pre-positioning system and method of surgical robot and electronic equipment

Abstract

The invention discloses a pre-positioning system and method of a surgical robot and electronic equipment, and the pre-positioning system of the surgical robot comprises a laser transmitting module which is located on a suspension disc of an operating table and is used for transmitting laser to an operating bed; the shooting module is used for shooting the operating bed in the process that the operating table moves towards the operating bed to obtain an operating bed image; the image processing module is used for determining a first position of the laser on the operating bed image and a second position of the bottom end of the sleeve; wherein the bottom end of the sleeve is positioned on the contact surface of the sleeve and the skin of a patient on an operating bed; the controller is used for controlling the operating table to move and controlling the operating table to stop moving so as to align the suspension disc and the sleeve when the distance between the first position and the second position is smaller than or equal to the preset distance. According to the pre-positioning system, the operation steps of the pre-positioning method are simplified, and the pre-positioning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular, to a pre-positioning system, method and electronic device for a surgical robot. Background Art

[0002] In minimally invasive surgery using a surgical robot, it generally includes two parts: a patient operating table and a doctor operating table. Before using the surgical robot for surgery, it is necessary to ensure that the patient is lying properly on the operating table and the cannulas used in the surgery have been installed on the patient. Then, move the operating table near the patient's operating table and align the pose of the suspension tray at the top of the operating table with the cannulas on the patient. This process is called pre-positioning. Currently, the commonly used pre-positioning method is manual visual positioning, which specifically includes the following process: After the medical staff determines that the patient is lying properly on the operating table and the punching and cannula installation are completed, push the operating table near the operating table, and continuously adjust the position of the operating table by visual method until the medical staff visually judges that the distance between the laser marker emitted by the center of the suspension tray at the top of the operating table and the cannulas on the patient is within the set range, and then drag the robotic arm of the operating table to dock with the cannulas. After the docking is completed, the suspension tray is aligned.

[0003] However, since the operating table usually has a large volume and its push handle is located on the back, most of the vision of the medical staff will be blocked when the medical staff pushes it forward. The manual visual positioning method requires the medical staff to observe back and forth and repeatedly adjust the operating table during operation, which takes a long time and is prone to collision with other objects on the way. Therefore, the pre-positioning method of the prior art has complex operation difficulty, low efficiency and low accuracy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of high operation difficulty and low efficiency in the pre-positioning method in the prior art, and provide a pre-positioning system, method and electronic device for a surgical robot.

[0005] The present invention solves the above technical problem by the following technical solutions:

[0006] The present invention provides a pre-positioning system for a surgical robot, and the pre-positioning system for the surgical robot includes:

[0007] A laser emission module, which is located on the suspension tray of the operating table, and the laser emission module is used to emit laser to the operating table;

[0008] A photographing module, which is used to photograph the operating table during the movement of the operating table towards the operating table to obtain an operating table image;

[0009] An image processing module, configured to determine a first position of the laser and a second position of the bottom end of the cannula on the operating table image; wherein, the bottom end of the cannula is located on the contact surface between the cannula and the patient's skin on the operating table;

[0010] A controller, configured to control the movement of the operating table, and when the distance between the first position and the second position is less than or equal to a preset distance, control the operating table to stop moving to align the suspension tray and the cannula.

[0011] Preferably, the image processing module includes:

[0012] An image recognition unit, configured to extract the image features of each frame of the operating table image and the optical flow features between adjacent frames of the operating table image through an instance segmentation algorithm, and determine the mask image of the cannula in each frame of the operating table image according to the image features and the optical flow features;

[0013] A determination unit, configured to determine the second position of the bottom end of the cannula according to the mask images of each frame.

[0014] Preferably, the image recognition unit includes an instance segmentation algorithm sub-unit, a feature extraction sub-unit, and a calculation sub-unit;

[0015] The instance segmentation algorithm sub-unit is configured to input a first RGB image into the instance segmentation algorithm to output a first mask image; and is also configured to input a second RGB image into the instance segmentation algorithm to output a second mask image;

[0016] Wherein, the first RGB image and the second RGB image are RGB images of adjacent frames of the operating table;

[0017] The feature extraction sub-unit is configured to extract a first optical flow feature of the first RGB image and a second optical flow feature of the second RGB image;

[0018] The calculation sub-unit is configured to subtract the first depth image from the second depth image to obtain a frame difference image; and is also configured to determine an optical flow image according to the first optical flow feature, the second optical flow feature, and the frame difference image; wherein, the first depth image and the second depth image are depth images of adjacent two frames of the operating table;

[0019] The instance segmentation algorithm sub-unit is further configured to input the optical flow image into the instance segmentation algorithm to output a third mask image;

[0020] The feature extraction sub-unit is further configured to extract the common features of the first mask image and the second mask image, and determine a fourth mask image according to the common features;

[0021] The calculation sub-unit is further configured to sum a first matrix representing the fourth mask image and a second matrix representing the optical flow feature to determine a fifth mask image; wherein, the fifth mask image and the second mask image correspond to the same moment.

[0022] Preferably, the image processing module further includes an image positioning unit;

[0023] The image positioning unit is configured to, when the shape of the laser is cross-shaped, determine the cross intersection position by the Hough transform line detection method, and determine the cross intersection position as the first position.

[0024] Preferably, the pre-positioning system of the surgical robot further includes:

[0025] A marking module, configured to, when there are at least two cannulas on the surgical bed image, mark the second positions at the bottoms of the multiple cannulas respectively to distinguish different cannulas.

[0026] Preferably, the pre-positioning system of the surgical robot further includes: a pusher and a display screen;

[0027] The pusher is configured to transmit the surgical bed image to the display screen;

[0028] The display screen is configured to display the surgical bed image.

[0029] Preferably, the pre-positioning system of the surgical robot further includes:

[0030] A socketing module, configured to socket the robotic arm of the surgical robot and the cannula when the operating table stops moving.

[0031] The present invention also provides a pre-positioning method for a surgical robot, and the pre-positioning method for the surgical robot includes:

[0032] Obtain a surgical bed image; wherein, the surgical bed image is acquired during the process that the laser is emitted on the surgical bed and the operating table moves towards the surgical bed;

[0033] Determine a first position of the laser on the surgical bed image; wherein, the laser is used to represent the pose of the suspension tray of the operating table;

[0034] Determine a second position of the bottom end of the cannula on the surgical bed image; wherein, the bottom end of the cannula is located on the contact surface between the cannula and the patient's skin on the surgical bed;

[0035] Obtain the distance between the first position and the second position;

[0036] When the distance is less than or equal to a preset distance, stop moving the operating table to align the suspension tray and the cannula.

[0037] Preferably, the step of obtaining the second position of the bottom end of the cannula on the operating table image includes:

[0038] Extracting the image features of each frame of the operating table image and the optical flow features between adjacent frames of the operating table image through an instance segmentation algorithm;

[0039] Determining the mask image of the cannula in each frame of the operating table image according to the image features and the optical flow features;

[0040] Determining the second position of the bottom end of the cannula according to the mask images of each frame.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and used to run on the processor. When the processor executes the computer program, the pre-positioning method of the surgical robot described above is implemented.

[0042] On the basis of conforming to the common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred examples of the present invention.

[0043] The positive and progressive effects of the present invention are as follows:

[0044] In the pre-positioning system of the present invention, during the movement of the operating table towards the operating bed, the distance between the bottom end of the cannula and the laser on the operating bed image is continuously and automatically detected. When the distance is less than or equal to the preset distance, the movement of the operating table is stopped to align the suspension tray on the operating table and the cannula on the patient lying on the operating bed, thereby simplifying the operation steps of the pre-positioning method and improving the efficiency of pre-positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Structural diagram of the pre-positioning system of the surgical robot according to Embodiment 1 of the present invention;

[0046] Figure 2 Structural diagram of the operating table of the pre-positioning system of the surgical robot according to Embodiment 1 of the present invention;

[0047] Figure 3 Structural diagram of the operating bed of the pre-positioning system of the surgical robot according to Embodiment 1 of the present invention;

[0048] Figure 4 Structural diagram of the center point determination model of the pre-positioning system of the surgical robot according to Embodiment 1 of the present invention;

[0049] Figure 5 Execution flowchart of a specific example of the pre-positioning system of the surgical robot according to Embodiment 1 of the present invention;

[0050] Figure 6 Flowchart of the pre-positioning method of the surgical robot according to Embodiment 2 of the present invention;

[0051] Figure 7 This is a schematic structural diagram of the electronic device according to Embodiment 3 of the present invention. Detailed implementation manners

[0052] The present invention will be further described below by way of embodiments, but the present invention is not limited to the scope of the described embodiments.

[0053] Embodiment 1

[0054] This embodiment provides a pre-positioning system for a surgical robot, which is applied to a minimally invasive surgery platform. The minimally invasive surgery platform includes an operating bed and an operating table. During the pre-positioning process, the operating table continuously moves towards the operating bed until the distance between the bottom end of the cannula and the laser is less than or equal to a preset distance, and then the operating table stops moving to complete the pre-positioning.

[0055] See Figure 1 , the pre-positioning system of the surgical robot includes:

[0056] A laser emission module 101, which is located on the suspension tray of the operating table, and the laser emission module 101 is used to emit a laser to the operating bed.

[0057] Among them, the shape of the laser can include a cross shape, a star shape, a triangle shape, etc. The function of emitting the laser is to indicate the pose of the suspension tray.

[0058] A shooting module 102, which is used to shoot the operating bed during the movement of the operating table towards the operating bed to obtain an image of the operating bed. Among them, the shooting module 102 includes a 3D (three-dimensional) camera, a color camera, an infrared camera, a depth camera, etc.

[0059] An image processing module 103, which is used to determine the first position of the laser and the second position of the bottom end of the cannula on the image of the operating bed.

[0060] Among them, the bottom end of the cannula is located on the contact surface between the cannula and the patient's skin on the operating bed.

[0061] A controller 104, which is used to control the movement of the operating table and, when the distance between the first position and the second position is less than or equal to the preset distance, control the operating table to stop moving to align the suspension tray and the cannula.

[0062] Among them, the method for judging whether the distance between the first position and the second position is less than or equal to the preset distance includes judging by human eye observation, installing ranging sensors such as radar, laser, and infrared devices for measurement and judgment, and calculating and judging according to distance calculation formulas such as Euclidean distance, geometric algorithm, Mahalanobis distance, and Manhattan distance.

[0063] It should be noted that the preset distance can be set according to the actual situation, which can be based on manual experience, can also adopt the system default value, or can be determined in real time according to the on-site observation results.

[0064] Figure 2 The operating table of the minimally invasive surgery platform in this embodiment includes a suspension tray 201, a shooting module 102, a controller 104, a pusher 106, a display screen 107, a pusher hand 202, the laser 203 emitted by the laser emission module, and a robotic arm 204.

[0065] Figure 3 For the operating bed of the minimally invasive surgery platform, before pre-positioning, the patient lies on the operating bed, and the medical staff inserts the cannula 1 on the patient.

[0066] It should be noted that the cannula 1 may be one or multiple. The robotic arm 204 may be one or multiple.

[0067] In the pre-positioning system of this embodiment, during the process of the operating table moving towards the operating bed, the distance between the bottom end of the cannula and the laser on the operating bed image is continuously and automatically detected. When the distance is less than or equal to the preset distance, the movement of the operating table is stopped to align the suspension tray on the operating table and the cannula on the patient lying on the operating bed, thereby simplifying the operation steps of the pre-positioning method and improving the efficiency of pre-positioning.

[0068] In an alternative embodiment, refer to Figure 1 , the image processing module 103 includes:

[0069] An image recognition unit 1031, configured to extract the image features of each frame of the operating bed image and the optical flow features between adjacent frames of the operating bed image through an instance segmentation algorithm, and determine the mask image of the cannula in each frame of the operating bed image according to the image features and the optical flow features.

[0070] In an alternative embodiment, refer to Figure 1 , the image recognition unit 1031 includes an instance segmentation algorithm sub-unit 10311, a feature extraction sub-unit 10312, and a calculation sub-unit 10313.

[0071] The instance segmentation algorithm sub-unit 10311 is configured to input the first RGB image into the instance segmentation algorithm to output the first mask image. It is also configured to input the second RGB image into the instance segmentation algorithm to output the second mask image. Wherein, the first RGB image and the second RGB image are adjacent-frame operating bed RGB images.

[0072] The feature extraction sub-unit 10312 is configured to extract the first optical flow feature of the first RGB image and the second optical flow feature of the second RGB image.

[0073] The calculation subunit 10313 is configured to obtain a frame difference image by subtracting the first depth image from the second depth image. It is also configured to determine an optical flow image based on the first optical flow feature, the second optical flow feature, and the frame difference image. The first depth image and the second depth image are the depth images of the operating table for two adjacent frames. The frame difference image is a two-channel image, and the optical flow image is a three-channel image.

[0074] The instance segmentation algorithm subunit 10311 is further configured to input the optical flow image into an instance segmentation algorithm to output a third mask image.

[0075] The feature extraction subunit 10312 is further configured to extract the common features of the first mask image and the second mask image, and determine a fourth mask image based on the common features.

[0076] The calculation subunit 10313 is further configured to sum the first matrix representing the fourth mask image and the second matrix representing the optical flow feature to determine a fifth mask image. The fifth mask image and the second mask image correspond to the same moment.

[0077] The following introduces the specific execution step process of an instance segmentation algorithm for an image recognition unit to determine a mask image:

[0078] S1. Obtain the RGB image of the operating table and the depth image of the operating table for adjacent frames from the shooting module.

[0079] Denote the RGB image of the operating table for the previous frame as the first RGB image, and the RGB image of the operating table for the next frame as the second RGB image. Denote the depth image of the operating table for the previous frame as the first depth image, and the depth image of the operating table for the next frame as the second depth image.

[0080] S2. Input the first RGB image into an instance segmentation algorithm to output a first mask image (Mask1) and the confidence C1 of the first mask image; input the second RGB image into the instance segmentation algorithm to output a second mask image (Mask2) and the confidence C2 of the second mask image. Mask1 is used to represent the mask image with static parameters for the previous frame. Mask2 is used to represent the mask image with static parameters for the next frame.

[0081] S3. Use the L-K (Lucas-Kanada) optical flow algorithm to extract the optical flow features of the first RGB image and the second RGB image, and obtain the two-channel optical flow features of the two RGB images, denoted as OL. OL is used to represent the motion displacement of each pixel in the image within the time of two adjacent frames.

[0082] S4. Subtract the first depth image from the second depth image to obtain a frame difference image, and combine the single-channel frame difference image with OL obtained in step S3 into a three-channel optical flow image.

[0083] S5. Input the optical flow image into the instance segmentation algorithm to output the third mask image (Mask3). Mask3 is used to represent the mask image with dynamic parameters in the previous frame.

[0084] S6. Extract the common features of Mask1 and Mask3, denoted as the fourth mask image (Mask4). Mask4 is used to represent the mask image with the common dynamic and static parameters in the previous frame.

[0085] S7. Encode Mask4 to obtain the corresponding mask image Mask4’ with two channels, and add Mask4’ to OL channel by channel.

[0086] The following is the encoding formula:

[0087]

[0088]

[0089] Among them, 1 is used to represent the first channel; 2 is used to represent the second channel; x is used to represent the speed of each frame of the mask image in the horizontal coordinate direction; y is used to represent the speed of each frame of the mask image in the vertical coordinate direction.

[0090] S8. Decode Mask4’ into a single-channel fifth mask image (Mask5). Mask5 is used to represent the mask image with the common dynamic and static parameters in the next frame.

[0091] The following is the decoding formula:

[0092]

[0093] Among them, x is used to represent the speed of each frame of the mask image in the horizontal coordinate direction; y is used to represent the speed of each frame of the mask image in the vertical coordinate direction.

[0094] The determination unit 1032 is configured to determine the second position of the bottom end of the cannula according to the mask images of each frame.

[0095] In this embodiment, the image recognition unit determines the mask image through the instance segmentation algorithm, and determines the second position of the bottom end of the cannula according to the mask image, so as to automatically obtain the position of the cannula on the patient during the movement of the operating table, thereby improving the pre-positioning efficiency.

[0096] In an alternative embodiment, the determination unit 1032 is further configured to determine the second position of the bottom end of the cannula according to the center point determination model.

[0097] Among them, the center point determination model is obtained by training a neural network with the training sample pairs of the mask image and the position of the mask center point marked.

[0098] The following introduces the specific steps for the determination unit to determine the model based on the center point and determine the second position of the bottom end of the sleeve:

[0099] S9. Input Mask2 and Mask5 into the center point determination model respectively to output the mask center point positions of Mask2 and Mask5. Denote the mask center point position of Mask2 as P1; denote the mask center point position of Mask5 as P2.

[0100] See Figure 4 , the center point determination model includes a 1x1x3 convolutional layer, a VGG network (a convolutional neural network) feature extraction module, a pooling layer, a 1x1x2 fully connected layer, and a normalization function sigmoid function.

[0101] S10. Perform weighted summation on P1 and P2 to obtain the second position P of the bottom end of the sleeve.

[0102] The formula for weighted summation is as follows:

[0103]

[0104] Where P is the second position of the bottom end of the tube, P1 is the mask center point position of the second mask image, P2 is the mask center point position of the fifth mask image, C1 is the confidence of the first mask image; C2 is the confidence of the second mask image, and α is the proportion of the confidence of the second mask image in the total confidence of the first mask image and the second mask image.

[0105] In this embodiment, the determination unit inputs the mask image into the center point determination model to obtain the mask center point position, thereby determining the second position of the bottom end of the sleeve, improving the efficiency of determining the sleeve position, and further improving the efficiency of pre-positioning.

[0106] In an optional embodiment, see Figure 1 , the image processing module further includes an image positioning unit 1033.

[0107] The image positioning unit 1033 is used to determine the cross intersection position by the Hough transform line detection method when the shape of the laser is cross-shaped, and determine the cross intersection position as the first position.

[0108] In this embodiment, the cross intersection position of the cross-shaped laser is determined as the first position by the Hough transform line detection method.

[0109] In an optional embodiment, see Figure 1 , the pre-positioning system of the surgical robot further includes:

[0110] The marking module 105 is used to mark the second positions at the bottoms of multiple trocars respectively when there are at least two trocars on the operating table image, so as to distinguish different trocars.

[0111] Among them, the marking methods include marking with different graphics, marking with digital numbers, marking with words with different meanings, etc.

[0112] In this embodiment, when there are multiple trocars on the patient, the operating table may also have multiple robotic arms. The docking between the trocars and the robotic arms is usually set according to the habits of medical staff. In order to facilitate the subsequent minimally invasive surgical operations of medical staff, the second positions at the bottoms of different trocars are marked on the image of the operating table to distinguish different trocars.

[0113] In an optional real-time mode, refer to Figure 1 , the pre-positioning system of the surgical robot further includes: a pusher 106 and a display screen 107.

[0114] The pusher 106 is used to transmit the operating table image to the display screen.

[0115] The display screen 107 is used to display the operating table image.

[0116] In this embodiment, during the movement of the operating table towards the operating bed, the pusher is used to transmit the field of view in front of the operating table to the display screen in real time, so as to facilitate the medical staff operating personnel to know the field of view in front through the display screen, thereby avoiding the medical staff operating personnel from observing back and forth. On the one hand, it can stop the movement of the operating table in time when encountering an obstacle, or change the pushing direction of the operating table to avoid the obstacle; on the other hand, it can also control the distance between the operating table and the patient to be within a relatively safe range, improving the operation safety.

[0117] In an optional embodiment, refer to Figure 1 , the pre-positioning system of the surgical robot further includes:

[0118] The sleeving module 108 is used to sleeve the robotic arm and the trocar of the surgical robot when the operating table stops moving.

[0119] The following introduces a specific example to show the specific operation steps during the pre-positioning process. Figure 5 This is the flowchart of this example:

[0120] S501. The patient lies well on the operating bed, and the punching and trocar installation are completed.

[0121] Among them, there may be 1 trocar or multiple trocars.

[0122] S502. Start the operating table, turn on the laser emission module at the top of the suspension tray on the operating table, and complete the preparatory work such as installing the sterile cover and adjusting the configuration of the robotic arm. Ensure that the laser emission module can emit laser normally without being blocked by the robotic arm.

[0123] The laser emission module in this example emits cross-shaped laser.

[0124] S503. Turn on the camera of the shooting module, ensure that the laser is within the camera's field of view, and at this time, the pusher starts working synchronously, and the camera image is displayed on the display screen in real time and smoothly.

[0125] In this example, a 3D camera is used. In addition to obtaining a color image, it can also synchronously obtain its corresponding depth image, and only the color image obtained is displayed on the display screen.

[0126] S504. The operator pushes the operating table, and the pusher detects the first position where the center of the laser is displayed on the display screen in real time.

[0127] The laser in this example is a cross-shaped laser, which is composed of two intersecting straight lines. Therefore, for each frame of the display screen image, straight line detection (usually using the Hough transform straight line detection method) is performed. First, the positions of the two straight lines are detected, and then the intersection point of the two straight lines is calculated, which is the first position of the laser center. Since the relative position relationship between the laser emission module and the camera is fixed, the position of the laser displayed on the display screen remains unchanged, and only one detection is required.

[0128] S505. Determine whether the cannula appears on the display screen of the display.

[0129] During the process of the medical staff pushing the operating table closer to the patient's operating bed, one or more cannulas installed on the patient will appear in the camera's field of view at a certain moment and thus be displayed on the display screen. The position of the bottom end of the cannula can be detected through human-computer interaction and other methods, that is, interaction buttons are set on the display screen. When the medical staff observes that the cannula appears completely on the display screen, they click the interaction button to execute step S507; otherwise, execute step S506.

[0130] S506. Continue to push the cart.

[0131] S507. Detect the second position of the bottom end of the cannula in the display screen image.

[0132] In this example, a real-time detection and segmentation of the casing is performed by an instance segmentation algorithm based on spatio-temporal multi-dimensional information, and the approximate position of the bottom end of the casing is obtained in real time according to the position of the center point of the mask after segmentation. In addition, for the case of multiple casings, after the bottom end positions of multiple casings are detected for the first time, through human-computer interaction, each bottom end position of the casing is numbered, and the medical staff operator is allowed to select the number corresponding to the bottom end position of the casing, so as to lock the casing object to be detected subsequently.

[0133] S508. Convert the first position and the second position into a three-dimensional space, calculate the distance between the first position and the second position, and display the distance on the display screen.

[0134] In this example, the internal parameters T of the camera lens are known intrinsic and the coordinates (u1, v1), (u2, v2) of the first position and the second position in the pixel coordinate system, and the depth positions of the first position and the second position with respect to the camera are and Then, the first position and the second position can be converted into the camera space coordinate system through the following formula.

[0135]

[0136]

[0137] Among them, [x1, y1, z1] is used to represent the coordinate position of the first position in the camera space coordinate system, [x2, y2, z2] is used to represent the coordinate position of the second position in the camera space coordinate system, T intrinsic is used to represent the internal parameters of the camera lens, is used to represent the depth position of the first position, is used to represent the depth position of the second position, (u1, v1, 1) is used to represent the coordinates of the first position in the pixel coordinate system, and (u2, v2, 1) is used to represent the coordinates of the second position in the pixel coordinate system.

[0138] Calculate the distance L between the first position and the second position in the three-dimensional space according to the Euclidean distance:

[0139]

[0140] Among them, L is the Euclidean distance value between the first position and the second position, x1 is the x-axis coordinate value of the first position, y1 is the y-axis coordinate value of the first position, z1 is the z-axis coordinate value of the first position, x2 is the x-axis coordinate value of the second position, y2 is the y-axis coordinate value of the second position, and z2 is the z-axis coordinate value of the second position.

[0141] S509. Determine whether the distance is within the threshold range. If not, return to step S506. If so, execute step S510. Here, the threshold can be set manually or use the default threshold. In this example, the default threshold is 8 cm.

[0142] S510. The display screen gives a prompt, suggesting that the operator stop pushing the cart.

[0143] At this time, the medical staff can stop pushing the cart immediately after seeing the prompt, or can adjust the position of the operating table again according to the actual situation.

[0144] S511. Connect the cannula to the robotic arm.

[0145] S512. The controller can obtain the position of the cannula in the space coordinate system of the operating table by calling the kinematics of the operating table. According to this position, it can guide the medical staff to adjust the suspension tray to align the operating table with the cannula.

[0146] Embodiment 2

[0147] This embodiment provides a pre-positioning method for a surgical robot. Refer to Figure 6 , the pre-positioning method of the surgical robot includes:

[0148] S601. Obtain an image of the operating bed.

[0149] Among them, the image of the operating bed is collected when the laser is emitted on the operating bed and the operating table is moving towards the operating bed.

[0150] S602. Determine the first position of the laser on the image of the operating bed.

[0151] Among them, the laser is used to represent the pose of the suspension tray of the operating table.

[0152] S603. Determine the second position of the bottom end of the cannula on the image of the operating bed.

[0153] Among them, the bottom end of the cannula is located on the contact surface between the cannula and the patient's skin on the operating bed.

[0154] S604. Obtain the distance between the first position and the second position.

[0155] S605. When the distance is less than or equal to the preset distance, stop moving the operating table to align the suspension tray and the cannula.

[0156] Among them, the preset distance is set according to the actual situation.

[0157] In this embodiment, during the movement of the operating table towards the operating bed, the distance between the bottom end of the cannula and the laser on the image of the operating bed is continuously and automatically detected. When the distance is less than or equal to the preset distance, the movement of the operating table is stopped to align the suspension tray on the operating table with the cannula on the patient lying on the operating bed, thereby simplifying the operation steps of the pre-positioning method and improving the efficiency of pre-positioning.

[0158] In an alternative embodiment, step S603 includes:

[0159] S6031. Extract the image features of each frame of the operating bed image and the optical flow features between adjacent frames of the operating bed image through an instance segmentation algorithm.

[0160] S6032. Determine the mask image of the cannula in each frame of the operating bed image according to the image features and optical flow features.

[0161] S6033. Determine the second position of the bottom end of the cannula according to the mask images of each frame.

[0162] In this embodiment, the mask image is determined through an instance segmentation algorithm, and the second position of the bottom end of the cannula is determined according to the mask image, so as to automatically obtain the position of the cannula on the patient during the movement of the operating table, thereby improving the pre-positioning efficiency.

[0163] In an alternative embodiment, step S6032 includes:

[0164] Input the first RGB image into the instance segmentation algorithm to output the first mask image.

[0165] Input the second RGB image into the instance segmentation algorithm to output the second mask image.

[0166] Wherein, the first RGB image and the second RGB image are adjacent-frame RGB images of the operating bed.

[0167] Extract the first optical flow feature of the first RGB image and the second optical flow feature of the second RGB image.

[0168] Subtract the first depth image from the second depth image to obtain a frame difference image.

[0169] Determine the optical flow image according to the first optical flow feature, the second optical flow feature, and the frame difference image.

[0170] Wherein, the first depth image and the second depth image are adjacent two-frame depth images of the operating bed; the frame difference image is a two-channel image, and the optical flow image is a three-channel image.

[0171] Input the optical flow image into the instance segmentation algorithm to output the third mask image.

[0172] Extract the common features of the first mask image and the second mask image, and determine the fourth mask image according to the common features.

[0173] Sum the first matrix representing the fourth mask image and the second matrix representing the optical flow features to determine the fifth mask image.

[0174] Among them, the fifth mask image and the second mask image correspond to the same moment.

[0175] In an optional embodiment, step S6033 includes:

[0176] Determine the second position of the bottom end of the cannula according to the center point determination model.

[0177] Among them, the center point determination model is obtained by training a neural network with training samples of mask images and mask center point positions marked.

[0178] In this embodiment, input the mask image into the center point determination model to obtain the mask center point position, so as to determine the second position of the bottom end of the cannula, improve the efficiency of determining the cannula position, and further improve the efficiency of pre-positioning.

[0179] In an optional embodiment, when the shape of the laser is cross-shaped, step S602 includes:

[0180] Determine the cross intersection position by the Hough transform line detection method, and determine the cross intersection position as the first position.

[0181] In an optional embodiment, when there are at least two cannulas on the operating table image, the pre-positioning method of the surgical robot further includes:

[0182] Mark the second positions of the bottom ends of multiple cannulas respectively to distinguish different cannulas.

[0183] In this embodiment, when there are multiple cannulas on the patient, the operating table may also have multiple robotic arms. The docking between the cannulas and the robotic arms is usually set according to the habits of medical staff. In order to facilitate the subsequent minimally invasive surgical operations of medical staff, mark the second positions of the bottom ends of different cannulas on the operating table image to distinguish different cannulas.

[0184] In an optional embodiment, the pre-positioning method of the surgical robot further includes:

[0185] Real-time display the operating table image.

[0186] In this embodiment, during the movement of the operating table towards the operating bed, the view in front of the operating table is transmitted to the display screen in real time, facilitating medical staff to obtain the front view through the display screen, thereby avoiding the need for medical staff to observe back and forth. On the one hand, it can stop the movement of the operating table in time when encountering obstacles, or change the pushing direction of the operating table to avoid obstacles; on the other hand, it can also control the distance between the operating table and the patient to be within a relatively safe range, improving the operation safety.

[0187] Embodiment 3

[0188] This embodiment provides an electronic device. Figure 7 It is a schematic diagram of the modules of the electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the pre-positioning method of the surgical robot in Embodiment 1. Figure 7 The shown electronic device 30 is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0189] As Figure 7 shown, the electronic device 30 can be presented in the form of a general computing device, for example, it can be a server device. The components of the electronic device 30 may include but are not limited to: at least one of the above-mentioned processors 31, at least one of the above-mentioned memories 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0190] The bus 33 includes a data bus, an address bus, and a control bus.

[0191] The memory 32 may include volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322, and may further include a read-only memory (ROM) 323.

[0192] The memory 32 may further include a program / utility 325 having a set (at least one) of program modules 324. Such program modules 324 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0193] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the pre-positioning method of the surgical robot in Embodiment 1 of the present invention.

[0194] The electronic device 30 can also communicate with one or more external devices 34 (such as a keyboard, a pointing device, etc.). Such communication can be carried out through the input / output (I / O) interface 35. Moreover, the model generation device 30 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 36. As Figure 7 shown, the network adapter 36 communicates with other modules of the model generation device 30 through the bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the model generation device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (redundant array of independent disks) systems, tape drives, and data backup storage systems, etc.

[0195] It should be noted that, although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0196] Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that this is only an example, and the protection scope of the present invention is defined by the appended claims. Without departing from the principles and essence of the present invention, those skilled in the art can make various changes or modifications to these embodiments, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A pre-positioning system for a surgical robot, characterized in that, The pre-positioning system of the surgical robot includes: A laser emission module, which is located on the suspension plate of the operating table, and is used to emit laser to the operating bed; A photographing module, which is used to photograph the operating bed during the movement of the operating table towards the operating bed to obtain an image of the operating bed; An image processing module, which is used to determine the first position of the laser and the second position of the bottom end of the cannula on the image of the operating bed; wherein, the bottom end of the cannula is located on the contact surface between the cannula and the patient's skin on the operating bed; A controller, which is used to control the movement of the operating table, and when the distance between the first position and the second position is less than or equal to a preset distance, control the operating table to stop moving to align the suspension plate and the cannula.

2. The pre-positioning system of the surgical robot according to claim 1, characterized in that, The image processing module includes: An image recognition unit, which is used to extract the image features of each frame of the operating bed image and the optical flow features between adjacent frames of the operating bed image through an instance segmentation algorithm, and determine the mask image of the cannula in each frame of the operating bed image according to the image features and the optical flow features; A determination unit, which is used to determine the second position of the bottom end of the cannula according to each frame of the mask image.

3. The pre-positioning system of the surgical robot according to claim 2, characterized in that, The image recognition unit includes an instance segmentation algorithm sub-unit, a feature extraction sub-unit, and a calculation sub-unit; The instance segmentation algorithm sub-unit is used to input the first RGB image into the instance segmentation algorithm to output the first mask image; and is also used to input the second RGB image into the instance segmentation algorithm to output the second mask image; Wherein, the first RGB image and the second RGB image are RGB images of the operating bed in adjacent frames; The feature extraction sub-unit is used to extract the first optical flow feature of the first RGB image and the second optical flow feature of the second RGB image; The calculation sub-unit is used to subtract the first depth image from the second depth image to obtain a frame difference image; and is also used to determine the optical flow image according to the first optical flow feature, the second optical flow feature, and the frame difference image; wherein, the first depth image and the second depth image are depth images of the operating bed in two adjacent frames; The instance segmentation algorithm sub-unit is also used to input the optical flow image into the instance segmentation algorithm to output the third mask image; The feature extraction sub-unit is also used to extract the common features of the first mask image and the second mask image, and determine the fourth mask image according to the common features; The calculation sub-unit is also used to sum the first matrix representing the fourth mask image and the second matrix representing the optical flow feature to determine the fifth mask image; wherein, the fifth mask image and the second mask image correspond to the same moment.

4. The pre-positioning system of the surgical robot according to claim 1, wherein The image processing module further includes an image positioning unit; The image positioning unit is used to, when the shape of the laser is cross-shaped, determine the cross intersection position by the Hough transform line detection method, and determine the cross intersection position as the first position.

5. The pre-positioning system of the surgical robot according to claim 1, characterized in that, The pre-positioning system of the surgical robot further includes: A marking module, which is used to mark the second positions of the bottom ends of multiple cannulas respectively when there are at least two cannulas on the image of the operating bed to distinguish different cannulas.

6. The pre-positioning system of the surgical robot according to claim 1, wherein, The pre-positioning system of the surgical robot further includes: a pusher and a display screen; The pusher is configured to transmit the surgical bed image to the display screen; The display screen is configured to display the surgical bed image.

7. The pre-positioning system of the surgical robot according to claim 1, characterized in that, The pre-positioning system of the surgical robot further includes: A socketing module, configured to socket the robotic arm of the surgical robot and the cannula when the operating table stops moving.

8. A pre-positioning method for a surgical robot, characterized in that, The pre-positioning method of the surgical robot includes: Obtaining a surgical bed image; wherein, the surgical bed image is acquired during the process that a laser is emitted onto the surgical bed and the operating table moves towards the surgical bed; Determining a first position of the laser on the surgical bed image; wherein, the laser is used to characterize the pose of the suspension tray of the operating table; Determining a second position of the bottom end of the cannula on the surgical bed image; wherein, the bottom end of the cannula is located on the contact surface between the cannula and the patient's skin on the surgical bed; Obtaining the distance between the first position and the second position; When the distance is less than or equal to a preset distance, stopping the movement of the operating table to align the suspension tray and the cannula.

9. The pre-positioning method of the surgical robot according to claim 8, characterized in that, The step of obtaining the second position of the bottom end of the cannula on the surgical bed image includes: Extracting the image features of each frame of the surgical bed image and the optical flow features between adjacent frames of the surgical bed image through an instance segmentation algorithm; Determining the mask image of the cannula in each frame of the surgical bed image according to the image features and the optical flow features; Determining the second position of the bottom end of the cannula according to the mask images of each frame.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and configured to run on the processor, characterized in that, When the processor executes the computer program, it implements the pre-positioning method of the surgical robot according to claim 8 or 9.

Images