Surgical robot control method, system, device and medium

Abstract

The application discloses a kind of surgical robot control method, system, equipment and medium.The control method includes: obtaining the image including cannula and the depth information of the cannula;The cannula in the image is identified, and the feature point of the cannula is obtained according to the identification result;The target depth information corresponding to the feature point is obtained;The positioning information of the cannula is obtained based on the feature point and the target depth information;The operating end of the surgical robot is controlled to move based on the positioning information.Through identifying the cannula in the scene where the surgical robot is located, the feature point and depth information of the cannula are obtained, the positioning information of the cannula is accurately obtained based on the feature point and the depth information, the automatic movement of the mechanical arm of the robot based on the cannula positioning information is realized, manual dragging operation is replaced, and the movement time of the surgical robot is shortened.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a control method, system, device and medium for a surgical robot. Background Technology

[0002] In the current medical field, surgical robots are widely used due to their precision and efficiency. During surgery, surgeons need to insert several cannulas into the patient's body. Common laparoscopic surgical robots do not have automatic positioning capabilities, requiring manual manipulation of the robot's multiple robotic arms to connect with the cannulas and begin the surgery. Because the robot cannot accurately know the spatial position of the cannulas, it must be manually moved to the vicinity of the cannulas for docking, which is difficult, time-consuming, and labor-intensive. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the defect in the prior art that the robot cannot determine the spatial position of the cannula, which leads to the inability of the surgical robot to move automatically, and to provide a control method, system, device and medium for a surgical robot.

[0004] The present invention solves the above-mentioned technical problems through the following technical solution:

[0005] A first aspect provides a control method for a surgical robot, the control method comprising:

[0006] Acquire an image including the cannula and the depth information of the cannula;

[0007] The sleeve in the image is identified, and the feature points of the sleeve are obtained based on the identification results;

[0008] Obtain the target depth information corresponding to the feature points;

[0009] The positioning information of the sleeve is obtained based on the feature points and the target depth information;

[0010] The control end of the surgical robot moves based on the positioning information.

[0011] Preferably, the step of identifying the sleeve in the image and obtaining the feature points of the sleeve based on the identification result includes:

[0012] The image is input into a pre-trained cannula recognition model, which outputs a mask for the cannula.

[0013] The feature points of the sleeve are obtained based on the mask identification.

[0014] Preferably, the step of obtaining the target depth information corresponding to the feature point includes:

[0015] The area of ​​the sleeve in the image is used as the foreground, and the other areas are used as the background;

[0016] Obtain first depth information of foreground pixels and second depth information of background pixels within a preset range centered on the feature point;

[0017] The target depth information corresponding to the feature point is calculated based on the first depth information and the second depth information.

[0018] Preferably, the step of obtaining the positioning information of the sleeve based on the feature points and the target depth information includes:

[0019] Obtain the geometric centroid of the sleeve;

[0020] If the feature point meets the preset conditions, then the feature point is taken as the target feature point;

[0021] The preset conditions include that the distance between the feature point and the geometric centroid meets a first distance threshold requirement, and that the target depth information meets a depth threshold requirement;

[0022] The positioning information of the sleeve is obtained based on the target feature points and the corresponding target depth information.

[0023] Preferably, the control method further includes:

[0024] If the feature point does not meet the preset condition, the positioning information of the sleeve is obtained based on the geometric centroid and the depth information corresponding to the geometric centroid.

[0025] Preferably, after identifying the sleeve in the image, the method further includes:

[0026] Identify the target object in the image;

[0027] The sleeve is disposed on the target object;

[0028] The contact area between the target object and the sleeve is obtained based on the feature points;

[0029] The positioning information of the sleeve is obtained based on the feature points and the contact area.

[0030] Preferably, the step of obtaining the positioning information of the sleeve based on the feature points and the contact area includes:

[0031] In response to the distance between the feature point and the contact area meeting the second distance threshold requirement, the position information of any feature point that meets the requirement is used as the positioning information of the sleeve;

[0032] Alternatively, the feature point with the smallest distance from the contact area can be selected, and the position information of the feature point can be used as the positioning information of the sleeve.

[0033] Preferably, the step of controlling the operating terminal to move based on the positioning information includes:

[0034] The positioning information of the sleeve is transformed from the pixel coordinate system to the robot coordinate system;

[0035] The control terminal moves based on the robot coordinate system.

[0036] Preferably, the number of operating terminals is greater than or equal to the number of sleeves, and the control method further includes:

[0037] If the number of sleeves identified in the image is not less than two, then the positioning information corresponding to each sleeve is obtained;

[0038] The number of control terminals equal to the number of the sheaths is matched with the positioning information of the corresponding sheaths, and each control terminal moves based on the positioning information of the corresponding sheath.

[0039] Preferably, the step of matching the number of operating terminals, which is the same as the number of sleeves, with the positioning information corresponding to the sleeves includes:

[0040] Obtain the movement path of each of the sleeves and each of the operating ends;

[0041] Select the movement path that does not intersect with other movement paths as the target path;

[0042] Based on the number of sleeves, select a corresponding number of target paths and control the operation terminal corresponding to the target path to move.

[0043] The second aspect provides a control system for a surgical robot, the control system including an image acquisition module, a recognition module, a positioning module and a control module;

[0044] The image acquisition module is used to acquire an image including the sleeve and the depth information of the sleeve;

[0045] The recognition module is used to identify the sleeve in the image and obtain the feature points of the sleeve based on the recognition result;

[0046] The positioning module is used to acquire target depth information corresponding to the feature points; and is also used to obtain positioning information of the sleeve based on the feature points and the target depth information.

[0047] The control module is used to control the operating end of the surgical robot to move based on the positioning information.

[0048] Preferably, the recognition module includes a mask output unit and a feature point recognition unit;

[0049] The mask output unit is used to input the image into a pre-trained sleeve recognition model and output the mask of the sleeve.

[0050] The feature point recognition unit is used to identify the feature points of the sleeve based on the mask.

[0051] Preferably, the positioning module is further configured to: take the area of ​​the sleeve in the image as the foreground and other areas as the background; acquire first depth information of foreground pixels and second depth information of background pixels within a preset range centered on the feature point; and calculate the target depth information corresponding to the feature point based on the first depth information and the second depth information.

[0052] As one possible approach, the positioning module is also used to obtain the geometric centroid of the sleeve; in response to the feature point meeting a preset condition, the feature point is used as the target feature point;

[0053] The preset conditions include that the distance between the feature point and the geometric centroid meets a first distance threshold requirement, and that the target depth information meets a depth threshold requirement;

[0054] The positioning module is also used to obtain the positioning information of the sleeve based on the target feature points and the corresponding target depth information.

[0055] Preferably, the positioning module is further configured to, in response to the feature point not meeting the preset conditions, obtain the positioning information of the sleeve based on the geometric centroid and the depth information corresponding to the geometric centroid.

[0056] Preferably, the image acquisition module is further configured to identify a target object in the image; wherein the sleeve is disposed on the target object;

[0057] The image acquisition module is also used to obtain the contact area between the target object and the sleeve based on the feature points;

[0058] The positioning module is also used to obtain the positioning information of the sleeve based on the feature points and the contact area.

[0059] Preferably, the positioning module is further configured to, in response to the distance between the feature point and the contact area meeting the second distance threshold requirement, use the position information of any feature point that meets the requirement as the positioning information of the sleeve;

[0060] Alternatively, the feature point with the smallest distance from the contact area can be selected, and the position information of the feature point can be used as the positioning information of the sleeve.

[0061] Preferably, the control module includes a coordinate system transformation unit and a displacement control unit;

[0062] The coordinate system transformation unit is used to transform the positioning information of the sleeve from the pixel coordinate system to the robot coordinate system;

[0063] The displacement control unit is used to control the operation end to move based on the robot coordinate system.

[0064] Preferably, the number of operating terminals is greater than or equal to the number of sleeves, and the displacement control unit is further configured to, in response to the fact that the number of sleeves identified in the image is not less than two, acquire the positioning information corresponding to each sleeve; control the number of operating terminals equal to the number of sleeves to match the positioning information of the corresponding sleeve, and each operating terminal moves based on the positioning information of the corresponding sleeve.

[0065] Preferably, the control module further includes a path planning unit;

[0066] The path planning unit is used to obtain the movement path of each of the sleeves and each of the operating ends; and select the movement path that does not intersect with other movement paths as the target path;

[0067] The displacement control unit is also used to select a corresponding number of target paths based on the number of sleeves, and control the operation end corresponding to the target path to move.

[0068] A third aspect provides an electronic device including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the control method of the surgical robot as described above.

[0069] The fourth aspect provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method for the surgical robot as described above.

[0070] The positive and progressive effects of this invention are as follows: by identifying the cannula in the scene where the surgical robot is located, the feature points and depth information of the cannula are obtained, and the position information of the cannula is accurately obtained based on the feature points and depth information, so as to realize the automated movement of the robot's robotic arm based on the cannula position information, replacing manual dragging operation and shortening the movement time of the surgical robot. Attached Figure Description

[0071] Figure 1 This is a flowchart of the control method for the surgical robot according to Embodiment 1 of the present invention;

[0072] Figure 2 This is a schematic diagram of mask recognition in Embodiment 1 of the present invention;

[0073] Figure 3 This is a schematic diagram of the sleeve feature points in Embodiment 1 of the present invention;

[0074] Figure 4 This is a flowchart illustrating the process of acquiring depth information of feature points in the control method of the surgical robot according to Embodiment 1 of the present invention;

[0075] Figure 5 This is a flowchart illustrating the process of obtaining cannula positioning information in the control method of the surgical robot according to Embodiment 1 of the present invention.

[0076] Figure 6 This is a schematic diagram of the scenario of the sleeve and the target object in Embodiment 1 of the present invention;

[0077] Figure 7 This is a schematic diagram of the control system of the surgical robot according to Embodiment 2 of the present invention;

[0078] Figure 8 This is a schematic diagram of the hardware structure of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0079] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0080] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0081] Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] Example 1

[0083] This embodiment provides a control method for a surgical robot, such as... Figure 1 As shown, the control method includes:

[0084] S101. Obtain an image including the sleeve and the depth information of the sleeve;

[0085] S102. Identify the sleeve in the image and obtain the feature points of the sleeve based on the identification result;

[0086] S103. Obtain the target depth information corresponding to the feature points;

[0087] S104. Obtain the positioning information of the sleeve based on the feature points and the target depth information;

[0088] S105. Control the operation end of the surgical robot to move based on the positioning information.

[0089] In this solution, images of the cannula and its depth information are acquired by setting up a camera and a depth sensor. Based on the identification of the cannula and the corresponding depth information, the cannula positioning information can be obtained accurately and quickly without the need for 3D data registration of the cannula. This solution has better universality for different types of cannulas. Based on the cannula positioning information, the surgical robot is controlled to move automatically, and the operation end of the surgical robot can be deployed before the operation, reducing the user's operational burden and improving the user experience.

[0090] As one possible approach, step S102 includes the following steps:

[0091] The image is input into a pre-trained cannula recognition model, which outputs a mask for the cannula.

[0092] The feature points of the sleeve are obtained based on the mask identification.

[0093] In this plan, such as Figure 2 As shown, depending on the user's usage scenario and needs, the acquired image can be an RGB (a color standard) image or a grayscale image. The image's wavelength space is not limited to visible light; false-color images generated from infrared, ultraviolet, or other wavelengths can be used. Preferably, the camera is a multi-view camera or a structured light camera, and the depth sensor is a LiDAR. The pre-trained sleeve recognition model uses a mask extraction model, including but not limited to Mask R-CNN (mask region convolutional neural network) model, and outputs the mask corresponding to the identified sleeve through the mask region convolutional neural network model. This achieves the recognition of sleeves in the image and obtains the feature points corresponding to the sleeve based on the mask recognition result. In one embodiment, such as... Figure 3As shown, the edges of the sleeve are extracted using feature points and algorithms such as the Canny (an edge detection operator). Feature point 1 and its coordinates are obtained by sampling the edges at equal intervals. It should be noted that the feature points can be the geometric center of the image or other points that characterize the sleeve, which can quickly achieve preliminary positioning of the sleeve and has high recognition accuracy.

[0094] As a feasible approach, such as Figure 4 As shown, step S103 includes the following steps:

[0095] S1031. The area of ​​the sleeve in the image is used as the foreground, and the other areas are used as the background;

[0096] S1032. Obtain the first depth information of the foreground pixels and the second depth information of the background pixels within a preset range centered on the feature point;

[0097] S1033. Calculate the target depth information corresponding to the feature point based on the first depth information and the second depth information.

[0098] In this scheme, the image is divided into foreground and background regions based on the mask recognition results. The depth information corresponding to the feature point is calculated based on the depth information of the foreground and background pixels around the feature point. The calculation method includes, but is not limited to, calculating the mean, variance or median of the first depth information and the second depth information. Alternatively, weights are set for the first depth information and the second depth information respectively, and the weighted average of the first depth information and the second depth information is calculated to improve the accuracy of the target depth information corresponding to the feature point.

[0099] As a feasible approach, such as Figure 5 As shown, step S104 includes the following steps:

[0100] S1041. Obtain the geometric centroid of the sleeve;

[0101] S1042. In response to the feature point meeting the preset conditions, the feature point is taken as the target feature point;

[0102] The preset conditions include that the distance between the feature point and the geometric centroid meets a first distance threshold requirement, and that the target depth information meets a depth threshold requirement;

[0103] Among them, meeting the first distance threshold requirement and meeting the depth threshold requirement can be set to greater than, less than or equal to, depending on the actual usage.

[0104] S1043. The positioning information of the sleeve is obtained based on the target feature points and the corresponding target depth information.

[0105] In this scheme, to improve the positioning accuracy of the cannula, fixed feature points such as the geometric centroid of the cannula are used. Feature points are screened based on preset conditions, and the distance between each feature point on the cannula and the center point is calculated as a morphological feature. Euclidean distance, Manhattan distance or Chebyshev distance can be selected. High-precision feature points are obtained based on distance and depth screening, which serve as the positioning information of the surgical robot's moving target.

[0106] As one possible implementation, the control method further includes:

[0107] If the feature point does not meet the preset condition, the positioning information of the sleeve is obtained based on the geometric centroid and the depth information corresponding to the geometric centroid.

[0108] In this scheme, if the feature points cannot meet the preset conditions after obtaining the cannula mask recognition result, the positioning information of the cannula is obtained by selecting a fixed feature point of the cannula, such as the geometric centroid, as the positioning reference of the cannula, and combining the depth information of the geometric centroid point. Preferably, the depth information of the geometric centroid is calculated based on the depth information of the pixels within a preset range around the geometric centroid. Considering the shape characteristics of the cannula, the geometric centroid of the cannula is located above the slender rod of the cannula, and the position information of the geometric centroid is adjusted in combination with the depth value of the cannula inserted into the wound.

[0109] As one possible approach, in response to the feature point not meeting the preset conditions, the user can actively select the identified feature point, obtain the depth information of the selected feature point, and thus obtain the sleeve's positioning information. The active selection can be performed via touchscreen, mouse, or buttons, including but not limited to dragging, clicking, double-clicking, or selecting.

[0110] As one possible approach, after identifying the sleeve in the image, the method further includes:

[0111] Identify the target object in the image;

[0112] The sleeve is disposed on the target object;

[0113] The contact area between the target object and the sleeve is obtained based on the feature points;

[0114] The positioning information of the sleeve is obtained based on the feature points and the contact area.

[0115] In this plan, such as Figure 6 As shown, the camera acquires images of the cannula and the patient with the cannula in place. By identifying the location information of feature points in the contact area between the cannula and the patient, higher stability and accuracy can be achieved. Selecting feature points in or near this area and combining them with the depth information of the corresponding feature points can obtain high-precision cannula positioning information.

[0116] As one possible approach, the step of obtaining the sleeve's positioning information based on the feature points and the contact area includes:

[0117] In response to the distance between the feature point and the contact area meeting the second distance threshold requirement, the position information of any feature point that meets the requirement is used as the positioning information of the sleeve;

[0118] Alternatively, the feature point with the smallest distance from the contact area can be selected, and the position information of the feature point can be used as the positioning information of the sleeve.

[0119] In this solution, feature points near or closest to the contact area are selected as the positioning points of the sleeve based on a preset distance. In one embodiment, feature points on the contact edge between the sleeve and the target object can be selected as the positioning information of the sleeve, so as to avoid the image of the positioning information caused by the position deviation of the feature points due to the swing of the sleeve and improve the accuracy of the sleeve positioning information.

[0120] As one possible approach, step S105 includes the following steps:

[0121] The positioning information of the sleeve is transformed from the pixel coordinate system to the robot coordinate system;

[0122] The control terminal moves based on the robot coordinate system.

[0123] In this solution, the coordinate system of the cannula positioning information is transformed by combining the internal parameters of the camera and the external parameters and matrices of the depth sensor through the Zhang Zhengyou calibration method, enabling the surgical robot to move to the position of the positioning information. Users can confirm or correct the coordinate system transformation results to further improve the accuracy of the subsequent movement of the surgical robot.

[0124] As one possible implementation, the number of operating terminals is greater than or equal to the number of sleeves, and the control method further includes:

[0125] If the number of sleeves identified in the image is not less than two, then the positioning information corresponding to each sleeve is obtained;

[0126] The number of control terminals equal to the number of the sheaths is matched with the positioning information of the corresponding sheaths, and each control terminal moves based on the positioning information of the corresponding sheath.

[0127] In this solution, multiple cannulas are typically placed on the patient's body during abdominal surgery. The positioning information of each cannula is identified and obtained, and the corresponding number of robotic operating ends are controlled to move towards the target positioning information, thereby achieving automated docking between the robotic operating ends and the cannulas.

[0128] One feasible approach is to affix identifiable markings to the cannulas. These markings characterize the surgical instruments or endoscopes that are matched with the cannulas. Based on the identification of the cannulas, the operating end of a surgical robot equipped with the corresponding surgical instruments or endoscopes is controlled to move towards the corresponding cannulas, based on the markings on the cannulas. The markings include, but are not limited to, identifiable graphics such as QR codes or barcodes, or identifiable shapes or structures on the surface or edges of the cannulas. This enables precise pairing and automated movement of multiple cannulas with multiple operating ends.

[0129] As one possible approach, the step of matching the number of control terminals (equal to the number of sleeves) with the positioning information corresponding to the sleeves includes:

[0130] Obtain the movement path of each of the sleeves and each of the operating ends;

[0131] Select the movement path that does not intersect with other movement paths as the target path;

[0132] Based on the number of sleeves, select a corresponding number of target paths and control the operation terminal corresponding to the target path to move.

[0133] In this solution, to avoid collisions between multiple operating ends of the surgical robot, non-intersecting movement paths are selected to ensure the accuracy of automated movement of the operating ends of the surgical robot and to guarantee the operational safety of the surgical robot.

[0134] The surgical robot control method provided in this embodiment identifies the cannula in the scene where the surgical robot is located, obtains the feature points and depth information of the cannula, and obtains the position information of the cannula based on the feature points and depth information. This enables the robotic arm of the robot to move automatically based on the position information of the cannula, replacing manual dragging operations and shortening the movement time of the surgical robot.

[0135] Example 2

[0136] This embodiment provides a control system 200 for a surgical robot, such as... Figure 7 As shown, the control system includes an image acquisition module 201, a recognition module 202, a positioning module 203, and a control module 204;

[0137] The image acquisition module 201 is used to acquire an image including the sleeve and the depth information of the sleeve;

[0138] The recognition module 202 is used to recognize the sleeve in the image and obtain the feature points of the sleeve based on the recognition result;

[0139] The positioning module 203 is used to acquire the target depth information corresponding to the feature point; and is also used to obtain the positioning information of the sleeve based on the feature point and the target depth information.

[0140] The control module 204 is used to control the operating end of the surgical robot to move based on the positioning information.

[0141] In one possible implementation, the recognition module 202 includes a mask output unit and a feature point recognition unit;

[0142] The mask output unit is used to input the image into a pre-trained sleeve recognition model and output the mask of the sleeve.

[0143] The feature point recognition unit is used to identify the feature points of the sleeve based on the mask.

[0144] As one possible implementation, the positioning module 203 is further configured to use the area of ​​the sleeve in the image as the foreground and other areas as the background; acquire first depth information of foreground pixels and second depth information of background pixels within a preset range centered on the feature point; and calculate the target depth information corresponding to the feature point based on the first depth information and the second depth information.

[0145] As one possible approach, the positioning module 203 is also used to obtain the geometric center of gravity of the sleeve; in response to the feature point meeting a preset condition, the feature point is taken as the target feature point;

[0146] The preset conditions include that the distance between the feature point and the geometric centroid meets a first distance threshold requirement, and that the target depth information meets a depth threshold requirement;

[0147] The positioning module 203 is also used to obtain the positioning information of the sleeve based on the target feature points and the corresponding target depth information.

[0148] As one possible approach, the positioning module 203 is also configured to, in response to the feature point not meeting the preset conditions, obtain the positioning information of the sleeve based on the geometric centroid and the depth information corresponding to the geometric centroid.

[0149] In one possible implementation, the image acquisition module 201 is also used to identify a target object in the image; wherein the sleeve is disposed on the target object;

[0150] The image acquisition module 201 is further configured to obtain the contact area between the target object and the sleeve based on the feature points;

[0151] The positioning module 203 is also used to obtain the positioning information of the sleeve based on the feature points and the contact area.

[0152] As one possible implementation, the positioning module 203 is further configured to, in response to the distance between the feature point and the contact area meeting a second distance threshold requirement, use the position information of any feature point that meets the requirement as the positioning information of the sleeve;

[0153] Alternatively, the feature point with the smallest distance from the contact area can be selected, and the position information of the feature point can be used as the positioning information of the sleeve.

[0154] In one possible implementation, the control module 204 includes a coordinate system transformation unit and a displacement control unit;

[0155] The coordinate system transformation unit is used to transform the positioning information of the sleeve from the pixel coordinate system to the robot coordinate system;

[0156] The displacement control unit is used to control the operation end to move based on the robot coordinate system.

[0157] In one possible manner, the number of operating terminals is greater than or equal to the number of sleeves. The displacement control unit is further configured to, in response to the fact that the number of sleeves identified in the image is not less than two, acquire the positioning information corresponding to each sleeve; control the number of operating terminals equal to the number of sleeves to match the positioning information of the corresponding sleeve, and each operating terminal moves based on the positioning information of the corresponding sleeve.

[0158] As one possible implementation, the control module 204 also includes a path planning unit;

[0159] The path planning unit is used to obtain the movement path of each of the sleeves and each of the operating ends; and select the movement path that does not intersect with other movement paths as the target path;

[0160] The displacement control unit is also used to select a corresponding number of target paths based on the number of sleeves, and control the operation end corresponding to the target path to move.

[0161] It should be noted that the working principle of the control system 200 of the surgical robot in this embodiment is the same as that of the control method of the surgical robot in Embodiment 1, so it will not be described again here.

[0162] The control system of the surgical robot provided in this embodiment obtains the feature points and depth information of the cannula by identifying the cannula in the scene where the surgical robot is located. Based on the feature points and depth information, the system accurately obtains the position information of the cannula, enabling the robotic arm to move automatically based on the cannula position information, replacing manual dragging operations and shortening the movement time of the surgical robot.

[0163] Example 3

[0164] like Figure 8 The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of the present invention. It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the control method for the surgical robot described in Embodiment 1. Figure 8 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0165] The electronic device 30 may be in the form of a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0166] Bus 33 includes a data bus, an address bus, and a control bus.

[0167] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0168] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0169] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the control method of the surgical robot in Embodiment 1 of the present invention.

[0170] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generated device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. As shown, network adapter 36 communicates with other modules of the model-generated device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0171] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0172] Example 4

[0173] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method of the surgical robot of Embodiment 1.

[0174] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0175] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to execute the control method for the surgical robot of Embodiment 1.

[0176] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0177] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A control method for a surgical robot, characterized in that, The control method includes: Acquire an image including the cannula and the depth information of the cannula; The sleeve in the image is identified, and the feature points of the sleeve are obtained based on the identification results; Obtain the target depth information corresponding to the feature points; The positioning information of the sleeve is obtained based on the feature points and the target depth information, including: Obtain the geometric centroid of the sleeve; If the feature point meets the preset conditions, then the feature point is taken as the target feature point; The preset conditions include, but are not limited to, the distance between the feature point and the geometric centroid meeting a distance threshold requirement, and the target depth information meeting a depth threshold requirement; The positioning information of the sleeve is obtained based on the target feature points and the corresponding target depth information; The control end of the surgical robot moves based on the positioning information.

2. The control method for the surgical robot according to claim 1, characterized in that, The step of identifying the sleeve in the image and obtaining the feature points of the sleeve based on the identification result includes: The image is input into a pre-trained cannula recognition model, which outputs a mask for the cannula. The feature points of the sleeve are obtained based on the mask identification.

3. The control method for the surgical robot according to claim 2, characterized in that, The step of obtaining the target depth information corresponding to the feature points includes: The area of ​​the sleeve in the image is used as the foreground, and the other areas are used as the background; Obtain first depth information of foreground pixels and second depth information of background pixels within a preset range centered on the feature point; The target depth information corresponding to the feature point is calculated based on the first depth information and the second depth information.

4. The control method for the surgical robot according to claim 1, characterized in that, The control method further includes: If the feature point does not meet the preset condition, the positioning information of the sleeve is obtained based on the geometric centroid and the depth information corresponding to the geometric centroid.

5. The control method for the surgical robot according to claim 1, characterized in that, After identifying the cannula in the image, the process further includes: Identify the target object in the image; The sleeve is disposed on the target object; The contact area between the target object and the sleeve is obtained based on the feature points; The positioning information of the sleeve is obtained based on the feature points and the contact area.

6. The control method for the surgical robot according to claim 5, characterized in that, The step of obtaining the positioning information of the sleeve based on the feature points and the contact area includes: In response to the distance between the feature point and the contact area satisfying the second distance threshold condition, the position information of any feature point that meets the requirements is used as the positioning information of the sleeve; Alternatively, the feature point with the smallest distance from the contact area can be selected, and the position information of the feature point can be used as the positioning information of the sleeve.

7. The control method for the surgical robot according to claim 1, characterized in that, The step of controlling the operating terminal to move based on the positioning information includes: The positioning information of the sleeve is transformed from the pixel coordinate system to the robot coordinate system; The control terminal moves based on the robot coordinate system.

8. The control method for the surgical robot according to claim 1, characterized in that, The number of operating terminals is greater than or equal to the number of sleeves, and the control method further includes: If the number of sleeves identified in the image is not less than two, then the positioning information corresponding to each sleeve is obtained; The number of control terminals equal to the number of the sheaths is matched with the positioning information of the corresponding sheaths, and each control terminal moves based on the positioning information of the corresponding sheath.

9. The control method for the surgical robot according to claim 8, characterized in that, The step of matching the number of control terminals (equal to the number of sleeves) with the positioning information of the corresponding sleeves includes: Obtain the movement path of each of the sleeves and each of the operating ends; Select the movement path that does not intersect with other movement paths as the target path; Based on the number of sleeves, select a corresponding number of target paths and control the operation terminal corresponding to the target path to move.

10. A control system for a surgical robot, characterized in that, The control system includes an image acquisition module, a recognition module, a positioning module, and a control module; The image acquisition module is used to acquire an image including the sleeve and the depth information of the sleeve; The recognition module is used to identify the sleeve in the image and obtain the feature points of the sleeve based on the recognition result; The positioning module is used to acquire target depth information corresponding to the feature points; and is also used to obtain positioning information of the sleeve based on the feature points and the target depth information. The positioning module is also used to obtain the geometric center of gravity of the sleeve; in response to the feature point meeting the preset conditions, the feature point is taken as the target feature point; the preset conditions include the distance between the feature point and the geometric center of gravity meeting the first distance threshold requirement, and the target depth information meeting the depth threshold requirement; The positioning module is also used to obtain the positioning information of the sleeve based on the target feature points and the corresponding target depth information; The control module is used to control the operating end of the surgical robot to move based on the positioning information.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes a computer program, it implements the control method for the surgical robot as described in any one of claims 1-9.

12. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the control method for the surgical robot as described in any one of claims 1-9.

Images