Suture control method, control system, readable storage medium and robot system

Abstract

The present invention provides a suturing control method, a control system, a readable storage medium, and a robot system. The suturing control method includes the following steps: step 1: obtaining a planned suturing trajectory; step 2: driving a first needle holder to clamp a suturing needle and insert it into a suturing object along the planned suturing trajectory, obtaining real-time morphological information of the suturing object based on real-time image information of the suturing area, and updating the planned suturing trajectory based on the real-time morphological information of the suturing object; then driving the first needle holder to clamp the suturing needle and insert it out of the suturing object along the updated planned suturing trajectory; step 3: after the suturing needle has penetrated the suturing object to a predetermined length, driving a second needle holder to clamp the suturing needle, releasing the first needle holder, and then driving the second needle holder to clamp the suturing needle and move it to a predetermined position. Such a configuration can effectively cope with the position change and morphological change of the suturing object caused by the needle threading action, so that the suturing needle can penetrate from a predetermined needle exit point, effectively improving the accuracy of suturing.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a suturing control method, a control system, a readable storage medium and a robot system. Background Art

[0002] Surgical robotic systems are designed to precisely perform complex surgical procedures using minimally invasive methods. Faced with the limitations of traditional surgery, robotic surgical systems have been developed to replace traditional surgery. They offer minimal incision, minimal bleeding, and rapid recovery, significantly shortening postoperative hospital stays and significantly improving survival and recovery rates. They are highly sought after by both doctors and patients, and are now widely used in various clinical procedures as high-end medical devices.

[0003] In some applications, surgical robotic systems can also be used to perform suturing operations. Typically, these systems use two robotic arms to sequentially grasp the suture needle and perform the suturing operation. However, during the suturing process, the needle insertion movement can cause the position of the sutured tissue to shift, resulting in a significant deviation between the actual suturing trajectory and the preset suturing trajectory, leading to low suturing accuracy. Summary of the Invention

[0004] The object of the present invention is to provide a suturing control method, a suturing control system, a readable storage medium and a robot system to solve the problem of low suturing accuracy in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present invention provides a suturing control method, which includes:

[0006] Step 1: Obtain the planned suture trajectory;

[0007] Step 2: driving the first needle holder to clamp the suture needle and insert it into the suture object along the planned suture trajectory, obtaining real-time morphological information of the suture object based on real-time image information of the suture area, and updating the planned suture trajectory based on the real-time morphological information of the suture object; and then driving the first needle holder to clamp the suture needle and insert it out of the suture object along the updated planned suture trajectory;

[0008] Step 3: After the suture needle passes through the suture object to reach a predetermined length, drive the second needle holder to clamp the suture needle, release the first needle holder, and then drive the second needle holder to clamp the suture needle and move it to a predetermined position.

[0009] Optionally, before step 2, the suturing control method further includes: calibrating the relative coordinate relationship between the joint point of the first needle holder and the needle head of the suture needle; in step 2, after driving the first needle holder to clamp the suture needle and penetrate the suture object along the planned suturing trajectory, the real-time coordinates of the needle head of the suture needle are obtained based on the real-time image information of the suture area and the relative coordinate relationship; and then, according to the real-time coordinates of the needle head of the suture needle and based on the updated planned suturing trajectory, the driving trajectory of the first needle holder is corrected to drive the suture needle out of the suture object.

[0010] Optionally, the step of calibrating the relative coordinate relationship between the joint point of the first needle holder and the needle head of the suture needle includes:

[0011] Based on the real-time image information of the suturing area and according to the target detection algorithm, the head of the first needle holder and the suturing needle are identified;

[0012] intercepting an image of the head of the first needle holder, and obtaining coordinates of joint points of the first needle holder based on a key point algorithm;

[0013] intercepting an image of the suture needle and obtaining the coordinates of the needle head of the suture needle according to a key point algorithm;

[0014] Based on the coordinates of the joint point of the first needle holder and the coordinates of the needle head of the suture needle, the relative coordinate relationship between the joint point of the first needle holder and the needle head of the suture needle is calibrated.

[0015] Optionally, in step three, after the step of driving the second needle holder to clamp the suture needle, a judgment is made on whether the second needle holder successfully clamps the suture needle based on the real-time image information of the suture area; if the judgment result is yes, the step of loosening the first needle holder is continued, and then the step of driving the second needle holder to clamp the suture needle and move it to a predetermined position is performed; if the judgment result is no, the step of driving the second needle holder to clamp the suture needle is re-executed.

[0016] Optionally, in step three, the step of judging whether the second needle holder successfully clamps the suture needle based on the real-time image information of the suture area includes: identifying the head of the second needle holder according to the target detection algorithm based on the real-time image information of the suture area; intercepting the image of the second needle holder, and judging whether the suture needle is successfully clamped according to the classification algorithm.

[0017] Optionally, in step three, the step of driving the second needle holder to clamp the suture needle includes:

[0018] Based on the real-time image information of the suture area, a segmented image of the suture needle is identified by an image segmentation algorithm, and the center line and outer contour of the suture needle are extracted according to the segmented image of the suture needle;

[0019] Along the extension direction of the center line of the suture needle, from the tail end of the suture needle toward the needle head, the position at 2 / 3 to 4 / 5 of the length of the center line is set as the second clamping point;

[0020] In combination with the outer contour of the suture needle, a direction of a tangent vector perpendicular to the second clamping point is set as a second clamping direction;

[0021] The second needle holding forceps is driven to clamp the suture needle at the second clamping point along the second clamping direction.

[0022] Optionally, in step 1, the step of obtaining the planned suturing trajectory includes:

[0023] Based on real-time image information of the stitching area, a segmented image of the outer contour of the stitched object is identified by an image segmentation algorithm, and a central axis of the stitched object is extracted according to the segmented image of the outer contour of the stitched object;

[0024] According to the segmented image of the outer contour of the sutured object, a group of entry points and exit points are set in a direction perpendicular to the central axis of the sutured object to obtain the planned suture trajectory.

[0025] Optionally, in step one, the planned suturing trajectory is also updated according to the input information; and / or, in step two, after driving the first needle holder to clamp the suture needle and penetrate the suture object along the planned suturing trajectory, the planned suturing trajectory is also updated according to the input information.

[0026] Optionally, if there is a next stitch to be sutured, the suture control method further includes:

[0027] Step 4: driving the first needle holder to clamp the suture needle and releasing the second needle holder; and

[0028] Step 5: Update the planned suture trajectory of the next stitch to be sutured, and return to repeat steps 2 and 3.

[0029] Optionally, in step four, after driving the first needle holder to clamp the suture needle, a judgment is made as to whether the first needle holder successfully clamps the suture needle based on real-time image information of the suture area; if the judgment result is yes, the step of loosening the second needle holder is continued; if the judgment result is no, the step of driving the first needle holder to clamp the suture needle is re-executed.

[0030] Optionally, in step four, the step of judging whether the first needle holder successfully clamps the suture needle based on the real-time image information of the suture area includes: identifying the head of the first needle holder according to the target detection algorithm based on the real-time image information of the suture area; intercepting the image of the first needle holder, and judging whether the suture needle is successfully clamped according to the classification algorithm.

[0031] Optionally, in step 4, the step of driving the first needle holder to clamp the suture needle includes:

[0032] Based on the real-time image information of the suture area, a segmented image of the suture needle is identified by an image segmentation algorithm, and the center line and outer contour of the suture needle are extracted according to the segmented image of the suture needle;

[0033] Along the extension direction of the center line of the suture needle, from the tail end of the suture needle toward the needle head, a position at 1 / 5 to 1 / 3 of the length of the center line is set as the first clamping point;

[0034] In combination with the outer contour of the suture needle, a direction of a tangent vector perpendicular to the first clamping point is set as a first clamping direction;

[0035] The first needle holding forceps is driven to clamp the suture needle at the first clamping point along the first clamping direction.

[0036] In order to solve the above technical problems, the second aspect of the present invention further provides a suturing control system, which is used to implement the suturing control method described above; the suturing control system includes: an image acquisition module, an analysis and processing module and a motion control module;

[0037] The image acquisition module is used to obtain real-time image information of the suture area;

[0038] The analysis and processing module is used to obtain a planned suturing trajectory and update the planned suturing trajectory based on real-time morphological information of the sutured object;

[0039] The motion control module is used to drive the first needle holder and the second needle holder to move according to the analysis and processing results of the analysis and processing module.

[0040] To solve the above technical problem, the third aspect of the present invention further provides a readable storage medium having a program stored thereon, which implements the steps of the suturing control method described above when the program is executed.

[0041] In order to solve the above technical problems, the fourth aspect of the present invention further provides a robot system, which includes a first needle holding forceps, a second needle holding forceps and the suturing control system as described above.

[0042] To sum up, in the suturing control method, control system, readable storage medium and robot system provided by the present invention, the suturing control method includes: step one: obtaining a planned suturing trajectory; step two: driving a first needle holder to clamp the suturing needle and insert it into the suturing object along the planned suturing trajectory, obtaining real-time morphological information of the suturing object based on real-time image information of the suturing area, and updating the planned suturing trajectory based on the real-time morphological information of the suturing object; and then driving the first needle holder to clamp the suturing needle and pass it out of the suturing object along the updated planned suturing trajectory; step three: after the suturing needle passes through the suturing object to a predetermined length, driving the second needle holder to clamp the suturing needle, releasing the first needle holder, and then driving the second needle holder to clamp the suturing needle and move it to a predetermined position.

[0043] With such a configuration, after the suture needle penetrates the suture object, the real-time morphological information of the suture object can be monitored through the real-time image information of the suture area, and the suture trajectory can be updated and planned based on the real-time morphological information of the suture object. This can effectively cope with the position changes and morphological changes of the suture object caused by the needle-threading action, so that the suture needle can penetrate from the predetermined needle exit point, effectively improving the accuracy of suturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0045] Figure 1 is a schematic diagram of a suturing control system according to an embodiment of the present invention;

[0046] Figure 2a to Figure 2f 1 is a schematic diagram of the operation flow of the suturing control method according to an embodiment of the present invention;

[0047] Figure 3a to Figure 3c is a schematic diagram of a sutured object deformed by the puncture of a suture needle according to an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a target detection algorithm according to an embodiment of the present invention;

[0049] Figure 5a and Figure 5b is a schematic diagram of a key point algorithm according to an embodiment of the present invention;

[0050] Figure 6a and Figure 6b is a schematic diagram of a classification algorithm according to an embodiment of the present invention;

[0051] Figure 7 is a schematic diagram of determining a clamping point and a clamping direction according to an embodiment of the present invention;

[0052] Figure 8 is a schematic diagram of an image segmentation algorithm according to an embodiment of the present invention;

[0053] Figure 9 1 is a schematic diagram of the operational flow of obtaining a planned suturing trajectory according to an embodiment of the present invention;

[0054] Figure 10 2 is a schematic diagram of gradually updating the planned suturing trajectory according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the objects, advantages, and features of the present invention more clearly apparent, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale. They are only used to conveniently and clearly assist in illustrating the purposes of the embodiments of the present invention. In addition, the structures shown in the drawings are often part of the actual structure. In particular, different drawings may need to illustrate different focuses and sometimes use different scales.

[0056] As used herein, the singular forms "a," "an," and "the" include plural referents, the term "or" is generally used in a sense that includes "and / or," the term "several" is generally used in a sense that includes "at least one," and the term "at least two" is generally used in a sense that includes "two or more." Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Thus, features designated "first," "second," and "third" may explicitly or implicitly include one or at least two of the features. References to "one end" and "the other end" and "proximal" and "distal" generally refer to corresponding portions and not just endpoints. In manual or hand-operated applications, the terms "proximal" and "distal" are defined herein relative to an operator, such as a surgeon or clinician. The term "proximal" refers to a location of an element closer to the operator, and the term "distal" refers to a location of an element closer to the surgical instrument and, therefore, farther from the operator. In addition, as used in the present invention, "installed", "connected", "connected", and one element is "set" on another element should be understood in a broad sense, usually only indicating that there is a connection, coupling, cooperation or transmission relationship between the two elements, and the connection, coupling, cooperation or transmission between the two elements can be direct or indirect through an intermediate element, and cannot be understood as indicating or implying the spatial position relationship between the two elements, that is, one element can be in any orientation such as inside, outside, above, below or on one side of another element, unless the content clearly indicates otherwise. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In addition, directional terms such as above, below, up, down, upward, downward, left, right, etc. are used relative to the exemplary embodiments as they are shown in the figures, with the upward or upper direction toward the top of the corresponding figure, and the downward or lower direction toward the bottom of the corresponding figure.

[0057] The object of the present invention is to provide a suturing control method, a control system, a readable storage medium and a robot system to solve the problem of low suturing accuracy in the prior art.

[0058] The following description is given with reference to the accompanying drawings.

[0059] An embodiment of the present invention provides a robot system, in an application scenario for automatic suturing, the robot system includes an execution end, the execution end includes at least two robotic arms, a first needle holder 11 and a second needle holder 12 (see Figure 2d). The first needle holding forceps 11 and the second needle holding forceps 12 are respectively mounted on two robotic arms, which are used to move under command control and drive the first needle holding forceps 11 and the second needle holding forceps 12 mounted thereon to move so as to perform suturing operations. In some application scenarios, the robotic system is a surgical robot system. It can be understood that the surgical robot system can be a master-slave remote-operated surgical robot system, in which the execution end is its slave end, or a single-end surgical robot system, in which the operator directly operates the execution end to perform the operation, and the present invention is not limited to this.

[0060] like Figure 1 As shown, the surgical robot system includes a suturing control system, which includes an image acquisition module 81, an analysis and processing module 82, and a motion control module 83. The image acquisition module 81 is used to acquire real-time image information of the suturing area, and the analysis and processing module 82 is used to obtain a planned suturing trajectory and update the planned suturing trajectory based on the real-time morphological information of the sutured object. The motion control module 83 is used to drive the first needle holder 11 and the second needle holder 12 to move according to the analysis and processing results of the analysis and processing module 82. In one exemplary embodiment, the image acquisition module 81 includes a binocular endoscope for acquiring stereoscopic image data within the abdominal cavity. The image acquisition module 81 may also include an external camera or other image acquisition device commonly used in the art. Preferably, the image acquisition module 81 also includes a cold light source for supplemental illumination during binocular endoscope imaging. Preferably, the suturing control system also includes a display device or other module for displaying the real-time image information of the suturing area acquired by the image acquisition module 81. Optionally, the analysis and processing module 82 includes a server, embedded device, or other device commonly used in the art with image data analysis and processing capabilities. The motion control module 83 includes, for example, a main control device and a motion controller of the surgical robot, and is capable of executing drive control of the robotic arm, the first needle holder 11, and the second needle holder 12 according to instructions. Preferably, the suturing control system further includes a human-computer interaction module 84, which includes, but is not limited to, a mouse, a keyboard, a touch screen, etc., for inputting information and outputting feedback information such as image information.

[0061] Based on the suture control system described above, please refer to Figure 2a to Figure 2f An embodiment of the present invention provides a suturing control method for performing a suturing operation on a suturing object 7. The suturing control method can be implemented based on the above-mentioned suturing control system. In an exemplary embodiment, the suturing object 7 can be a wound on a patient. It should be noted that the suturing object 7 is not limited to a wound on a patient, but can also be a wound model prosthesis, etc., which can be used for operator training or surgical verification, etc. The present invention does not limit the suturing application scenario of the robotic system.

[0062] The suturing control method comprises:

[0063] Step 1 S1: Get the planned suture trajectory; Figure 2a As shown, the planned suture trajectory includes a group of entry points 101 and exit points 102 distributed on both sides of the suture object 7. Optionally, around the suture object 7, if multiple stitches need to be sutured, there may be multiple planned suture trajectories, which constitute a planned suture trajectory set. The multiple planned suture trajectories are preferably arranged at equal intervals. The planned suture trajectory set can be pre-set, for example, it can be calculated by the analysis and processing module 82, or it can be obtained according to the operator input. Step 1 S1 does not limit the acquisition method of the planned suture trajectory and the planned suture trajectory set. In particular, the planned suture trajectory obtained by step 1 S1 refers to a specific planned suture trajectory in the planned suture trajectory set, preferably the planned suture trajectory of the first needle located at the end of the suture object 7 in the planned suture trajectory set. Of course, if only one stitch is needed, it can be understood that there is only one planned suture trajectory in the planned suture trajectory set.

[0064] Step 2 S2: Figure 2b As shown, the first needle holder 11 is driven to clamp the suture needle 2 and penetrate the suture object 7 along the planned suture trajectory. Based on the real-time image information of the suture area, the real-time morphological information of the suture object 7 is obtained, and the planned suture trajectory is updated based on the real-time morphological information of the suture object 7; then the first needle holder is driven to clamp the suture needle and penetrate the suture object 7 along the updated planned suture trajectory; it should be noted that the suture area refers to an area including the suture object 7 and the adjacent space of the suture object 7, and its range can be set according to actual surgical needs. Preferably, the real-time image information of the suture area includes at least part of the suture object 7, the suture needle 2, the first needle holder 11 and the second needle holder 12. It can be understood that the suture needle 2 penetrates the suture object 7 along the planned suture trajectory from the penetration point 101 of the planned suture trajectory, and exits the suture object 7 from the exit point 102.

[0065] Please refer to Figures 3a to 3cSince the suture object 7 is generally an elastic tissue, the suture object 7 will produce a certain deformation under the action of the puncture force when the suture needle 2 penetrates and exits the suture object 7. As a result, the planned suture trajectory obtained before suturing is no longer applicable to the subsequent suture path. In step S2, after the suture needle 2 penetrates the suture object 7 for a certain length, it can temporarily stop penetrating. At this time, by identifying the real-time image information of the suture area, the real-time morphological information of the suture object 7 can be obtained, and based on the real-time morphological information of the current suture object 7, the planned suture trajectory is updated, such as adjusting the position of the exit point 102. This can effectively cope with the position change and morphological change of the suture object 7 caused by the needle threading action, so that the suture needle 2 can penetrate from the updated exit point 102, effectively improving the accuracy of suture. It should be noted that the steps of obtaining the real-time morphological information of the sutured object 7 based on the real-time image information of the sutured area and updating the planned suture trajectory based on the real-time morphological information of the sutured object 7 can be calculated by the analysis and processing module 82, or the operator can manually identify the real-time morphological information of the sutured object 7 and redraw and input it through the human-computer interaction module 84.

[0066] Step 3 S3: After the suture needle 2 passes through the suture object 7 to a predetermined length (e.g. Figure 2c As shown), drive the second needle holder 12 to clamp the suture needle 2 (as shown Figure 2d As shown), loosen the first needle holder 11 (as Figure 2e ), and then driving the second needle holder 12 to grip the suture needle 2 and move it to a predetermined position; in this step, the predetermined length can be set based on the length and shape of the suture needle 2 and the size of the suture object 7, and the predetermined position can be set according to the actual surgical scenario. In one exemplary embodiment, the process of driving the second needle holder 12 to grip the suture needle 2 and move it to the predetermined position requires avoiding obstacles such as the suture object 7 and passing over the suture object 7 until reaching the predetermined position.

[0067] Optionally, if there is a next needle to be sutured, the suture control method further includes step four S4: driving the first needle holding forceps 11 to clamp the suture needle 2 (such as Figure 2f As shown), loosen the second needle holder 12; and

[0068] Step 5 S5: Update the planned suture trajectory for the next stitch, and return to repeat step 2 S2 and step 3 S3. Optionally, updating the planned suture trajectory for the next stitch refers to selecting the planned suture trajectory of the next stitch adjacent to the stitch currently being sutured from the planned suture trajectory set. Therefore, when returning to repeat step 2 S2 and step 3 S3, the next stitch adjacent to the stitch currently being sutured will be sutured. Once all planned suture trajectories in the planned suture trajectory set along a certain direction of the suture object 7 have been sutured, the suture is complete.

[0069] With such a configuration, after the suture needle 2 penetrates the suture object, the real-time morphological information of the suture object 7 can be monitored through the real-time image information of the suture area, and the suture trajectory can be updated and planned based on the real-time morphological information of the suture object 7, which can effectively cope with the position changes and morphological changes of the suture object 7 caused by the needle-threading action, so that the suture needle 2 can pass through the predetermined exit point 102, effectively improving the accuracy of suturing.

[0070] Optionally, before step 2 S2, the suturing control method further includes:

[0071] Step S1a: calibrating the relative coordinate relationship between the joint point 111 of the first needle holder 11 and the needle head 21 of the suture needle 2;

[0072] In step two S2, after driving the first needle holder 11 to clamp the suture needle 2 and penetrate the suture object 7 along the planned suture trajectory, the real-time coordinates of the needle head 21 of the suture needle 2 are obtained based on the real-time image information of the suture area and the relative coordinate relationship; and then, according to the real-time coordinates of the needle head 21 of the suture needle 2 and based on the updated planned suture trajectory, the driving trajectory of the first needle holder 11 is corrected to drive the suture needle 2 to penetrate the suture object 7.

[0073] Generally, such as Figure 3a As shown, the suture needle 2 has a needle head 21 and a tail end 22, the needle head 21 is used for puncture, and the tail end 22 is connected to the suture. It can be understood that when the first needle holding forceps 11 is driven to clamp the suture needle 2 and insert it into the suture object 7, the needle head 21 of the suture needle 2 is inserted into the insertion point 101 of the planned suture trajectory, and then according to the curvature of the suture needle 2, the first needle holding forceps 11 is driven to rotate the suture needle 2 and insert it into the suture object 7. It can be understood that before the suture needle 2 penetrates the suture object 7, the needle head 21 of the suture needle 2 can be captured by the image acquisition module 81, and then the analysis and processing module 82 can obtain the real-time coordinates of the needle head 21 through calculation. However, once the suture needle 2 penetrates the suture object 7, the needle head 21 of the suture needle 2 is obscured by the suture object 7 (such as Figure 3bAs shown), it cannot be directly captured by the image acquisition module 81. At this time, it is equivalent to losing the real-time coordinates of the needle 21. After the planned suture trajectory changes with the position change and shape change of the suture object 7 caused by the needle threading action, it is difficult to accurately control the actual trajectory of the suture needle 2. The inventors have found that since the first needle holding forceps 11 includes a joint point 111, after the first needle holding forceps 11 clamps the suture needle 2, the coordinates of the joint point 111 and the coordinates of the needle 21 are relatively fixed, and the joint point 111 is always outside the suture object 7 during the entire suturing process. Therefore, the coordinates of the needle 21 can be indirectly obtained by monitoring the coordinates of the joint point 111. Based on the above research, in this embodiment, before step two S2, the relative coordinate relationship between the joint point 111 of the first needle holder 11 and the needle head 21 of the suture needle 2 has been first calibrated based on step S1a. In this way, it is only necessary to capture the coordinates of the joint point 111 of the first needle holder 11 through the image acquisition module 81 to convert the real-time coordinates of the needle head 21 located inside the suture object 7, so that the movement trajectory of the suture needle 2 can be accurately controlled, effectively improving the accuracy of suturing.

[0074] Optionally, the step S1a of calibrating the relative coordinate relationship between the joint point 111 of the first needle holder 11 and the needle head 21 of the suture needle 2 includes:

[0075] Step S1a1: Based on the real-time image information of the suture area and according to the target detection algorithm, the head of the first needle holder 11 and the suture needle 2 are identified;

[0076] Step S1a2: intercepting an image of the head of the first needle holder 11, and obtaining the coordinates of the joint point 111 of the first needle holder 11 according to a key point algorithm;

[0077] Step S1a3: intercepting the image of the suture needle 2 and obtaining the coordinates of the needle head 21 of the suture needle 2 according to a key point algorithm;

[0078] Step S1a4: calibrate the relative coordinate relationship between the joint point 111 of the first needle holder 11 and the needle head 21 of the suture needle 2 based on the coordinates of the joint point 111 of the first needle holder 11 and the coordinates of the needle head 21 of the suture needle 2.

[0079] If the target detection algorithm and key point algorithm can be based on a neural network algorithm, those skilled in the art can select appropriate target detection algorithms and key point algorithms based on existing technologies to implement the above steps. Figure 4 5 , the target detection algorithm and the key point algorithm are exemplarily illustrated.

[0080] Please refer to Figure 4, which shows the target frame 110 of the head of the first needle holder 11 and the target frame 20 of the suture needle 2 identified by the target detection algorithm. Among them, Conv represents convolution operation, NMS represents non-maximum suppression, FC represents full connection, and rule represents suture linear mapping. The convolution layer uses the Google inceptionV1 network, corresponding to Figure 4 The first stage in the network consists of 20 layers. This layer is mainly used for feature extraction to improve the generalization ability of the model. The target detection layer first passes through 4 convolutional layers and 2 fully connected layers, and finally generates a 7x7x30 output. The purpose of passing through 4 convolutional layers first is to improve the generalization ability of the model. Yolo divides a 448x448 original image into 7x7 grids, and then each cell is responsible for detecting targets whose center points fall within the grid. The NMS screening layer is to filter out the most suitable ones from multiple results (multiple bounding boxes). First, filter out the boxes with scores lower than the threshold, perform NMS non-maximum suppression on the remaining boxes, and remove boxes with relatively high overlap. In this way, the final most suitable boxes and their categories are obtained.

[0081] Please refer to Figure 5a and Figure 5b , which shows the process of extracting the joint point 111 of the first needle holder 11 using the key point algorithm. Figure 5a In the illustrated process, "Conv" represents convolution, "Concat" represents feature map channel fusion, "Upsample" represents upsampling, "BatchNorm2d" represents batch normalization, and "Relu" represents nonlinear mapping. The real-time image information of the sutured area captured by the image acquisition module 81 is compressed to 512 x 512 and then input into the feature extraction network (feature extractor). Features f4, f3, f2, and f1 output by layers 1, 2, 3, and 4 are fed into the feature fusion layer. After upsampling, f1 is concatenated with f2 and then subjected to 1x1 and 3x3 convolutions to obtain h2. The same operation is performed to obtain h3 and h4. h4 undergoes a 3x3 convolution and is then fed into the convolution prediction branch to produce a score map. The coordinates of the first needle holder 11 in the score map are displayed as a heat map. Peak values are calculated to determine the coordinates of all nodes 11 associated with the first needle holder 11, which are then fed back into the original image through coordinate mapping.

[0082] Optionally, in step three S3, after the step of driving the second needle holder 12 to clamp the suture needle 2, it is also judged whether the second needle holder 12 successfully clamps the suture needle 2 based on the real-time image information of the suture area; if the judgment result is yes, continue to execute the step of loosening the first needle holder 11, and then drive the second needle holder 12 to clamp the suture needle 2 and move it to a predetermined position; if the judgment result is no, re-execute the step of driving the second needle holder 12 to clamp the suture needle 2.

[0083] In step three S3, when the length of the suture needle 2 reaches a predetermined length, the second needle holder 12 can be used to clamp the suture needle 2 and start the needle exchange action. However, the failure of the needle exchange action often occurs in previous practices. Therefore, this embodiment also adds an analysis and judgment step on whether the second needle holder 12 successfully clamps the suture needle 2. Only when the second needle holder 12 accurately clamps the suture needle 2 will the first needle holder 11 release its clamping of the suture needle 2, and the second needle holder 12 can pull the suture needle 2 out of the suture object 7. Otherwise, it is necessary to continue to adjust the position of the second needle holder 12 until the second needle holder 12 can accurately clamp the suture needle 2.

[0084] Optionally, in step three S3, the step of judging whether the second needle holder successfully clamps the suture needle based on the real-time image information of the suture area includes:

[0085] Step S31: Based on the real-time image information of the suture area and according to the target detection algorithm, the head of the second needle holder 12 is identified;

[0086] Step S32: capturing an image of the second needle holder 12 and determining whether the suture needle 2 is successfully clamped based on a classification algorithm.

[0087] The target detection algorithm here can be referred to above and will not be repeated here. Classification algorithms include but are not limited to image processing algorithms, artificial intelligence algorithms and deep learning algorithms. Figure 6a and Figure 6b , the classification algorithm is demonstrated. Figure 6aIn the process shown, conv represents convolution operation, Maxpool represents maximum pooling, and FC represents full connection. Traditional convolutional networks or fully connected networks will more or less have problems such as information loss and loss when transmitting information. At the same time, it may cause gradient disappearance or gradient explosion, making it impossible to train very deep networks. In this embodiment, the network used for classification uses a network similar to VGG, and improves it on this basis by adding residual units through a short-circuit mechanism. The basic unit structure is still the convolution, BN, and activation function architecture, but a residual connection is added to the output position of each unit. The unit output is added to the unit input and finally passes through an activation function as the final output. To a certain extent, the problem of information loss and loss is solved. By directly bypassing the input information to the output, the integrity of the information is protected. The entire network only needs to learn the difference between the input and output, which simplifies the learning objectives and difficulty.

[0088] For further information, please refer to Figure 7 In step three S3, the step of driving the second needle holder 12 to clamp the suture needle 2 includes:

[0089] Step S61: Based on the real-time image information of the suture area, the segmented image of the suture needle 2 is identified by an image segmentation algorithm, and the center line 23 (referring to the central axis extending along the bending direction of the suture needle 2) and the outer contour 24 of the suture needle 2 are extracted according to the segmented image of the suture needle 2.

[0090] Step S62: along the extension direction of the center line 23 of the suture needle, from the tail end 22 of the suture needle toward the needle head 21, set the position at 2 / 3 to 4 / 5 of the length of the center line 23 as the second clamping point 26.

[0091] Step S63 : In combination with the outer contour 24 of the suture needle 2 , the direction of the tangent vector 261 passing through the second clamping point 26 is set as the second clamping direction 28 .

[0092] Step S64 : driving the second needle holding forceps 12 to clamp the suture needle 2 at the second clamping point 26 along the second clamping direction 28 .

[0093] In step S61, those skilled in the art can select a suitable image segmentation algorithm according to the existing technology. Figure 8, which shows an example of an image segmentation algorithm. The basic network framework is a classic fully convolutional network (i.e., there is no fully connected operation in the network). The input of the network is a 572 * 572 image with mirrored edges (input image tile), which can be obtained by compression processing of real-time image information of the stitching area. The left side of the network is a series of downsampling operations consisting of convolution and Max Pooling. The compression path consists of 4 blocks, each of which uses 3 effective convolutions and 1 Max Pooling downsampling. After each downsampling, the number of Feature Maps is multiplied by 2, so there is Figure 8 The Feature Map size changes shown in . Finally, a Feature Map of size 32 * 32 is obtained. The right part of the network is called the expansive path. It is also composed of 4 blocks. Before the start of each block, the size of the Feature Map is multiplied by 2 through deconvolution, and its number is halved (the last layer is slightly different), and then merged with the Feature Map of the symmetrical compression path on the left. Since the sizes of the Feature Maps of the compression path on the left and the expansion path on the right are different, the network is normalized by cropping the Feature Map of the compression path to the same size as the expansion path. The convolution operation of the expansion path still uses the effective convolution operation, and the final Feature Map size is 338 * 338. Since this task is a binary classification task, the network has two output Feature Maps.

[0094] Optionally, in step 4 S4, the step of driving the first needle holder 11 to clamp the suture needle 2 includes:

[0095] Step S71: Based on the real-time image information of the suture area, the segmented image of the suture needle 2 is identified by an image segmentation algorithm, and the center line 23 (referring to the central axis extending along the bending direction of the suture needle 2) and the outer contour 24 of the suture needle 2 are extracted according to the segmented image of the suture needle 2.

[0096] Step S72: along the extension direction of the center line 23 of the suture needle, from the tail end 22 of the suture needle toward the needle head 21, set the position at 1 / 5 to 1 / 3 of the length of the center line 23 as the first clamping point 25.

[0097] Step S73 : In combination with the outer contour 24 of the suture needle 2 , the direction of the tangent vector 251 passing through the first clamping point 25 is set as the first clamping direction 27 .

[0098] Step S74: driving the first needle holder 11 to clamp the suture needle 2 at the first clamping point 25 along the first clamping direction 27. The setting principle, setting purpose and implementation method of steps S71 to S74 can be referred to the aforementioned steps S61 to S64 and will not be repeated here.

[0099] Optionally, in step four S4, after the step of driving the first needle holder 11 to clamp the suture needle 2, it is also judged whether the first needle holder 11 successfully clamps the suture needle 2 based on the real-time image information of the suture area; if the judgment result is yes, the step of loosening the second needle holder 12 is continued; if the judgment result is no, the step of driving the first needle holder 11 to clamp the suture needle 2 is re-executed.

[0100] Optionally, in step 4 S4, the step of judging whether the first needle holder 11 successfully clamps the suture needle 2 based on the real-time image information of the suture area includes:

[0101] Step S41: Based on the real-time image information of the suture area and according to the target detection algorithm, the head of the first needle holder 11 is identified;

[0102] Step S42: Capture the image of the first needle holder 11 and determine whether the suture needle 2 is successfully clamped based on the classification algorithm. The setting principle, setting purpose and implementation method of steps S41 and S42 can be referred to the aforementioned steps S31 and S32 and will not be repeated here.

[0103] Similarly, after the second needle holder 12 clamps the suture needle 2 and moves to the predetermined position, an analysis and judgment step can be added to determine whether the first needle holder 11 successfully clamps the suture needle 2. Only when the first needle holder 11 accurately clamps the suture needle 2 will the second needle holder 12 release its clamp on the suture needle 2. Otherwise, the position of the first needle holder 11 needs to be adjusted until the first needle holder 11 can accurately clamp the suture needle 2 to avoid the suture needle 2 falling off due to not being clamped by any needle holder.

[0104] Optional, please refer to Figure 9 In step 1 S1, the step of obtaining the planned suturing trajectory includes:

[0105] Step S11: Based on the real-time image information of the stitching area, a segmented image 71 of the outer contour of the stitched object 7 is identified by an image segmentation algorithm, and the central axis 72 of the stitched object 7 is extracted based on the segmented image 71 of the outer contour of the stitched object 7. The image segmentation algorithm in step S11 can be referred to above and will not be repeated here.

[0106] Step S12: according to the segmented image 71 of the outer contour of the sutured object 7, a group of entry points 101 and exit points 102 are set in a direction perpendicular to the central axis 72 of the sutured object 7 to obtain the planned suture trajectory.

[0107] Preferably, in step 1 S1, the planned suturing trajectory is also updated according to the input information; and / or, in step 2 S2, after driving the first needle holder 11 to clamp the suture needle 2 and penetrate the suture object 7 along the planned suturing trajectory, the planned suturing trajectory is also updated according to the input information. In some embodiments, the planned suturing trajectory is not limited to being automatically calculated by the analysis and processing module 82, but can also be obtained by input by an operator (such as a doctor). Optionally, in step 1 S1 and step 2 S2, the operator can evaluate the feasibility of the planned suturing trajectory automatically calculated by the analysis and processing module 82 on the basis of the planned suturing trajectory automatically calculated by the analysis and processing module 82, and then input a modified trajectory to guide the adjustment and update of the planned suturing trajectory, or can manually outline and input the planned suturing trajectory without basing it on the planned suturing trajectory automatically calculated by the analysis and processing module 82. Based on this, the suturing control system also includes a human-computer interaction module, which is used to obtain the input information input by the operator. As Figure 10 As shown, it shows the process of gradually updating the planned suture trajectory as each stitch is sutured.

[0108] Furthermore, an embodiment of the present invention also provides a readable storage medium having a program stored thereon, which, when executed, implements the steps of the suturing control method described above. Furthermore, an embodiment of the present invention also provides a computer device comprising a processor and the readable storage medium described above, the processor being configured to execute the program stored on the readable storage medium. The readable storage medium may be provided independently or integrated into a robotic system, and the present invention is not limited thereto.

[0109] In summary, in the suturing control method, control system, readable storage medium, and robot system provided by the present invention, the suturing control method includes: step 1: obtaining a planned suturing trajectory; step 2: driving a first needle holder to clamp the suturing needle along the planned suturing trajectory to penetrate the suturing object, based on the real-time image information of the suturing area, obtaining the real-time morphological information of the suturing object, and updating the planned suturing trajectory based on the real-time morphological information of the suturing object; then driving the first needle holder to clamp the suturing needle along the updated planned suturing trajectory to penetrate the suturing object; step 3: after the suturing needle penetrates the suturing object to a predetermined length, driving a second needle holder to clamp the suturing needle, releasing the first needle holder, and then driving the second needle holder to clamp the suturing needle to a predetermined position. With such a configuration, after the suturing needle penetrates the suturing object, the real-time morphological information of the suturing object can be monitored through the real-time image information of the suturing area, and the planned suturing trajectory can be updated based on the real-time morphological information of the suturing object, which can effectively cope with the position change and morphological change of the suturing object caused by the needle threading action, so that the suturing needle can penetrate from the predetermined needle exit point, effectively improving the accuracy of suturing.

[0110] It should be noted that the above embodiments can be combined with each other. The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention. Any changes and modifications made by ordinary technicians in the field of the present invention based on the above disclosure are within the scope of protection of the claims.

Claims

1. A readable storage medium having a program stored thereon, characterized in that: When the program is executed, the following steps are implemented: Step 1: Obtain the planned suture trajectory; Step 2: driving the first needle holder to clamp the suture needle and insert it into the suture object along the planned suture trajectory, obtaining real-time morphological information of the suture object based on real-time image information of the suture area, and updating the planned suture trajectory based on the real-time morphological information of the suture object; and then driving the first needle holder to clamp the suture needle and insert it out of the suture object along the updated planned suture trajectory; The stitched object will undergo a certain deformation; Step 3: After the suture needle has passed through the suture object to a predetermined length, the second needle holder is driven to clamp the suture needle, the first needle holder is released, and the second needle holder is driven to clamp the suture needle and move it to a predetermined position; Wherein, the first needle holding forceps and the second needle holding forceps are respectively mounted on two robotic arms.

2. The readable storage medium according to claim 1, wherein Before step 2, when the program is executed, it also implements: calibrating the relative coordinate relationship between the joint point of the first needle holder and the needle head of the suture needle; in step 2, after driving the first needle holder to clamp the suture needle and penetrate the suture object along the planned suture trajectory, the real-time coordinates of the needle head of the suture needle are obtained based on the real-time image information of the suture area and the relative coordinate relationship; and then, according to the real-time coordinates of the needle head of the suture needle and based on the updated planned suture trajectory, the driving trajectory of the first needle holder is corrected to drive the suture needle out of the suture object.

3. The readable storage medium according to claim 2, wherein: The step of calibrating the relative coordinate relationship between the joint point of the first needle holder and the needle head of the suture needle includes: Based on the real-time image information of the suturing area and according to the target detection algorithm, the head of the first needle holder and the suturing needle are identified; intercepting an image of the head of the first needle holder, and obtaining coordinates of joint points of the first needle holder based on a key point algorithm; intercepting an image of the suture needle and obtaining the coordinates of the needle head of the suture needle according to a key point algorithm; Based on the coordinates of the joint point of the first needle holder and the coordinates of the needle head of the suture needle, the relative coordinate relationship between the joint point of the first needle holder and the needle head of the suture needle is calibrated.

4. The readable storage medium according to claim 1, wherein In step three, after the step of driving the second needle holder to clamp the suture needle, it is also judged whether the second needle holder successfully clamps the suture needle based on the real-time image information of the suture area; if the judgment result is yes, the step of loosening the first needle holder is continued, and then the step of driving the second needle holder to clamp the suture needle and move it to a predetermined position is executed; if the judgment result is no, the step of driving the second needle holder to clamp the suture needle is executed again.

5. The readable storage medium according to claim 4, wherein: In step three, the step of judging whether the second needle holder successfully clamps the suture needle based on the real-time image information of the suture area includes: based on the real-time image information of the suture area, according to the target detection algorithm, identifying the head of the second needle holder; intercepting the image of the second needle holder, and judging whether the suture needle is successfully clamped according to the classification algorithm.

6. The readable storage medium according to claim 1, wherein: In step three, the step of driving the second needle holder to clamp the suture needle includes: Based on the real-time image information of the suture area, a segmented image of the suture needle is identified by an image segmentation algorithm, and the center line and outer contour of the suture needle are extracted according to the segmented image of the suture needle; Along the extension direction of the center line of the suture needle, from the tail end of the suture needle toward the needle head, the position at 2 / 3 to 4 / 5 of the length of the center line is set as the second clamping point; In combination with the outer contour of the suture needle, a direction of a tangent vector perpendicular to the second clamping point is set as a second clamping direction; The second needle holding forceps is driven to clamp the suture needle at the second clamping point along the second clamping direction.

7. The readable storage medium according to claim 1, wherein: In step 1, the step of obtaining the planned suturing trajectory includes: Based on real-time image information of the stitching area, a segmented image of the outer contour of the stitched object is identified by an image segmentation algorithm, and a central axis of the stitched object is extracted according to the segmented image of the outer contour of the stitched object; According to the segmented image of the outer contour of the sutured object, a group of entry points and exit points are set in a direction perpendicular to the central axis of the sutured object to obtain the planned suture trajectory.

8. The readable storage medium according to claim 1, wherein: In step one, the planned suturing trajectory is also updated according to the input information; and / or, in step two, after driving the first needle holder to clamp the suturing needle and penetrate the suturing object along the planned suturing trajectory, the planned suturing trajectory is also updated according to the input information.

9. The readable storage medium according to claim 1, wherein: If there is a next stitch to be sewn, the program also achieves the following when executed: Step 4: driving the first needle holder to clamp the suture needle and releasing the second needle holder; and Step 5: Update the planned suture trajectory of the next stitch to be sutured, and return to repeat steps 2 and 3.

10. The readable storage medium according to claim 9, wherein: In the fourth step, after the step of driving the first needle holder to clamp the suture needle, a judgment is made as to whether the first needle holder successfully clamps the suture needle based on real-time image information of the suture area; if the judgment result is yes, the step of releasing the second needle holder is continued; If the judgment result is no, the step of driving the first needle holding forceps to clamp the suture needle is executed again.

11. The readable storage medium according to claim 10, wherein: In step four, the step of judging whether the first needle holder successfully clamps the suture needle based on the real-time image information of the suture area includes: based on the real-time image information of the suture area, according to the target detection algorithm, identifying the head of the first needle holder; intercepting the image of the first needle holder, and judging whether the suture needle is successfully clamped according to the classification algorithm.

12. The readable storage medium according to claim 10, wherein: In step 4, the step of driving the first needle holder to clamp the suture needle includes: Based on the real-time image information of the suture area, a segmented image of the suture needle is identified by an image segmentation algorithm, and the center line and outer contour of the suture needle are extracted according to the segmented image of the suture needle; Along the extension direction of the center line of the suture needle, from the tail end of the suture needle toward the needle head, a position at 1 / 5 to 1 / 3 of the length of the center line is set as the first clamping point; In combination with the outer contour of the suture needle, a direction of a tangent vector perpendicular to the first clamping point is set as a first clamping direction; The first needle holding forceps is driven to clamp the suture needle at the first clamping point along the first clamping direction.

13. A suturing control system, characterized in that: The method comprises a readable storage medium according to any one of claims 1 to 12; the suturing control system comprises: an image acquisition module, an analysis and processing module, and a motion control module; The image acquisition module is used to obtain real-time image information of the suture area; The analysis and processing module is used to obtain a planned suturing trajectory and update the planned suturing trajectory based on real-time morphological information of the sutured object; The motion control module is used to drive the first needle holder and the second needle holder to move according to the analysis and processing results of the analysis and processing module.

14. A robot system, characterized in that: It comprises a first needle holding forceps, a second needle holding forceps and a suturing control system according to claim 13.

Images