Intelligent surgical instrument and its combined method for staple line detection and marking

Abstract

Intelligent surgical instruments and systems that combine local sensing, computer vision, and artificial intelligence algorithms to detect staple lines and potential abnormalities, and integrate the detected information into laparoscopic or open surgical video using an AI data processing unit.

Description

[Technical Field]

[0001] Cross-Citation of Related Applications This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 385,991, filed December 4, 2022, the entire text of which is incorporated herein by reference. [Background technology]

[0002] Surgical staplers have been used as instruments in open and laparoscopic surgery. Particularly since the commercialization of laparoscopic video in the 1980s, laparoscopic staplers / cutters have become the laparoscopic solution for a wide range of complex general surgery, colorectal surgery, gynecology, bariatric and metabolic surgery, and thoracic surgery. Until recently, mechanical surgical staplers were open-loop devices where feedback was entirely dependent on the surgeon's judgment and experience.

[0003] Staple line leakage during surgery is a challenge, especially during gastroplasty (sleeve gastrectomy). Properly shaped B-shaped staples are associated with reduced leakage rates. One factor that can improve B-shaped staple performance is optimal application of pressure to the tissue during staple insertion. Excessive pressure (i.e., the clamped tissue is too thick) prevents the staple from closing and forming a B-shape. Insufficient pressure (i.e., the clamped tissue is too thin) results in insufficient gripping force, leading to tissue staking. The pressure experienced by the tissue depends on the duration of application of pressure because tissue dehydrates under pressure. Therefore, quantifying tissue pressure during the clamping and insertion process can inform the decision-making process and improve staple performance. Another factor that can potentially improve staple performance is the proper use of the stapler during surgery. Proper use includes, but is not limited to, selecting the appropriate staple cartridge size and applying optimal local shear force to the tissue. Summary of the Invention [Problem to be solved by the invention]

[0004] Recent advances in surgery include adding power, basic and advanced sensing capabilities, and wireless data communication capabilities to staplers. These additional capabilities can be designed into and implemented in staplers, and by connecting them to a computing unit, they can be expanded from standalone devices to connected, intelligent systems. Thus, the system can understand closure and driving forces and provide dynamic force feedback, improving ease of use, instrument selection, and potentially stapling consistency. [Means for solving the problem]

[0005] Embodiments of the present application disclose one or more sensors embedded in the distal actuator of the anastomosis instrument. The sensors can be embedded in the anvil or staple cartridge. These sensors quantify tissue pressure during the clamping and driving process, and the data is used for further computational processing to provide operational guidance to the user and improve staple line morphology. The sensor data is also used to calculate the local shear stress of the tissue resulting from the applied force and provide guidance to the user in selecting the appropriate staple cartridge size.

[0006] Embodiments of the present application disclose a surgical system including a surgical stapler with force detection and control capabilities and interfaced with a data receiving and processing unit that can process feedback data from the surgical system in combination with real-time laparoscopic or open surgical video, thereby providing information to the operator by displaying enhanced real-time video on a screen with areas at risk marked. [Brief explanation of the drawings]

[0007] [Figure 1] 1 shows a driving force curve of a staple driving process during a medical surgery procedure. [Figure 2a]10 shows an example image from a video displaying staple lines. [Figure 2b] 10 shows another exemplary image from enhanced real-time video displaying marked staple lines. [Figure 3a] 1 illustrates an exemplary console connected to an anastomosis instrument, a laparoscope, and a monitor according to an embodiment of the present application. [Figure 3b] 3b illustrates exemplary components of the console shown in FIG. 3a according to an embodiment of the present application. [Figure 4] 1 shows an exemplary image from a real-time video displaying detected staple lines according to an embodiment of the present application. [Figure 5a] 10 shows an image identifying staple line segments from a surgical real-time video image according to an embodiment of the present application. [Figure 5b] 10 shows an image identifying staple line segments from a surgical real-time video image according to an embodiment of the present application. [Figure 5c] 10 shows an image identifying staple line segments from a surgical real-time video image according to an embodiment of the present application. [Figure 5d] 10 shows an image identifying staple line segments from a surgical real-time video image according to an embodiment of the present application. [Figure 6] 3c illustrates exemplary hardware components of the console shown in FIG. 3b according to an embodiment of the present application. [Figure 7] 1 is a flowchart illustrating exemplary steps involved in a method for staple line detection during a medical surgical procedure according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, those skilled in the art should understand that the embodiments described below are illustrative and not exhaustive. These are intended to explain the present invention and should not be construed as limiting the scope of the present invention. All other embodiments that can be obtained by those skilled in the art based on the embodiments described herein without requiring creative work are intended to be included in the scope of the present application.

[0009] As an example, FIG. 1 shows a driving force curve 100 of a staple driving process during a medical procedure. As can be seen, there is a clear drop in driving force 102 during the staple driving process. While useful, it is very difficult to make a clinical decision based on this information alone (i.e., a clear drop in driving force). However, by combining real-time video of the medical procedure, staple line imaging, and specific tissue interactions as input data, it is possible to identify the cause of the drop in driving force, which may be related to insufficient tissue clamping, missing staples, or tissue abnormalities. In these cases, the video and data processing unit can calculate abnormality parameters and blend various indicators (e.g., different colors or patterns) into the real-time video.

[0010] 2b shows an example of marking a portion of a staple line 202 in an enhanced real-time video according to one embodiment of the present application, so that if the surgeon determines that a reduction in staple driving force (and the resulting consequences) poses a high risk, the surgeon can be alerted to perform the necessary surgical maneuvers to make the repair. In contrast, FIG. 2a shows an image from a surgical real-time video that displays a staple line 200 but does not have any markings.

[0011] As an example application, embodiments of the system may be applicable to bariatric and colorectal surgery. The system can flag areas of abnormal tissue thickness (either too thick or too thin) along the staple line or flag areas containing inadequate tissue (high risk of leakage) during impaction. The surgeon can then perform further evaluation based on the alert (e.g., flagged staple line segments) to determine whether additional suturing or reinforcement is required.

[0012] 3 shows one embodiment of the system. The system includes a console 300, which is connected to an anastomosis instrument 302, a laparoscope 304, and a monitor 306. The connection between the console 300 and the other devices 302, 304, and 306 may be wired or wireless (e.g., Wi-Fi, LAN, or Bluetooth). The anastomosis instrument 302 can transmit its own driving force data to the console 300. The laparoscope 304 can transmit captured real-time video to the console 300. The console processes the driving force data and real-time video and outputs an alert to the monitor 306.

[0013] In the embodiment shown in FIG. 3b, the console 300 includes multiple modules: a target detection module 308, a novel staple line identification module 310, a tracking module 312, a drive curve and staple line segment association module 314, and a classification module 316.

[0014] In such an embodiment, the target detection module 308 can use a neural network (e.g., Region-based Convolutional Neural Networks (RCNN) or You Only Look Once (YOLO)) to locate the staple line in the image. Figure 4 provides an example image from a real-time video showing the staple line with striae 402 detected.

[0015] Multiple staple driving may occur during a single surgery. The new staple line identification module 310 compares the detected staple lines with previously detected staple lines to identify newly generated staple lines. If this is the first time the anastomosis instrument is driven, the new staple line identification module 310 considers all staple lines to be new staple lines.

[0016] Figure 5a is an image showing the first staple drive during surgery. Figure 5b shows the new staple line 502 created by the first staple drive of Figure 5a, as identified by new staple line identification module 310. Figure 5c is an image showing the second staple drive during surgery. New staple line identification module 310 compares the image of Figure 5c with the image of Figure 5a and identifies the newly created second staple line 504 (see Figure 5d).

[0017] Returning to Figure 3b, tracking module 312 utilizes a neural network to track each staple line segment (each segment corresponding to one stroke of the anastomosis instrument), i.e., tracking module 312 can identify the location of each staple line in each frame in which the staple line is visible.

[0018] The firing curve and staple line segment association module 314 associates the most recent staple firing curve with a new staple line segment after the tracking module 312 detects the new staple line segment. This association matches each firing curve (e.g., the firing curve in FIG. 1) with a staple line segment (e.g., the staple line segments in FIGS. 5b and 5d).

[0019] The classification module 316 uses the constructed convolutional neural network (CNN) to take the images of the matched staple drive curves and staple line segments as input and classify the input into one of the following categories: a) normal staple drive; b) abnormal staple drive due to insufficient tissue clamping; c) abnormal staple drive with possible staple loss; or d) abnormal staple drive due to tissue abnormality.

[0020] The target detection module 308, novel staple line identification module 310, tracking module 312, drive curve and staple line segment association module 314, and classification module 316 of FIG. 3b may be implemented in software, firmware, hardware, or any combination thereof.

[0021] 6 illustrates exemplary hardware components of the console shown in FIG. 3b according to one embodiment of the present application. The console 600 includes an input / output (I / O) interface 612, a processing unit 614, a storage unit 616, a memory module 618, and a user interface 620, all interconnected by a connection 622.

[0022] The I / O interface 612 is configured to communicate with a laparoscope and an anastomosis instrument connected to the console 600. The I / O interface 612 also communicates with a monitor and displays alerts on the monitor. In some embodiments, separate I / O interfaces may be used to communicate with various devices, such as a laparoscope, an anastomosis instrument, and a monitor. Communication may occur through any suitable wired or wireless channel. The processing unit 614 is configured to receive data from the laparoscope and anastomosis instrument and process the data to detect and mark staple lines. The processing unit 614 is also configured to generate and send signals via the I / O interface 612 to display alerts on the monitor.

[0023] The storage unit 616 and / or memory module 618 are configured to store one or more computer programs executable by the processing unit 614 to perform the functions of the device 600. For example, the various example modules of Figure 3b may be stored in the storage unit 616 and / or memory module 618. The storage unit 616 and memory module 618 are non-transitory computer-readable media that store instructions that, when executed, cause the one or more processing units 614 to perform the methods described in various embodiments of the present application.

[0024] The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor, tape, optical, portable, non-portable, or other types of computer-readable media or computer-readable storage devices. As disclosed, the computer-readable medium has computer instructions stored thereon. In some embodiments, the computer-readable medium may be a disk or flash drive having computer instructions stored thereon.

[0025] It will be understood that the console 600 of FIG. 6 may include additional components not shown in FIG. 6, and that some components shown in FIG. 6 may be optional in some embodiments.

[0026] Figure 7 shows, in a flow chart, exemplary steps performed by the console 300 of Figure 3. First, the console receives driving force data from an anastomotic instrument and real-time video captured by a laparoscope during a medical procedure (e.g., a surgical procedure) (step 701). Next, the console locates staple lines from a frame of real-time video (step 702). This can be done using a neural network. The located staple lines are compared with previously detected staple lines to identify newly generated staple lines (step 703). If this is the first time the anastomotic instrument is driven, all staple lines are considered new staple lines.

[0027] Next, the console uses the neural network to track each staple line segment (each staple line segment corresponds to one drive of the anastomotic instrument) (step 704). That is, the console identifies the location of each staple line in each frame of the real-time video in which the staple line is visible. As new staple line segments are identified, the console associates them with an updated staple drive curve generated based on drive force data received from the anastomotic instrument (step 705). Essentially, each drive curve of the anastomotic instrument is associated one-to-one with a staple line segment identified from the real-time video.

[0028] The console uses a pre-built convolutional neural network (CNN) to classify the matched implant curve and staple line segment images into the following categories (step 706): a) Normal staple driving b) Abnormal impact due to insufficient tissue clamping c) Abnormal driving that may result in staples being missing d) Abnormal implantation due to tissue abnormality. The classification results are immediately output to the user (step 707).

[0029] The console also records and classifies visual characteristics of tissue clamping, anvil position, and staple cartridge size. Visual characteristics include organ / tissue classification, shape, abnormality location, relative thickness, color, color change rate, and superficial vasculature. This data is used to train a machine learning model that predicts pressure changes based on tissue characteristics and anvil position, allowing it to recommend the optimal staple cartridge size.

[0030] The console also collects pressure sensor data during the clamping process and compares it with a preset threshold. The clamping process consists of two stages: a clamp closing stage and a resting stage, and there is a high correlation between average tissue thickness and maximum pressure. If the maximum pressure exceeds the threshold, the tissue is deemed too thick for the staple cartridge; if it is lower, it is deemed too thin. In either case, the system recommends replacing the staple cartridge.

[0031] Based on the collection and analysis of shear force on the tissue, the system recommends repositioning the anastomotic device according to changes in pressure distribution on the cartridge. One of the causes of poor B-shaped staple formation is human error, where the surgeon may apply unnecessary strain or tension to the tissue. If pressure distribution is uneven, the intelligent system recommends repositioning the anastomotic device, such as by rotating or translating it, thereby minimizing fluctuations in pressure distribution and improving staple formation accuracy.

[0032] The surgical stapler based on this embodiment can be commercialized as a standalone product or as a combined product with an artificial intelligence processing unit and an imaging system.

[0033] Although the present embodiment has been described in detail with reference to the accompanying drawings, various improvements and modifications may occur to those skilled in the art, and it is understood that such improvements and modifications fall within the scope of the appended claims.

Claims

1. 1. A staple line detection system during a medical procedure, comprising: a target detection module configured to detect staple lines in the image; a new staple line identification module configured to compare the detected staple lines with previously detected staple lines and identify newly generated staple lines; a tracking module configured to track each staple line segment, each staple line segment corresponding to one impact of an anastomotic instrument; a drive curve and staple line segment association module configured to associate a current staple drive curve with the new staple line segment after the tracking module detects the new staple line segment; a classification module configured to receive the associated firing curve and the new staple line segment as input and classify the input into one of a plurality of categories; 1. A staple line detection system during a medical procedure comprising:

2. The system of claim 1 , wherein the target detection module is configured to locate staple lines in the image using a neural network.

3. 3. The system of claim 2, wherein the neural network comprises a region-based convolutional neural network (RCNN) or a You Only Look Once (YOLO) detection.

4. 2. The system of claim 1, wherein the new staple line identification module is configured to consider the detected staple line as a new staple line if the detected staple line results from an initial impact of an anastomotic instrument.

5. The system of claim 1 , wherein the tracking module is configured to track each staple line segment using a neural network.

6. 2. The system of claim 1, wherein the classification module is configured to use a constructed convolutional neural network (CNN) to receive inputs of associated firing curves and novel staple line segments and classify the inputs into one of a plurality of categories.

7. The system of claim 1, wherein the plurality of categories include (a) normal staple driving, (b) abnormal staple driving due to insufficient tissue clamping, (c) abnormal staple driving due to possible staple loss, and (d) abnormal staple driving due to tissue abnormality.

8. The system according to claim 1 , wherein the system is connected to an anastomosis instrument and configured to receive driving force data of the anastomosis instrument.

9. The system of claim 1 , wherein the system is connected to a laparoscope and configured to receive real-time video captured by the laparoscope during surgery.

10. The system of claim 1 , wherein the system is coupled to a monitor and configured to output an alert to the monitor, the alert being generated based on a category into which the input is categorized.

11. 1. A computer-implemented method for identifying staple lines, comprising: detecting staple lines in the image; comparing the detected staple lines with previously detected staple lines to identify newly generated staple lines; tracking each staple line segment, each of said staple line segments corresponding to one impact of an anastomotic instrument; after detecting a new staple line segment in the tracking step, associating a current staple driving curve with said new staple line segment; taking as input the associated firing curve and new staple line segment and classifying said input into one of a plurality of categories; 11. A computer-implemented method for identifying staple lines, comprising:

12. The method of claim 11 , further comprising using a neural network to locate staple lines in the image.

13. The method of claim 12 , wherein the neural network comprises a region-based convolutional neural network or one-storage detection.

14. 12. The method of claim 11, wherein comparing the detected staple line with previously detected staple lines to identify a new staple line comprises considering the detected staple line to be a new staple line if the staple line results from an initial staple drive of an anastomosis instrument.

15. The method of claim 11 , wherein tracking each staple line segment comprises tracking each staple line segment using a neural network.

16. 12. The method of claim 11, wherein classifying the input into one of a plurality of categories comprises using a constructed convolutional neural network to classify the input into one of a plurality of categories, the constructed convolutional neural network taking the associated firing curve and new staple line segment as input.

17. The method of claim 11, wherein the plurality of categories include (a) normal staple driving, (b) abnormal staple driving due to insufficient tissue clamping, (c) abnormal staple driving due to possible staple loss, and (d) abnormal staple driving due to tissue abnormality.

18. The method of claim 11 , further comprising connecting to an anastomosis instrument to receive driving force data of the anastomosis instrument.

19. 12. The method of claim 11, further comprising connecting to a laparoscope to receive real-time video captured by the laparoscope.

20. The method of claim 11 , further comprising connecting to a monitor and outputting an alert to said monitor, said alert being generated based on a category into which the input has been classified.

Images