Video-based analysis of suturing events during surgical procedures using machine learning

Abstract

The analysis system trains the machine learning model to detect suturing events from the video of the surgical procedure. The machine learning model detects the time at which the stapling events occur as well as one or more characteristics of each stapling event, such as the length of staples, nip time, or other characteristics. A machine learning model is trained on a video of a surgical procedure, identifying when a suturing event has occurred by a learning process. A machine learning model may be applied to an input video to detect a sequence of stapler events. The stapler event sequences may also be analyzed and / or aggregated to generate various analysis data related to surgical procedures for application such as inventory management, performance evaluation, or prediction of patient outcomes.

Description

Background Art Technical Field

[0001] The embodiments are directed to machine learning techniques for detecting and analyzing surgical suturing events from videos of surgical procedures.

[0002] Description of Related Technology

[0003] Many surgical procedures (e.g., sleeve gastrectomy, Roux-en-Y gastric bypass, etc.) involve surgical suturing. Surgeons have a variety of options when performing surgical suturing. For example, surgical staples can be obtained in different lengths, from different manufacturers, and can be obtained with or without a support. For surgical procedures involving multiple surgical staples, the surgeon must decide which type of staples to use for different purposes and the specific order in which the staples are deployed during surgery. Different options for the use of surgical staplers can have a significant impact on inventory costs, surgical operation time, and patient outcomes. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is an example embodiment of a system for detecting and characterizing suturing events from video of a surgical procedure.

[0005] Figure 2 An example implementation of a machine learning system for detecting one or more suturing events and characteristics of the suturing events from a video of a surgical procedure.

[0006] Figure 3 is a flow chart of one embodiment of a method for detecting and characterizing suturing events from a video of a surgical procedure.

[0007] Figure 4 is an example visualization of suturing events detected in a video of a surgical procedure. DETAILED DESCRIPTION

[0008] The accompanying drawings (figures) and the following description describe certain embodiments by way of example only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. Reference will now be made to several embodiments, examples of which are shown in the accompanying drawings. Whenever feasible, similar or similar reference numerals may be used in the drawings and may indicate similar or similar functions.

[0009] The analysis system trains a machine learning model to detect suturing events from a video of a surgical procedure. The machine learning model detects the time at which a suturing event occurs and one or more characteristics of each suturing event, such as the length of the staple, the clamping time, or other characteristics. The machine learning model is trained on a video of a surgical procedure and identifies when a suturing event occurs through a learning process. The machine learning model can be applied to an input video to detect a sequence of suturing events. The suturing event sequence can also be analyzed and / or aggregated to generate various analytical data related to a surgical procedure for applications such as inventory management, performance evaluation, or predicting patient outcomes.

[0010] Figure 1 An example embodiment of a system 100 for detecting and characterizing suturing events from one or more surgical videos 105 is illustrated. Each suturing event involves delivering one or more nail loads to the patient's tissue during a medical operation using a stapler device. The stapler load is usually arranged in a stapler magazine of multiple nails of the same type, and the multiple nails are delivered simultaneously by the stapler when actuated. The type of stapler load can be characterized by its manufacturer, length, color, presence or absence of a support, the type of support, or other distinguishing features. The color of the stapler load is usually related to the length of the stapler by a predefined color coding. Specific color coding can vary for different stapler manufacturers. The suturing event can also include various front actuation or rear actuation activities related to the suturing event. For example, a surgeon can usually perform the clamping of tissue before actuating the stapler. Various medical operations can involve different suturing event sequences that can occur in various timings, and can involve individual suturing events utilizing different types of stapler loads, different counts of stapler loads, and different event durations.

[0011] The video 105 of the surgical procedure may be captured by one or more cameras positioned to have a field of view of the anatomical structure where the suturing event is performed. For example, the video 105 may be obtained from an overhead camera in the operating room, a head-mounted camera worn by the surgeon, an endoscopic camera, or a combination thereof. In one embodiment, the video 105 may include (or be associated with) various metadata related to the surgical procedure depicted in the video 105. For example, the metadata may identify the type of surgery being performed; information about the patient undergoing the surgery, the surgeon (or other medical practitioner); the facility where the surgical procedure was performed; and outcome information associated with the surgical procedure, such as patient recovery time, the onset of complications, or other metrics characterizing the outcome.

[0012] The analysis system 110 includes a suturing event detection module 112 and a surgical analysis module 126. The suturing event detection module 112 can automatically detect suturing events 114 in the received video 105. Each detected suturing event 114 can be characterized according to timing data indicating when the suturing event 114 occurred and one or more characteristics of each suturing event 114. The timing data may include a single timestamp (e.g., defining a video frame) or a time interval (e.g., defining a video segment). The timing data may be specified relative to the beginning of the video or relative to the frame number, or may be specified relative to the beginning of the detected surgical operation. The characteristics of each suturing event 114 that can be detected by the suturing event detection module 112 may include, for example, the identification of the stapler manufacturer, the color of the stapler load and / or the corresponding length of the nail (based on the relevant color coding of the identified manufacturer), the presence or absence of a support in the nail load, the type of support, the number of times the stapler is fired during the suturing event, the count of the deployed staplers, the duration of the suturing event, the clamping time and / or other characteristics of the clamped tissue associated with the suturing event 114 during the clamping event.

[0013] The suturing events 114 can be organized into a suturing event sequence associated with the surgical procedure depicted in the video 105. Here, the suturing event sequence can involve multiple suturing events, each of which can utilize the same or different types of staple loads and / or can have other different characteristics. In one embodiment, the suturing event detection module 112 can detect the suturing events 114 in substantially real time based on the live input video 105.

[0014] The surgical analysis module 116 generates various analytical data 118 associated with the detected suturing events 114, which can be used for applications such as inventory management, performance evaluation and / or patient outcome prediction. The analytical data 118 can be based on a single sequence of suturing events 114 associated with a single input video 105 for a single medical operation, or can be based on an aggregated sequence of suturing events 114 across multiple videos 105 of the same or different types of medical operations. For example, the surgical analysis module 116 can aggregate sequences of suturing events 114 to detect aggregate information related to inventory management, such as counts of different types of suturing loads used, usage rates, etc. In addition, the surgical analysis module 116 can aggregate suturing events 114 based on various filtering parameters (such as surgical type, involved practitioners, surgical location, etc.). For example, the surgical analysis module 116 can determine the average stapler length for a specific medical operation, the average number of suturing events per medical operation, or other aggregated data that can indicate the most common operations. In another example, the surgical analysis module 116 can generate aggregated analytical data 118 that characterizes sequences of suturing events 114 in multiple different types of surgical operations. In another example, the surgical analysis module 116 may generate analysis data 118 that indicates how commonly one staple load color (or length) follows another stapler load color (or length) across a series of different surgeries.

[0015] The surgical analysis module 116 can also generate various analytical data indicating the comparison between different suture event sequences 114. For example, the surgical analysis module 116 can generate an output that compares the detected sequence of the suture event 114 for a specific observed medical operation with the baseline sequence of the suture event associated with the medical operation and derives a similarity measure. The baseline sequence can be derived from the sequence of similar operations observed, or can be input from an expert knowledge source. The similarity measure can indicate the proximity of the sequence observed in the type of the stapler used, the timing of the suture event, the clamping time or other characteristics to the baseline. This information can also be used to automatically detect anomalies that are significantly deviated from conventional operations in specific observed operations. This type of information can be used to characterize the performance of practitioners and / or predict patient outcomes.

[0016] In further embodiments, the surgical analysis module 116 may generate various outcome predictions based on the observed sequence of suturing events 114. For example, the analysis system 110 may aggregate the observed suturing events 114 and correlate the sequence with observed patient outcomes. These correlations may be used to predict outcomes associated with a particular observed sequence of suturing events 114. For example, the surgical analysis module 116 may predict expected patient recovery time and / or the likelihood of complications based at least in part on the detected sequence of suturing events 114.

[0017] The surgical analysis module 116 can generate various visualizations or other outputs (e.g., audio outputs) related to the detected suturing events 114 and / or the analysis data 118. For example, for a particular video 105, the surgical analysis module 116 can generate a timeline indicating the relative timing of each suturing event 114 during the medical procedure and one or more characteristics of each suturing event (such as load color). The surgical analysis module 116 can also automatically generate an alert indicating a detected deviation from a standard operation and / or a detected outcome prediction.

[0018] In one embodiment, the surgical analysis module 116 can detect at least some of the analysis data 118 associated with the input video 105 in substantially real time so that the analysis data 118 can be output during the medical operation. In this case, the surgical analysis module 116 can generate specific recommendations to guide the practitioner through the surgical operation. For example, for a known medical operation including a sequence of surgical steps, the surgical analysis module 116 can detect which steps have been performed based on the detected suturing events 114 and generate an output indicating the recommended next step. In addition, the surgical analysis module 116 can generate specific recommendations about the type (e.g., color) of the stapler load to be applied during the medical operation based on the sequence of observed suturing events 114 and other metadata associated with the medical operation.

[0019] The output device 120 may receive stitching events 114, analysis data 118, input video 105, and / or other metadata from the analysis system 110, and generate one or more outputs from the presentation. The output device 120 may include a computing system (e.g., a mobile device, a tablet computer, a laptop computer, or a desktop computer) that may present a user interface for presenting stitching events 114 and / or analysis data 118. The sequence of stitching events 114 may be represented in a graphical interface that indicates the corresponding timing data associated with each stitching event 114 and the corresponding characteristics of the event 114. In some embodiments, the timing data may be presented together with the video 105 (e.g., as a visual overlay) and / or associated metadata (e.g., identifying surgery, patients, surgeons, facilities, etc.). The user interface may also provide access to the above-mentioned various aggregated analysis data 118 in the form of a chart, graph, table, or other data presentation structure. In another embodiment, the output device 120 may include an audio output device that generates audio data indicating stitching events 114 and / or analysis data 118. For example, in real-time analysis, the output device 120 may output recommended steps for a surgical procedure as an audio output.

[0020] The output device 120 may also include an indexed database of suturing events 114 and / or analytical data 118 that may be accessed based on various search queries or filters. For example, such a database may be used to view historical suturing sequences associated with a particular type of surgical procedure, suturing sequences used by a particular physician, relative counts of different types of suturing used by each medical facility, or other information.

[0021] The above-mentioned analysis system 110 can be used for various purposes associated with a healthcare provider. For example, in one application, detected suturing events can be processed by an inventory management system to track the inventory of nail loads and facilitate reordering. Here, when the tracked inventory drops below a threshold level, the inventory management system can automatically reorder the nails and / or can warn the administrator. In another example application, the analysis system 110 can be used to evaluate the performance of the surgeon based on factors (such as meeting the baseline standards associated with the suturing sequence, the number of nail loads used for each medical operation, the duration of the suturing sequence, or other performance metrics). In another example application, the analysis system 110 can learn the best operation by associating different stapler sequences with corresponding results to identify potential cause and effect relationships.

[0022] Figure 2 An example implementation of a machine learning system 200 that can be used to implement various aspects of the above-described analysis system 110 is illustrated. The machine learning system 200 includes a training video memory 205, a training stitching event sequence memory 255, training modules 215-A and 215-B for stitching event detection and analysis prediction, respectively (collectively referred to herein as training modules 215), machine learning model memories 220-A and 220-B for stitching event detection and analysis prediction, respectively (collectively referred to herein as machine learning model memories 220), and inference modules 225-A and 225-B for stitching event detection and analysis prediction, respectively (collectively referred to herein as inference modules 225). In other embodiments, the machine learning system 200 may include a training video memory 205, a training stitching event sequence memory 255, training modules 215-A and 215-B for stitching event detection and analysis prediction, respectively (collectively referred to herein as training modules 215), machine learning model memories 220-A and 220-B for stitching event detection and analysis prediction, respectively (collectively referred to herein as machine learning model memories 220), and inference modules 225-A and 225-B for stitching event detection and analysis prediction, respectively (collectively referred to herein as inference modules 225). Figure 2 Parts that are different or additional to those described.

[0023] The training video memory 205 stores training videos for training one or more machine learning models 220-A associated with suturing event detection. The training videos 205 each depict medical surgery and are associated with labels that characterize suturing events in surgery. Different labels can be associated based on each frame, each fragment and / or the whole video. For example, the label can indicate the timing of the suturing event (e.g., whether the suturing event occurs based on each frame or each fragment) and one or more characteristics of each suturing event. The suturing event may include a single deployment of a nail load or may include a series of deployments. The marked characteristics of the suturing event may include, for example, the color and / or related length of the nail load used in the suturing event, the presence or absence of a support in the nail load applied during the suturing event, the type of support, the manufacturer of the stapler used during the suturing event, the count of the deployed stapler, the duration of the suturing event, the length of time (i.e., the clamping time) that the stapler clamps the patient's tissue together, the firing rate of the stapler, and other characteristics describing the use of the staple during the suturing event. In various embodiments, the labels of the training video 205 are obtained from the expert reviewer responsible for annotating the video.

[0024] The training data may also include various metadata associated with the surgical procedures depicted in the video. For example, the training data may include information about the patient undergoing the surgery, the surgeon (or other medical practitioner), the facility where the surgery was performed, and outcome information associated with the surgery, such as patient recovery time, the occurrence of complications, or other metrics characterizing the outcome.

[0025] The training module 215-A trains one or more machine learning models 220-A for stitching event detection. In various embodiments, the training module 215-A can generate the machine learning model 220-A based on a learning technique such as regression, support vector machine, naive Bayes, decision tree, k-nearest neighbor, random forest, boosting algorithm, k-means, hierarchical clustering, neural network, multi-layer perceptron, convolutional neural network, recurrent neural network, sequence to sequence model, generative adversarial network, transformer, or a combination thereof.

[0026] In one embodiment, the training module 215-A may apply various pre-processing, such as filtering, normalization, segmentation, or other transformations, to the training videos 205. The training module 215-A may also include a feature extractor that extracts various features from the training videos 205 and generates a corresponding feature vector corresponding to each training video 205 input into the training algorithm. The features may represent various visual characteristics of the video and / or various potential features derived from the base video 205. In other embodiments, the training module 215-A may operate directly on the training videos 205 without having to calculate the underlying features.

[0027] The machine learning model 220-A generated by the training module 215-A may include a set of model parameters representing weights or biases for applying to a set of input variables according to one or more predefined functions. The learning process typically learns a set of model parameters that optimize an optimization criterion from the training video 205. In supervised learning, the training module 215-A may iteratively apply the model 220 to the training video 205 to generate one or more outputs, compare the one or more outputs to one or more labels to derive an error function, and apply an update algorithm (e.g., gradient descent or other techniques) to update the model parameters according to the optimization criterion (typically operating to reduce the error function). Over many iterations of the learning process using various training videos 105, the updates typically reduce the error between the inference and the label, thereby improving the predictive power of the model 220.

[0028] The inference module 225-A applies one or more machine learning models 220-A to the (unlabeled) input video 105 to infer the timing and characteristics of the stitching event 114. Here, the inference module 225-A may apply similar preprocessing and / or feature extraction (if any) as used by the training module 215-A. In one embodiment, the stitching event 114 may be represented as a vector of likelihood scores, which indicates the corresponding inference likelihood of various features present based on each frame or each fragment. The likelihood associated with the occurrence of the stitching event can be compared with a threshold to classify each frame or fragment in a binary manner, and identify the frames or fragments that are being detected as being associated with the stitching event. For each occurrence, the inference associated with the highest likelihood of each feature can also be identified.

[0029] In some embodiments, the training module 215-A trains a single machine learning model 220, such as a multi-level convolutional neural network, which jointly learns the occurrence of suturing events and various characteristics of suturing events. Then, the timing information and characteristics of each suturing event are jointly inferred by the inference module 225-A. In other embodiments, the training module 215-A can separately train different machine learning models 220-A associated with different attributes of the suturing event. For example, the first machine learning model 220-A can be trained to detect the occurrence of the suturing event 114, the second machine learning model 220 can be trained to detect the color of the nail load, and the third machine learning model can be trained to detect the manufacturer based on the suturing device used. Then, the inference module 225-A similarly applies a set of models 220-A to generate corresponding inferences of the suturing event 114. In this embodiment, various machine learning models 220-A can be trained (and applied) to different machine learning techniques accordingly, and these different machine learning techniques generate different types of machine learning models 220-A suitable for each corresponding inference task.

[0030] Although the training module 215-A and the inference module 225-A Figure 2 Although logically illustrated as separate components in the examples, the algorithms employed by each module 215-A, 225-A may include overlapping functionality (e.g., the same inference algorithm applied by the inference module 225-A may be employed by the training module 215-A during training). Therefore, in actual implementations, the training module 215-A and the inference module 225-A may be implemented at least in part using a shared set of executable instructions.

[0031] The training sequence memory 255 includes a database or other index storing a sequence of suture events associated with a medical procedure. The training sequence 255 may include the suture events 114 inferred by the above-mentioned inference module 225-A, or may include a sequence obtained from an external source (e.g., manually entered data associated with a medical procedure performed in history). In addition to storing timing and other characteristics associated with the sequence of suture events, the training sequence may also include various metadata associated with the basic surgery. Such metadata may include, for example, the type of surgical procedure, the surgeon performing the surgical procedure, the patient performing the surgical procedure (or its related demographic information), the date of the surgical procedure, the patient's recovery time after the surgical procedure, the complications experienced by the patient after the operation, or other descriptive information related to the surgical procedure.

[0032] The training module 215-B trains one or more machine learning models 220-B for deriving various analysis data 118 associated with the training sequence 255. In one embodiment, the training module 215-B may employ a supervised learning method that utilizes metadata as labels to generate a training result model that infers the relationship between the training sequence 255 and the associated postoperative outcome. In another embodiment, the training module 215-B may employ an unsupervised or semi-supervised learning method to learn the relationship between different training sequences 255. For example, the training module 215-B may train a clustering algorithm to cluster similar sequences and enable the detection of outlier sequences that do not substantially conform to the training sequence 255. In addition, the training module 215-B may learn the relationship between individual suturing events within the sequence to achieve inferences, such as predicting the characteristics of future suturing events based on a partial sequence of observed suturing events. The training module 215-B may train a single machine learning model 220-B to collectively generate different types of inferences related to the input suturing sequence 114, or may train multiple different machine learning models 220-B (which may be different types of machine learning models) to generate different types of inferences. The inference module 225-B applies one or more machine learning models 220-B to the (unlabeled) set of input stitching events 114, to the various analysis data 118, as described above. The training module 215-B and the inference module 225-B may employ any of the same learning and inference techniques described above with respect to the training module 215-A and the inference module 225-A.

[0033] The training module 215, the model storage 220 and the inference module 225 are Figure 2 2. The exemplary embodiments of the present invention are illustrated as logical, but can be implemented using at least some shared elements in the implementation. For example, the model storage 220 can include a shared database, and the various functions employed by the training modules 215 and 225 can be performed using a set of shared computer instructions.

[0034] In some embodiments, the training module 215 and the inference module 225 may be combined into respective joining modules 215, 225 that enable one or more predictive analysis outputs 118 to be directly inferred from the input video 105 without explicitly detecting the sequence of suturing events 114. For example, the machine learning model 220 may be trained to directly infer postoperative outcomes (e.g., recovery time, complications, etc.) based on the input video 105 without explicitly inferring the sequence of suturing events 114 (although information related to the suturing sequence may include potential features related to the model 220). In another example, the machine learning model 220 may directly output a similarity measure between the suturing sequence of the input video 105 and the baseline sequence without explicitly generating and outputting the suturing events 114 themselves.

[0035] Figure 2 The machine learning method described in is intended to illustrate one example technique for generating analysis data 118, but other types of analysis data 118 may be generated without necessarily employing a machine learning method. For example, various aggregations of stitched event data may be generated by applying one or more aggregation functions that do not rely on any machine learning model 220-B.

[0036] Figure 3 is a flow chart of one embodiment of a method for detecting the occurrence and characteristics of suturing events from an input video of a surgical procedure. In various embodiments, the method includes combining Figure 3 In addition, in some embodiments, the steps of the method can be combined with Figure 3 The order in which they are described is different from the order in which they are executed.

[0037] The analysis system 110 obtains 305 a video of a surgical procedure and obtains 310 a machine learning model that is trained to detect suturing events from the video and determine one or more characteristics of the suturing events. The analysis system 110 applies 315 the machine learning model to the video to detect one or more suturing events from the video and characteristics associated with each detected suturing event. Figure 1 and Figure 2 As further described, the machine learning model can detect the timing of the occurrence of suturing events and one or more characteristics of each suturing event (such as the length and / or color of the staples used during the suturing event, the amount of time the patient's tissue is clamped together during the suturing event, and the firing rate of the stapler during the suturing event). The analysis system 110 outputs 320 the identified suturing events as a suturing event sequence indicating the detection timing and characteristics. The analysis system 110 can also generate 325 various analytical data associated with the detected suturing event sequence (which can be based on another machine learning model), such as predicted patient outcomes, similarities and / or differences relative to a baseline sequence associated with a medical procedure, visualization of various aggregated data, performance evaluations related to medical surgery, recommendations associated with medical surgery, etc.

[0038] Figure 4It is an example visualization of a sequence 400 of suturing events output from the analysis system 110 relative to a video 105 of a surgical operation. In the illustrated example, the detected sequence 400 (which has a start time 405 and an end time 410) is depicted on a timeline associated with the video 105. Suturing events 415, 420, 425, 430 are detected at various times in the video 105, and their relative timing can be shown on the timeline. In addition, the relative length of each suturing event 415, 420, 425, 430 can be indicated. In one embodiment, each event 415, 420, 425, 430 can be color-coded and / or marked to indicate the type of nail load used during the corresponding suturing event. In an embodiment, the sequence 400 can be displayed together with the video 105 derived therefrom.

[0039] Embodiments of the system 100 and corresponding methods may be implemented by one or more computing systems. The one or more computing systems include at least one processor and a non-transitory computer-readable storage medium that stores instructions that can be executed by the at least one processor to perform the processes and functions described herein. The computing systems may include computing systems based on distributed networks in which the functions described herein are not necessarily performed on a single physical device. For example, some specific implementations may utilize cloud processing and storage technology, virtual machines, or other technologies.

[0040] The foregoing description of the embodiments has been given for the purpose of illustration; the foregoing description is not intended to be exhaustive or to limit the embodiments to the particular forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0041] Some portions of this description describe embodiments in terms of algorithms and symbolic representations of information operations. These operations, when described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent circuits, microcode, etc. In addition, it has proven convenient at times to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0042] Any of the steps, operations or processes described herein can be performed or implemented with one or more hardware or software modules, either alone or in combination with other devices. Embodiments may also relate to devices for performing the operations herein. The device may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device, which is started or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a tangible, non-transient computer-readable storage medium or any type of medium suitable for storing electronic instructions, and is coupled to a computer system bus. In addition, any computing system mentioned in this specification may include a single processor or may be an architecture that uses a multi-processor design to improve computing power.

[0043] Finally, the language used in this specification is selected primarily for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. Therefore, the scope of the present invention is not limited by this detailed description, but by any claims issued in an application based thereon. Therefore, the disclosure of the embodiments is intended to illustrate but not to limit the scope of the present invention, which is set forth in the appended claims.

Claims

1. A method for automatically characterizing suturing events in a surgical procedure based on an input video, the method comprising: receiving said input video of said surgical procedure; obtaining a first machine learning model trained by a supervised learning process using annotated training videos, the annotated training videos being labeled to indicate a timing of occurrences of suturing events and one or more characteristics of each of the suturing events; applying the first machine learning model to the input video to generate inferences that specify the detected sequence of suturing events based on relative timing of the suturing events during the surgical procedure and one or more characteristics associated with each of the detected sequences of suturing events; as well as The inference of the detected sequence of stitching events is output.

2. The method according to claim 1, wherein: The one or more characteristics include the length of the staple load used during the corresponding stapling event.

3. The method according to claim 1, wherein: The one or more characteristics include the color of the staple load and the manufacturer of the stapler that delivered the staple load during the corresponding stapling event.

4. The method according to claim 1, wherein: The one or more characteristics include a clamp time associated with a corresponding stapling event, the clamp time indicating an amount of time the patient's tissue is clamped together prior to delivery of the stapler load.

5. The method according to claim 1, wherein: The one or more characteristics include a detected presence or absence of a buttress in a stapler load delivered during a corresponding stapling event.

6. The method according to claim 1, further comprising: obtaining a second machine learning model trained to infer one or more outcome attributes of the surgical procedure based on a corresponding sequence of stapler events occurring during the surgical procedure; as well as Applying the second machine learning model to the detected sequence of suturing events inferred by the first machine learning model to predict one or more outcome attributes of the surgical procedure depicted in the input video.

7. The method according to claim 6, wherein: The one or more outcome attributes include a predicted recovery time for the patient.

8. The method according to claim 6, wherein: The one or more outcome attributes include one or more likelihoods of one or more post-operative complications.

9. The method according to claim 1, further comprising: comparing the detected sequence of surgical stapling events to a standard sequence of stapling events associated with the surgical procedure; as well as A similarity metric is generated that indicates the similarity of the detected sequence to the standard sequence.

10. The method according to claim 1, further comprising: aggregating the detected sequence of suturing events with a set of previously detected sequences of suturing events involving a different surgeon performing the surgical procedure; as well as One or more statistics are output based on an aggregation of the detected sequence of stitching events from the input video and the set of previously detected sequences of stitching events.

11. A non-transitory computer-readable storage medium storing instructions for automatically characterizing suturing events in a surgical procedure based on an input video, the instructions, when executed by a processor, causing the processor to perform steps comprising: receiving said input video of said surgical procedure; obtaining a first machine learning model trained by a supervised learning process using annotated training videos, the annotated training videos being labeled to indicate a timing of occurrences of suturing events and one or more characteristics of each of the suturing events; applying the first machine learning model to the input video to generate inferences that specify the detected sequence of suturing events based on relative timing of the suturing events during the surgical procedure and one or more characteristics associated with each of the detected sequences of suturing events; as well as The inference of the detected sequence of stitching events is output.

12. The non-transitory computer-readable storage medium of claim 11, wherein: The one or more characteristics include the length of the staple load used during the corresponding stapling event.

13. The non-transitory computer-readable storage medium of claim 11, wherein: The one or more characteristics include the color of the staple load and the manufacturer of the stapler that delivered the staple load during the corresponding stapling event.

14. The non-transitory computer-readable storage medium of claim 11, wherein: The one or more characteristics include a clamp time associated with a corresponding stapling event, the clamp time indicating an amount of time the patient's tissue is clamped together prior to delivery of the stapler load.

15. The non-transitory computer-readable storage medium of claim 11, wherein: The one or more characteristics include a detected presence or absence of a buttress in a stapler load delivered during a corresponding stapling event.

16. The non-transitory computer-readable storage medium of claim 11, further having encoded thereon instructions that, when executed by the processor, cause the processor to perform steps comprising: obtaining a second machine learning model trained to infer one or more outcome attributes of the surgical procedure based on a corresponding sequence of stapler events occurring during the surgical procedure; and Applying the second machine learning model to the detected sequence of suturing events inferred by the first machine learning model to predict one or more outcome attributes of the surgical procedure depicted in the input video.

17. The non-transitory computer readable storage medium of claim 16, wherein: The one or more outcome attributes include a predicted recovery time for the patient.

18. The non-transitory computer readable storage medium of claim 16, wherein: The one or more outcome attributes include a recovery time for the patient.

19. The non-transitory computer readable storage medium of claim 16, wherein: The one or more outcome attributes include one or more likelihoods of one or more post-operative complications.

20. A method for training a machine learning model to characterize suturing events in a surgical procedure based on a surgical video, the method comprising: obtaining a set of training videos depicting respective surgical procedures and respective labels identifying occurrences of suturing events and corresponding characteristics of the suturing events in the set of training videos; applying a supervised machine learning algorithm to learn model parameters of the machine learning model such that the machine learning model, when applied to an input video, generates inferences that specify the detected sequence of suturing events based on the relative timing of the suturing events during the surgical procedure and one or more characteristics associated with each of the detected sequences of suturing events; as well as The machine learning model is stored in a computer-readable storage medium.

Images