Aligned workflow compression and multi-dimensional workflow alignment

Abstract

An aspect includes a computer-implemented method that quantifies a change in a surgical method of a medical procedure. Surgical videos recording a plurality of cases of a medical procedure are analyzed to identify a change in a surgical method used by a service provider in performing the medical procedure. According to some aspects, variations in surgical methods are quantified.

Description

Background Art

[0001] The present invention relates generally to computing techniques, and more particularly to computing techniques for aligned workflow compression and multi-dimensional workflow alignment.

[0002] Computer-assisted systems, particularly computer-assisted surgery systems (CAS), rely on video data captured digitally during surgery. Such video data can be stored and / or streamed. In some cases, video data can be used to enhance a person's physical senses, perceptions, and responsiveness. For example, such a system can effectively provide information corresponding to a visual field extended in time and space, which enables a person to adjust current and future actions based on a portion of the environment not included in his or her physical visual field. Alternatively or in addition, video data can be stored and / or transmitted for several purposes, such as archiving, training, postoperative analysis, and / or patient consultation. The process of analyzing and comparing a large amount of video data from multiple surgical operations to identify commonalities may be highly subjective and error-prone (e.g., due to the amount of data and many factors that affect the workflow of each individual surgical operation being analyzed (e.g., patient condition, physician preference, etc.)). Summary of the invention

[0003] According to one aspect, a system includes a machine learning processing system, the machine learning processing system including one or more machine learning models, the one or more machine learning models are trained to identify multiple stages in a video of a surgical procedure. The system also includes a data analysis system, the data analysis system is configured to quantify the changes in surgical methods in multiple videos capturing the same type of surgical procedure. The quantification of the changes in surgical methods includes receiving multiple surgical videos, each of which captures the workflow of the same type of surgical procedure. Each of the multiple surgical videos is segmented into a segmented workflow including multiple stages, the segmentation being performed based on the stages identified by the machine learning processing system. The stages in the segmented workflows are aligned with the average workflow of the segmented workflows to create multiple aligned workflows. The number of stages in each of the multiple aligned workflows is the same as the number of stages in the average workflow. The entropy of each stage in the aligned workflows is calculated based on the Chao-Shen estimator. The entropy of each stage in the aligned workflow is output to a display device for display, wherein the entropy represents the changes in the surgical method of the surgical procedure.

[0004] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that calculating the entropy includes scaling the entropy by its maximum possible value to a value that falls within a specified range of values.

[0005] In addition to or as an alternative to one or more of the features described above or below, further aspects may include: quantifying the changes in surgical methods further includes calculating the average workflow based on the workflows in the plurality of surgical videos.

[0006] In addition to or as an alternative to one or more of the features described above or below, further aspects may include: calculating the average workflow includes analyzing all center-point workflows in the surgical videos, and selecting the average workflow from the center-point workflows based on the analysis.

[0007] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the analyzing includes performing adaptive dynamic time normalization-center of gravity averaging (ADBA) using each of the center point workflows as an initial average workflow.

[0008] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the analysis is optimized for categorical data and is performed based at least in part on mode values ​​at various stages on the segmentation workflows.

[0009] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the optimizing includes modifying the length of the average workflow by adding a stage to the average workflow or removing one of the stages from the average workflow.

[0010] In addition to or as an alternative to one or more of the features described above or below, further aspects may include outputting a surgical service provider's indication associated with one or more of the aligned workflows to the display device for display.

[0011] In addition to or as an alternative to one or more of the features described above or below, further aspects may include: each stage identified by the one or more machine learning models includes a value of multidimensional data, wherein the multidimensional data includes surgical tools and surgical stages.

[0012] According to another aspect, a computer-implemented method includes receiving, by a processor, a plurality of surgical videos, each of the plurality of surgical videos capturing a workflow of a same type of surgical procedure, and each of the plurality of surgical videos being segmented into a segmented workflow based on a plurality of data dimensions, the segmented workflow comprising stages. Analyzing the plurality of segmented workflows to calculate a multidimensional average workflow. Aligning the plurality of segmented workflows in the segmented workflows with the multidimensional average workflow. Based on the alignment, quantifying variations in surgical methods in the plurality of segmented videos.

[0013] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the analyzing includes modifying the length of the multidimensional averaging workflow by adding a stage to the multidimensional averaging workflow or removing one of the stages from the multidimensional averaging workflow.

[0014] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the method further includes outputting, by the processor, the aligned workflows and the graphical representation of the change to a display device.

[0015] In addition to or as an alternative to one or more of the features described above or below, further aspects may include: the graphical representation identifies a surgical service provider associated with each of the workflows.

[0016] In addition to or as an alternative to one or more of the features described above or below, further aspects may include: scaled entropy values ​​are used to quantify changes in the surgical methods.

[0017] In addition to one or more of the features described above or below, or as an alternative, further aspects may include: calculating the multidimensional average workflow includes analyzing all center point workflows in the surgical videos, and selecting the multidimensional average workflow from the center point workflows based on the analysis.

[0018] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the analyzing includes performing adaptive dynamic time normalization-center of gravity averaging (ADBA) using each of the center point workflows as an initial average workflow.

[0019] According to another aspect, a computer program product includes a memory device having computer executable instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform operations. The operations include visualizing changes in a surgical method for performing a surgical procedure. The visualization includes receiving a plurality of surgical videos, each of which captures a workflow of the same type of surgical procedure, and each of the plurality of surgical videos is segmented into a segmented workflow, the segmented workflow comprising stages. Changes in the workflows in the plurality of surgical videos are quantified. The quantification includes calculating the stages of an average workflow based at least in part on the plurality of surgical videos, and calculating an entropy value for each of the stages in the average workflow based at least in part on the average workflow and the plurality of surgical videos. A graphical representation of the quantified changes is output to a display device.

[0020] In addition to one or more of the features described above or below, or as an alternative, further aspects may include: the visualization includes receiving user input via a user interface of the graphical representation; and in response to the user input, outputting a second graphical representation to the display device, the second graphical representation including additional information describing characteristics of one or both of the service provider or the patient.

[0021] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that the graphical representation of the quantified change includes a graph of the change over the duration of the surgical procedure.

[0022] In addition to or as an alternative to one or more of the features described above or below, further aspects may include that each of the stages includes multi-dimensional data.

[0023] Additional technical features and benefits are achieved through the technology of the present invention. Various aspects of the present invention are described in detail herein and are considered part of the subject matter claimed for protection. For a better understanding, please refer to the detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The details of the exclusive rights described herein are particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of various aspects of the present invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0025] Figure 1 Described is a computer assisted surgery (CAS) system according to one or more aspects;

[0026] Figure 2 Surgical systems according to one or more aspects are described;

[0027] Figure 3 Described are systems for analyzing video captured by a video recording system according to one or more aspects;

[0028] Figure 4 depicts a block diagram of a data analysis system for quantifying changes in surgical approaches in accordance with one or more aspects;

[0029] Figure 5 A flow chart depicting a method for quantifying changes in surgical approach according to one or more aspects;

[0030] Figure 6 depicts a flow chart of a method for performing dynamic temporal normalization-barycentric averaging (DBA) according to one or more aspects;

[0031] Figure 7 depicts a flow chart of a method for performing adaptive DBA (ADBA) according to one or more aspects;

[0032] Figure 8 A block diagram depicting an alignment process according to one or more aspects;

[0033] Fig. 9 A block diagram depicting an example of visualization of intraoperative changes when an aligned workflow is longer than an average workflow according to one or more aspects;

[0034] Fig.10 depicts a block diagram of compression and entropy calculation of an aligned workflow and visualization of the results according to one or more aspects;

[0035] Fig.11 depicts a comparison of visualization of intraoperative changes using an extension phase and visualization of intraoperative changes using a compression phase according to one or more aspects;

[0036] Fig.12 Depicted are examples of multidimensional workflows according to one or more aspects;

[0037] Fig.13 A block diagram depicting a multi-dimensional workflow alignment according to one or more aspects;

[0038] Fig.14 Depicted are examples of visualizations of workflow standardization according to one or more aspects;

[0039] Fig.15 Depicted are examples of visualizations of intraoperative changes according to one or more aspects;

[0040] Fig.16 Depicted are examples of visualizations of aligned sequences according to one or more aspects; and

[0041] Fig.17 A computer system according to one or more aspects is described.

[0042] The figures depicted herein are illustrative. Without departing from the spirit of the present invention, there may be many variations of these figures and / or operations described herein. For example, actions may be performed in different orders, or actions may be added, deleted, or modified. In addition, the term "coupled" and variations thereof describe having a communication path between two elements, and do not imply a direct connection without an intermediate element / connection between the elements. All of these variations are considered part of this specification. DETAILED DESCRIPTION

[0043] Exemplary aspects of the technical solutions described herein include systems and methods for performing aligned workflow compression and multi-dimensional workflow alignment when quantifying changes in surgical methods for medical surgeries. A key requirement for being able to standardize surgery is the ability to understand and compare surgical variations. In addition, the ability to quantify changes and identify at which stage of the surgery the changes occur is also an important step in understanding the relationship between surgical procedures, patient outcomes, and efficiency. One or more aspects of the present invention provide a representative average sequence of a data set (which can be easily compared with other groups and can be used for multiple alignments) and quantifiable change values ​​throughout the surgery that can be used to understand surgical standardization. This provides a robust standardized metric that can measure compliance with the method and track and quantify improvements, such as those achieved due to training.

[0044] Modern methods use a nonlinear time scaling (NLTS) method to align a series of surgical workflows and use entropy to quantify the variation in the series. This variation has been used to compare standard surgical practices between surgeons of different training levels to understand the main differences between junior surgeons and senior surgeons. NLTS has three main steps: calculating the average workflow, multiple alignment of workflows to the average workflow, and calculating the entropy of the aligned workflows. In modern methods, the average sequence is calculated using dynamic time normalization-center average (DBA), and the entropy is calculated using Shannon entropy. These modern methods have been used to identify different surgical practice standards. However, modern methods have limitations because DBA is designed for workflows of the same length, which is often not the case with surgical workflows. During DBA, the length of the average sequence cannot be changed, which means that the initial average sequence selected at the beginning will set the length of the final average sequence. This means that the final average sequence according to the algorithm may not be the best possible average sequence for the data set. Since each sequence is aligned to this average sequence, this can have a large impact on the results. In addition, DBA is designed to be used with numerical data rather than categorical data (such as surgical stages). Another limitation of modern methods is that the use of Shannon entropy can lead to bias when applied to small sample sizes, as it is based on the probability of each stage occurring. When using small data sets, the observed probability of each stage may differ from its true probability.

[0045] The exemplary aspects of the technical solution described herein follow a logic similar to NLT and start with the calculation of an average sequence. The average sequence can be used to identify the key characteristics of a group, thereby allowing a large group of sequences to be simplified to a single representative sequence. As described herein, the proposed average method is based on adaptive DBA (ADBA), which is an improvement to the use of DBA because it is designed to be used for sequences of different lengths. However, the ADBA method is designed for digital data, and one or more aspects of the present invention described herein modify the modern ADBA method to be used for classified data (such as surgical stages or surgical instruments). In one or more aspects, the average optimization in DBA, the average optimization in ADBA, and the initialization of these two methods are modified. These changes use a center point sequence, i.e., a sequence with the minimum sum of squares distance when compared with a sequence in a set, as a method for finding the "best" average workflow.

[0046] In modern methods of initializing ADBA, any sequence can be selected as the initial average workflow or average sequence, which will then be iteratively optimized. However, when using categorical data (such as surgical stage data), the selection of the initial average sequence can affect the final average sequence. One or more aspects of the present invention address this shortcoming of modern methods by using all center point sequences in a set as the initial average sequence, and selecting the center point sequence from all resulting average sequences.

[0047] In modern methods of optimization in DBA, the average sequence of each element is found within the set of aligned sequences, and the elements in the average sequence are updated to have that value. In one or more aspects of the invention, the mode value is used instead. In the case of multiple modes, an average sequence is generated for each mode value as the element value. Then, the center point version of the average sequence continues to be used in the algorithm.

[0048] In modern methods of optimization in ADBA, elements are inserted or merged into an average sequence where the value of the new element is the average sequence of the elements on either side of it. According to one or more aspects of the invention, two average sequences are generated, one where the element has the value of the element to its right, and a second where the element has the value of the element to its left. The center point average sequence is then selected from the two and used going forward.

[0049] The modifications described above can be used to improve the final average sequence according to the modified ADBA so that the final average sequence is more similar to the sequences in the data set.

[0050] According to one or more aspects of the present invention, each sequence in the set is aligned to a corresponding average sequence using a compact multiple alignment (CMA). This alignment is performed on each sequence relative to the average sequence using dynamic time normalization (DTW). CMA can align multiple stages of a sequence to a single stage in the average sequence. Once all sequences are aligned, they are "unpacked", which includes extending or extending each sequence so that each stage is aligned only to a single stage. The alignment of the sequences is maintained by inserting a duplicate value of the previous stage in the sequence without stage unpacking. This prevents any stage within the sequence from being ignored due to alignment, thereby avoiding any data loss.

[0051] Entropy is purely based on sequencing rather than timing to give a numerical value of the change in the entire set at each point in the operation. Modern methods use one of several methods to calculate Shannon entropy, which is based on the probability of each stage occurring. Previous studies have found that Shannon entropy may be biased for small sample sizes. One or more aspects of the present invention address the bias problem associated with small sample sizes by using the Chao-Shen estimator of Shannon entropy, which has been found to be effective for a variety of sample sizes. Taking into account the number of sequences and the number of possible stages, the entropy calculation is scaled to the maximum possible entropy so that the entropy value ranges from 0 to 1. This allows entropy to be directly comparable between groups.

[0052] One or more aspects of the invention also include calculating error bands for entropy calculations to provide insight into potential errors due to small sample sizes. These error bands can be calculated using a constant error value based on the number of sequences in the collection and performed similarly to a bootstrap method when tested on a data set. This allows for an estimate of the uncertainty faced by the entropy calculations through a simple calculation, thereby improving the accuracy of comparisons between collections of sequences of different sizes.

[0053] In one or more aspects, when alignment of workflows to an average workflow is performed, stages in all workflows are extended to the maximum length required. For example, if four stages in a workflow are aligned to one stage in the average workflow, then all workflows will be extended to four in length at this time. This is done to ensure that no information is lost (e.g., no stages are deleted) and that all workflows are the same length at the end of the process. This method of extending the workflows can cause the aligned workflows to be longer than the average workflow, making some stages appear to be repeated multiple times in a row. When entropy calculations are performed on these aligned workflows, there is one entropy value per stage, which means that these extended stages have multiple entropy values. This may make it more difficult for some users to understand where surgical changes occurred because there is more than one entropy value per stage in the unaligned average workflow.

[0054] According to one or more aspects, a single entropy value for each stage in an average workflow or average sequence is calculated using aligned workflow compression, and the frequency of each stage occurring at each point in the operation is shown. In these aspects, entropy is calculated based on all stages aligned with a stage in the average workflow. Therefore, if for the first stage in the average stage, the aligned workflow has been extended so that all four stages are aligned with a single average stage, one aspect of the method described herein takes all four stages in the aligned workflow and calculates a single entropy combining these four stages. This entropy calculation is performed for each stage in the average workflow. The method can also return the frequency of each stage occurring in the aligned workflow to show what the most common stage is at this time. Aligned workflow compression can make it easier to understand the output describing the changes observed in the surgical operation because an entropy value for each stage in the average workflow is visualized.

[0055] In one or more aspects, the workflow includes one-dimensional data, such as a sequence of surgical stages. In one or more other aspects, the workflow includes multidimensional data, which includes two or more data values ​​(e.g., surgical stages, surgical instruments, and anatomical structures) of each part of the operation. One or more aspects described herein can be used to determine and output a visualization of the changes in multidimensional data related to a group of surgical operations. For example, the changes in multidimensional data (e.g., features such as surgical stages, surgical instruments, kinematics, anatomy, etc.) can be tracked at each point in the operation. According to these aspects, the average sequence is calculated in the same way as the average sequence of one-dimensional data, but at each point in the sequence, the mode value is found in each dimension of the data. Accordingly, when each workflow is aligned with the average workflow, it is aligned based on multiple data dimensions. For example, if two-dimensional data is being analyzed, the value at each point in the operation can be expressed as (value of dimension one, value of dimension two). The distance between (A, B) and (A, C) is equal to 0.5 (i.e., one of the two dimensions has different values). Similarly, the distance between (C, B) and (A, C) is equal to 1 (i.e., both dimensions have different values). In another example, if three-dimensional data is being analyzed, the distance between (A, B, C) and (A, B, D) is equal to 1 / 3 (one of the three dimensions has a different value), and the distance between (A, B, C) and (B, D, C) is equal to 2 / 3 (two of the three dimensions have different values).

[0056] In one example, one or more aspects of the present invention can be used to analyze laparoscopic Roux-en-Y gastric bypass surgery. An average method can be used to generate an average workflow for each group of surgical procedures (e.g., a group from each of several different hospitals). This can be used to identify differences in laparoscopic Roux-en-Y gastric bypass methods between hospitals. Comparison of average workflows can identify, for example, that one hospital in the group uses a post-colonic approach as the standard, while the other hospitals use the most common approach. This provides an easy-to-understand way to compare the methods of each hospital. In another example, the average workflow also identifies surgical tools, and each different combination of surgical stages and surgical tools represents a different point in the sequence in the average workflow. In another example, one or more aspects of the present invention are applied to laparoscopic cholecystectomy surgery, and an average workflow can be found based on the assigned level of the gallbladder, where the higher the level, the more complex the workflow, and the repetition of stages shows an increase in surgical complexity. The average workflow described herein can facilitate inter-group comparisons and identify its key characteristics.

[0057] In addition, one or more aspects of the present invention provide multiple alignments of workflows, which can calculate the possibility of many comparative measures. Compact multiple alignments of surgical workflows based on average sequences generate a set of aligned sequences of the same length with clear similarities and differences. The use of entropy can provide quantifiable change values ​​at each point in the surgery. Due to the scaling of entropy, the calculated values ​​can be directly compared between groups. This can identify high variations within a surgery and compare variations between hospitals.

[0058] Measurable improvements in surgical interventions can be achieved by objectively quantifying surgical standardization, process efficiency, and patient outcomes. One or more aspects of the present invention provide a method that can effectively achieve this for categorical data (such as surgical stage data, surgical tool data, and / or patient anatomical data). One or more aspects of the present invention improve the modern method of using NLTS for surgical stage data by using a new modified ADBA method and a Chao-Shen estimator of Shannon entropy scaled to its maximum possible value. One or more aspects of the present invention provide a representative average sequence of a data set of a surgical workflow (which can be easily compared with other groups (or data sets)) and quantifiable change values ​​that can be used to understand surgical standardization throughout the operation. Robust standardized metrics (such as the standardized metrics provided by one or more aspects of the present invention described herein) can measure surgical methods and track and quantify improvements achieved due to training.

[0059] In an exemplary aspect of the technical solutions described herein, surgical data captured by a computer-assisted surgery (CAS) system and segmented into surgical stages is input into the analysis described herein to quantify changes in surgical approaches to medical procedures.

[0060] Now go to Figure 1 , generally shows an example CAS system 100 according to one or more aspects. The CAS system 100 includes at least a computing system 102, a video recording system 104, and a surgical instrument system 106. Figure 1 As shown, the actor 112 can be a medical staff member performing a surgical procedure on the patient 110 using the CAS system 100. The medical staff member can be a surgeon, an assistant, a nurse, an administrator, or any other actor interacting with the CAS system 100 in a surgical environment. The surgical procedure can be any type of surgery, such as, but not limited to, cataract surgery, laparoscopic cholecystectomy, endoscopic endonasal transsphenoidal approach (eTSA) pituitary adenoma resection, or any other surgical procedure. In other examples, the actor 112 can be a technician, an administrator, an engineer, or any other such person interacting with the CAS system 100. For example, the actor 112 can record data from the CAS system 100, configure / update one or more properties of the CAS system 100, check the past performance of the CAS system 100, repair the CAS system 100, etc.

[0061] A surgical procedure may include multiple stages, and each stage may include one or more surgical actions. A "surgical action" may include incision, compression, stapling, clamping, suturing, cauterization, sealing, or any other such action performed to complete a stage in a surgical procedure. A "stage" refers to a surgical event consisting of a series of steps (e.g., closure). A "step" refers to the completion of a specified surgical goal (e.g., hemostasis). During each step, certain surgical instruments 108 (e.g., clamps) are used to achieve a specific goal by performing one or more surgical actions. In addition, a patient's specific anatomical structure may be the target of (multiple) surgical actions.

[0062] The video recording system 104 includes one or more cameras 105, such as operating room cameras, endoscope cameras, etc. The cameras 105 capture video data of the surgical procedure being performed. The video recording system 104 includes one or more video capture devices, which may include a camera 105 placed in the operating room to capture events around (i.e., outside the body) of the patient undergoing surgery. The video recording system 104 further includes a camera 105 (e.g., an endoscope camera) that passes through the patient 110 to capture endoscopic data. The endoscopic data provides video and images of the surgical procedure.

[0063] Computing system 102 includes one or more memory devices, one or more processors, a user interface device, and other components. Figure 1 All or part of the computing system 102 shown in FIG. 1 may be provided by, for example, Fig.17 The computing system 102 may be implemented in whole or in part by a computer system 1700 of the present invention. The computing system 102 may execute one or more computer executable instructions. The execution of the instructions facilitates the computing system 102 to perform one or more methods, including the methods described herein. The computing system 102 may communicate with other computing systems via wired and / or wireless networks. In one or more examples, the computing system 102 includes one or more trained machine learning models that can detect and / or predict features of a surgical procedure being performed or previously performed. Features may include structures (such as anatomical structures) and surgical instruments 108 in the captured surgical video. Features may further include events such as stages, actions, etc. in the surgical procedure. The detected features may further include actors 112 and / or patients 110. In one or more examples, based on the detection, the computing system 102 may provide suggestions for subsequent actions to be taken by the actor 112. Alternatively or additionally, the computing system 102 may provide one or more reports based on the detection. The detection performed by the machine learning model may be performed in an autonomous or semi-autonomous manner.

[0064] The machine learning model may include an artificial neural network (such as a deep neural network, a convolutional neural network, a recurrent neural network), an encoder, a decoder, or any other type of machine learning model. The machine learning model may be trained in a supervised, unsupervised, or hybrid manner. The machine learning model may be trained to perform detection and / or prediction using one or more types of data acquired by the CAS system 100. For example, the machine learning model may use video data captured via the video recording system 104. Alternatively or additionally, the machine learning model uses surgical instrument data from the surgical instrument system 106. In yet other examples, the machine learning model uses a combination of video data and surgical instrument data.

[0065] Additionally, in some examples, the machine learning model may also use audio data captured during a surgical procedure. The audio data may include sounds made by the surgical instrument system 106 when one or more surgical instruments 108 are activated. Alternatively or additionally, the audio data may include voice commands, snippets, or conversations from one or more actors 112. The audio data may further include sounds made by the surgical instruments 108 during their use.

[0066] In one or more examples, the machine learning model can detect surgical actions, surgical stages, anatomical structures, surgical instruments, and various other features from data associated with the surgical procedure. In some examples, the detection can be performed in real time. Alternatively or additionally, the computing system 102 analyzes surgical data (i.e., various types of data captured during the surgical procedure) in an offline manner (e.g., after the operation). In one or more examples, the machine learning model detects the surgical stage based on detecting some of the features such as anatomical structures, surgical instruments, etc.

[0067] The data collection system 150 can be used to store surgical data, including (multiple) videos captured during the surgical operation. The data collection system 150 includes one or more storage devices 152. The data collection system 150 can be a local storage system, a cloud-based storage system, or a combination thereof. Further, the data collection system 150 can use any type of cloud-based storage architecture, such as a public cloud, a private cloud, a hybrid cloud, etc. In some examples, the data collection system can use distributed storage, that is, the storage devices 152 are located in different geographical locations. The storage device 152 may include any type of electronic data storage medium for recording machine-readable data, such as a semiconductor-based storage medium, a magnetic-based storage medium, an optical-based storage medium, or a combination thereof. For example, the data storage medium may include a flash-based solid-state drive (SSD), a magnetic-based hard drive, a tape, an optical disk, etc.

[0068] In one or more examples, the data collection system 150 can be part of the video recording system 104, and vice versa. In some examples, the data collection system 150, the video recording system 104, and the computing system 102 can communicate with each other via a communication network, which can be wired, wireless, or a combination thereof. Communication between systems can include transmission of data (e.g., video data, instrument data, etc.), data manipulation commands (e.g., browse, copy, paste, move, delete, create, compress, etc.), data manipulation results, etc. In one or more examples, the computing system 102 can manipulate data that has been stored / being stored in the data collection system 150 based on outputs from one or more machine learning models (e.g., stage detection, anatomical structure detection, surgical tool detection, etc.). Alternatively or additionally, the computing system 102 can manipulate data that has been stored / being stored in the data collection system 150 based on information from the surgical instrument system 106.

[0069] In one or more examples, video captured by video recording system 104 is stored on data collection system 150. In some examples, computing system 102 collates portions of the video data stored on data collection system 150. In some examples, computing system 102 filters the video captured by video recording system 104 before the video is stored on data collection system 150. Alternatively or additionally, computing system 102 filters the video captured by video recording system 104 after the video is stored on data collection system 150.

[0070] Now go to Figure 2 , a surgical system 200 is generally shown according to one or more aspects. Figure 2 The example depicts an embodiment that may include or may be coupled to Figure 1 The surgical support system 202 of the CAS system 100. The surgical support system 202 can use one or more cameras 204 to acquire image or video data. The surgical support system 202 can also interface with multiple sensors 206 and effectors 208. The sensors 206 can be associated with surgical support equipment and / or patient monitoring. The effectors 208 can be robotic components or other devices that can be controlled by the surgical support system 202. The surgical support system 202 can also interact with one or more user interfaces 210 (such as various input and / or output devices). When the Figure 1 When performing a surgical procedure on a patient 110 in a surgical setting, the surgical support system 202 can store, access and / or update surgical data 214 associated with the training data set and / or real-time data. The surgical support system 202 can store, access and / or update surgical objectives 216 to help train and guide one or more surgical procedures. User configurations 218 can track and store user preferences.

[0071] Now go to Figure 3 , generally shows a system 300 for analyzing video and data according to one or more aspects. According to various aspects, from Figure 1 The video recording system 104 captures the video and data. The analysis can use machine learning to predict features in the video data including surgical stages and structures (e.g., instruments, anatomical structures, etc.). In one or more examples, the system 300 can be Figure 1 The computing system 102 or a portion thereof. According to some aspects, the system 300 uses data streams in the surgical data to identify the surgical status.

[0072] System 300 includes a data receiving system 305 that collects surgical data, including video data and surgical instrument data. Data receiving system 305 may include one or more devices (e.g., one or more user devices and / or servers) located within and / or associated with a surgical operating room and / or a control center. Data receiving system 305 may receive surgical data in real time, i.e., while a surgical procedure is being performed. Alternatively or additionally, data receiving system 305 may receive or access surgical data in an offline manner, e.g., by accessing a computer stored in a computer program object. Figure 1 The data in the data collection system 150 is accessed.

[0073] The system 300 further includes a machine learning processing system 310 that processes surgical data using one or more machine learning models to identify one or more features in the surgical data, such as surgical stages, instruments, anatomical structures, etc. It will be understood that the machine learning processing system 310 may include one or more devices (e.g., one or more servers), each of which may be configured to include part or all of one or more of the depicted components of the machine learning processing system 310. In some instances, part or all of the machine learning processing system 310 is located in the cloud and / or remote from an operating room and / or physical location corresponding to part or all of the data receiving system 305. It will be understood that several components of the machine learning processing system 310 are depicted and described herein. However, these components are only one example structure of the machine learning processing system 310, and in other examples, different combinations of these components may be used to construct the machine learning processing system 310. Such changes in the combination of components are included in the technical solutions described herein.

[0074] The machine learning processing system 310 includes a machine learning training system 325, which can be a separate device (e.g., a server) that stores its output as one or more trained machine learning models 330. The machine learning models 330 can be accessed by a machine learning execution system 340. In some examples, the machine learning execution system 340 can be separate from the machine learning training system 325. In other words, in some aspects, the device that "trains" the model is separate from the device that "infers" (i.e., uses the trained machine learning model 330 to perform real-time processing of surgical data).

[0075] In some examples, the machine learning processing system 310 further includes a data generator 315 for generating simulated surgical data (e.g., a virtual image set) or recording video data from the video recording system 104 to train the machine learning model 330. The data generator 315 can access (read / write) the data storage 320 to record data, including multiple images and / or multiple videos. The images and / or videos may include images and / or videos collected during one or more procedures (e.g., one or more surgical procedures). For example, the images and / or videos may have been Figure 1 The actor 112 (e.g., surgeon, surgical nurse, anesthesiologist, etc.) in the operation room may use a user device worn during surgery, a non-wearable imaging device located in the operating room, or an inserted Figure 1 In some examples, data storage 320 is associated with an endoscopic camera in a patient 110. Figure 1 In other examples, data storage 320 is part of data collection system 150.

[0076] Each image and / or video recorded in the data storage 320 for training the machine learning model 330 can be defined as a base image and can be associated with other data characterizing the associated surgery and / or rendering specification. For example, the other data can identify the type of surgery, the location of the surgery, one or more people involved in performing the surgery, the surgical goal and / or the surgical result. Alternatively or in addition, the other data can indicate the surgical stage corresponding to the image or video, the rendering specification corresponding to the image or video, and / or the type of imaging device that captured the image or video (for example, if the device is a wearable device, and / or the role of a specific person wearing the device, etc.). Further, the other data may include image segmentation data that identifies and / or characterizes one or more objects (e.g., tools, anatomical objects, etc.) depicted in the image or video. The representation can indicate the position, orientation, or posture of an object in the image. For example, the representation can indicate a set of pixels corresponding to an object and / or an object state resulting from past or current user operations. Positioning can be performed using various techniques for identifying objects in one or more coordinate systems.

[0077] The machine learning training system 325 trains the machine learning model 330 using the recorded data in the data storage 320, which may include simulated surgical data (e.g., a virtual image set) and actual surgical data. The machine learning model 330 may be defined based on a model type and a hyperparameter set (e.g., defined based on input from a client device). The machine learning model 330 may be configured based on a parameter set, which may be dynamically defined based on (e.g., continuous or repeated) training (i.e., learning, parameter adjustment). The machine learning training system 325 may define the parameter set using one or more optimization algorithms to minimize or maximize one or more loss functions. The (learned) parameter set may be stored as part of the trained machine learning model 330 using a specific data structure of the trained machine learning model 330. The data structure may also include one or more non-learnable variables (e.g., hyperparameters and / or model definitions).

[0078] The machine learning execution system 340 can access the (multiple) data structures of the machine learning model 330 and configure the machine learning model 330 accordingly for inference (i.e., prediction). The machine learning model 330 may include, for example, a fully convolutional network adaptation, an adversarial network model, an encoder, a decoder, or other types of machine learning models. The type of the machine learning model 330 may be indicated in the corresponding data structure. The machine learning model 330 may be configured according to one or more hyperparameters and a learned parameter set.

[0079] The machine learning model 330 receives the surgical data to be processed as input during execution and then generates one or more inferences based on the training. Figure 1 The video data captured by the video recording system 104 in the video recording system 104 may include a data stream (e.g., an array of intensity, depth, and / or RGB values) for each of a single image or a set of frames (e.g., including multiple images or images with sequence data), the set of frames representing a time window of fixed or variable length in the video. The video data captured by the video recording system 104 can be received by a data receiving system 305, which can include one or more devices located in an operating room where a surgical procedure is being performed. Alternatively, the data receiving system 305 can include a remotely located device to which the captured video data is streamed in real time during the performance of the surgical procedure. Alternatively or in addition, the data receiving system 305 accesses data from the data collection system 150 or any other data source (e.g., a local or remote storage device) in an offline manner.

[0080] The data receiving system 305 can process the received video and / or data. When a video stream in an encoded format is received, the processing can include decoding so that the data of the image sequence can be extracted and processed. The data receiving system 305 can also process other types of data included in the input surgical data. For example, the surgical data may include additional data streams such as audio data, RFID data, text data, measurements from one or more surgical instruments / sensors, etc., which may represent the stimulation / surgical state in the operating room. The data receiving system 305 synchronizes the different inputs from different devices / sensors before inputting them into the machine learning processing system 310.

[0081] Once trained, the machine learning model 330 can analyze the input surgical data and, in one or more aspects, can predict and / or characterize features (e.g., structures) included in video data including the surgical data. The video data can include continuous images and / or encoded video data (e.g., encoded using a digital video file / stream format and / or codec, such as MP4, MOV, AVI, WEBM, AVCHD, OGG, etc.). The prediction and / or characterization of features can include segmenting the video data or predicting the location of structures using a probabilistic heat map. In some instances, one or more machine learning models include or are associated with preprocessing or enhancements (e.g., intensity normalization, resizing, cropping, etc.) performed before segmenting the video data. The output of one or more machine learning models can include image segmentation or probabilistic heat map data indicating which structures (if any) of a defined set of structures are predicted within the video data, the location and / or location and / or posture of (multiple) structures within the video data, and / or the state of (multiple) structures. The location can be a set of coordinates in an image / frame in the video data. For example, the coordinates can provide a bounding box. The coordinates can provide a boundary around the predicted (multiple) structures. In one or more examples, the machine learning model 630 is trained to perform higher-level prediction and tracking, such as predicting the stage of a surgical procedure and tracking one or more surgical instruments used in the surgical procedure.

[0082] Although some techniques for predicting surgical stages ("stages") in a surgical procedure are described herein, it should be understood that any other stage prediction techniques that do not affect aspects of the technical solutions described herein may be used. In some examples, the machine learning processing system 310 includes a stage detector 350 that uses a machine learning model to identify stages within a surgical procedure ("operation"). The stage detector 350 uses a specific surgical tracking data structure 355 from a list of surgical tracking data structures. The stage detector 350 selects the surgical tracking data structure 355 based on the type of surgical procedure being performed. In one or more examples, the type of surgical procedure is predetermined or input by an actor 112. The surgical tracking data structure 355 identifies a set of potential stages that may correspond to a portion of a particular type of surgery.

[0083] In some examples, the surgical tracking data structure 355 can be a graph including a set of nodes and a set of edges, where each node corresponds to a potential stage. These edges can provide directional connections between nodes, which indicate (via directions) the expected order in which the stages will be encountered throughout the program iteration. The surgical tracking data structure 355 may include one or more branch nodes that feed into multiple next nodes and / or may include one or more divergence points and / or convergence points between nodes. In some instances, the stage indicates a surgical action (e.g., surgical action) being performed or already performed and / or indicates a combination of actions that have been performed. In some instances, the stage is related to the biological state of the patient undergoing surgery. For example, the biological state can indicate complications (e.g., blood clots, arterial / venous obstruction, etc.), prerequisites (e.g., lesions, polyps, etc.). In some examples, the machine learning model 330 is trained to detect "abnormal conditions" such as bleeding, arrhythmias, vascular abnormalities, etc.

[0084] Each node within the surgical tracking data structure 355 can identify one or more features of the stage corresponding to the node. These features can include visual features. In some instances, the node identifies one or more tools that are typically in use or available for use (e.g., on a tool tray) during the stage. The node also identifies one or more roles of people who are typically performing surgical tasks, typical types of movements (e.g., movement of hands or tools), etc. Therefore, the stage detector 350 can use the segmented data generated by the machine learning execution system 340 indicating the presence and / or characteristics of specific objects in the field of view to identify the estimated node corresponding to the real image data. The identification of the node (i.e., the stage) can be further based on the previously detected stage of a given program iteration and / or other detected inputs (e.g., verbal audio data including requests or comments from individuals to individuals, explicit identification of current or past stages, information requests, etc.).

[0085] The stage detector 350 outputs a stage prediction associated with the portion of the video data analyzed by the machine learning processing system 310. The stage prediction is associated with the portion of the video data analyzed by the machine learning execution system 340 by identifying the start time and the end time of the portion of the video data analyzed by the machine learning execution system 340. The output stage predictions may include segments of the video, wherein each segment corresponds to and includes an identification of a surgical stage as detected by the stage detector 350 based on the output of the machine learning execution system 340. Further, in one or more examples, the stage prediction may include additional data dimensions, such as, but not limited to, identification of structures (e.g., instruments, anatomical structures, etc.) identified by the machine learning execution system 340 in the analyzed portion of the video. The stage prediction may also include a confidence score for the prediction. Other examples may include various other types of information in the output stage predictions.

[0086] It should be noted that although some of the figures depict endoscopic video being analyzed, the technical solutions described herein may also be applicable to analyzing video and image data captured by a camera that is not an endoscope (i.e., a camera outside the patient's body) when performing open surgery (i.e., non-laparoscopic surgery). For example, the video and image data may be captured by a camera mounted on one or more personnel (e.g., a surgeon) in the operating room. Alternatively or additionally, the camera may be mounted on a surgical instrument, a wall, or other location in the operating room.

[0087] Now go to Figure 4 , generally shows a block diagram of a data analysis system 400 for quantifying changes in surgical methods according to one or more aspects. All or part of the data analysis system 400 may be composed of Figure 1 The CAS system 100 and / or Fig.17 The computer system 1700 is implemented as described above. Figure 4 The data analysis system 400 shown in FIG. 4 includes a surgical workflow database 402, a surgical workflow method analysis module 404, a surgical change visualization module 406, a surgical method change quantification module 408, and a display device 410. Figure 4 Examples of processing performed by the modules shown are as follows Figures 5 to 16 shown and described below.

[0088] The surgical workflow database 402 stores, for example, Figure 1 The video recording system 104 and / or Figure 2 The video data captured by the camera 204 is, for example, Figure 3 The machine learning processing system 310 is decomposed into various stages. According to various aspects of the technical solutions described herein, the surgical workflow database 402 is located at Figure 1 The data collection system storage device 152 or stored as Figure 2 The surgical data 214. The surgical workflow database 402 includes surgical video data recorded by one or more different surgical service providers during several iterations of the same medical procedure. Figure 4 The display device 410 shown in can be implemented by any user display device accessible to a user who has been granted access to view the analysis results described herein.Aspects include the display device 410 supporting a graphical user interface (GUI) for outputting graphical data.

[0089] All or part of the surgical workflow method analysis module 404, the surgical method variation quantification module 408, and the surgical variation visualization module 406 can be implemented by Figure 1 The surgical workflow method analysis module 404 receives surgical workflow data including video records of multiple surgical operations of the same type from the surgical workflow database 402, and creates groups of similar workflows. The video record or surgical workflow of each instance of a particular type of surgical operation is annotated with a stage and / or structure that has been identified, for example, by the machine learning processing system 310 and / or via manual annotation. The annotation can be used to segment the surgical workflow into surgical stages. Further, machine learning techniques are used to automatically group / cluster the surgical workflow. The grouping can be performed based on one or more factors. These factors may include, for example, but not limited to, the surgical stages identified in each surgical workflow, the medical staff (e.g., surgeons, staff, trainees / interns, etc.) performing the surgical workflow, the hospital / institution where the surgical workflow is being performed, the equipment used for the surgical workflow, etc., or a combination thereof. The surgical method change quantification module 408 calculates an entropy value and assigns the entropy value to a group of surgical workflows to indicate or quantify the changes in the surgical methods in the group. The entropy values ​​and identifiers of the group of surgical workflows may be stored in the surgical workflow database 402 along with pointers to the surgical workflows belonging to each group.

[0090] Figure 4 The surgical change visualization module 406 shown in FIG. 4 can provide different views of analyzing the data stored in the surgical workflow database 402 to the authorized user via the display device 410. For example, each grouping can be shown in a different color, or the surgical workflow captured by different surgical service providers can be shown in different colors. Examples are provided below. Figures 9 to 16In one or more aspects of the present invention, all or a portion of the surgical change visualization module 406 is executed by a processor located on a user device (e.g., a personal computer, a laptop computer, a mobile phone, etc.), and the display device 410 is located on or attached to the user device. All or a portion of the surgical workflow database 402 can be accessed by the surgical change visualization module 406 via one or more network connections.

[0091] Figure 4 The computing environment may be implemented using any appropriate logic, wherein, in various aspects, the logic as referred to herein may include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, etc.), software (e.g., an application, etc.), firmware, or any suitable combination of hardware, software, and firmware.

[0092] It should be understood that Figure 4 The block diagram is not intended to indicate that system 400 will include Figure 4 Instead, system 400 may include Figure 4 Any suitable fewer or additional components not shown in the figure, such as but not limited to one or more additional display devices 410 and / or surgical workflow database 402. In addition, Figure 4 The components shown in can be arranged in different ways. For example, the surgical workflow method analysis module 404 and the surgical method variation quantification module 408 can be located on different computer servers, or they can be part of the same processing unit.

[0093] Now go to Figure 5 , generally shows a flow chart of a method 500 for quantifying changes in a surgical approach according to one or more aspects. All or part of the method 500 may be, for example, Figure 1 CAS system 100, Figure 2 System 200, Figure 3 System 300 and / or Figure 4 The system 400 may be implemented in whole or in part. Figure 5 The method 500 shown in comprises a pipeline that can be used to quantify changes in surgical approach.

[0094] Figure 5 The method 500 shown in uses the NLTS method to extract an average sequence from a collection or dataset of surgical stage sequences (workflows). The average sequence can be used to identify key characteristics of the dataset and to easily compare workflow differences between different groups (e.g., hospitals, doctors, patients, etc.). The average sequence can also be used to align all workflows into a comparable format. The ability to perform multiple alignments allows the possibility of calculating many comparative measures. Figure 5The method 500 shown in also calculates entropy, which is used to quantify surgical variations in the workflow within the set throughout the procedure.

[0095] exist Figure 5 At block 502, surgical workflow data (e.g., surgical video) is acquired or received. Figure 1 The data collection system 150 acquires at least a portion of the surgical workflow data. The acquired surgical workflow data relates to a single type of surgical procedure or a group of similar procedures and may include workflow data from multiple surgical service providers, such as multiple different surgeons and / or multiple institutions (e.g., hospitals) and different surgeons. The surgical workflow data includes video data for each iteration of the surgical procedure, such as a laparoscopic video. The data set acquired at box 502 may include additional data for each surgical procedure, such as, but not limited to, hospital identification / characteristics (e.g., location, frequency of performing surgical procedures, etc.), surgeon identification / characteristics (e.g., years of practice, training, surgical experience, etc.), and / or patient identification / characteristics (e.g., age, height, weight, etc.).

[0096] At box 504, the surgical video is segmented into surgical stages using, for example, visual cues in the video. For example, visual cues may include, but are not limited to, surgical instruments, anatomical structures, etc. in the video. The segmentation may include annotating the video and breaking the video into multiple segments based on the annotations. According to various aspects of the technical solutions described herein, the duration of the segments may also be used to identify the surgical method. In addition, certain surgical stages may be repeated in the same case.

[0097] As used herein, the term "case" refers to one iteration of a surgical procedure, i.e., a single procedure represented by a single surgical workflow. The surgical workflow data or data set acquired at block 502 may include several cases (hundreds or even thousands of cases) of a particular type of surgical procedure obtained from different surgical service providers. The surgical workflow data of a single surgical service provider may also be obtained to analyze consistency and / or methods used by a single surgical service provider.

[0098] exist Figure 5At box 506, an average workflow or sequence is calculated based on a set of workflows. The group can be the entire data set or a subset including a specific hospital, a specific surgical method, or patients with specific characteristics. According to one or more aspects of the present invention, the calculation of the average sequence is based on a modified ADBA to allow ADBA to be applied to categorical data rather than numerical data. ADBA is an iterative process that identifies whether the average sequence is too long or too short compared to the sequences in the set, and it inserts or removes elements accordingly. The value of the element selected for insertion or removal is based on the mode value in the surgery in the set. This produces an average workflow that is as similar as possible to all sequences in the set. The average workflow can be used to reduce a set of multiple different workflows to one workflow that is easy to understand and compare.

[0099] A simplified example using one-dimensional data includes five surgical workflows or cases, each of which is segmented into multiple stages. "A", "B", "C", and "D" represent different stages of a surgical procedure. As shown below, surgical workflow 0 is segmented (e.g., Figure 5 The process of processing the surgical workflow 0 is described in detail in block 504 of FIG. 5 ) into an ordered sequence of stages including “ABCDADB”. “ABCDADB” is a symbolic representation of surgical workflow 0.

[0100] In this example, these segments include phases A, B, C, and D, and the sequence is as follows:

[0101] Surgical workflow 0: ABCDABD,

[0102] Surgical workflow 1: ABCBDABD,

[0103] Surgical workflow 2: AB CB AB D,

[0104] Surgical workflow 3: D AD B AB C, and

[0105] Surgical workflow 4: D AD CB AB C.

[0106] Using ADBA, the average sequence of these sequences or workflows is the same as the following surgical workflow 2: ABCB ABD.

[0107] For the first six elements of the averaged sequence, the phase in the averaged sequence is the mode of the elements in the set of sequences. For the first element in each of the sequences, the mode phase is A, the mode phase for the second element is B, the mode phase for the third element is C, the mode phase for the fourth element is B, the mode phase for the fifth element and the mode phase for the sixth element is B. The final element of the averaged sequence is phase D, which is the most common last element in the set of elements. The sequences are of different lengths, which is why the average phase is not necessarily the mode element near the ends, since the alignment using DTW has shifted the ends of the sequences relative to each other. See below for more information. Figure 6 and Figure 7 An exemplary process for calculating an average workflow is described.

[0108] After calculating the average workflow, the Figure 5 The process continues at block 508 of , where the workflow is aligned to the average workflow using a compact multiple alignment (CMA). CMA is used to align each sequence within the set with the corresponding average sequence. This alignment is performed for each sequence relative to the average sequence using DTW. CMA can align multiple stages of a sequence with a single stage in the average sequence. Once all sequences are aligned, they are "unpacked," which includes extending or extending each sequence so that each stage is aligned with only a single stage in the average sequence. The alignment of the sequences is maintained by inserting duplicate values ​​of the previous stage in the sequences where there is no stage unpacking. This prevents any stage within the sequence from being ignored due to the alignment, thereby avoiding any data loss.

[0109] Returning to the reference example, each sequence is aligned to the average sequence using DTW to produce the following aligned sequences:

[0110] Surgical workflow 0: AA BCDD AB D,

[0111] Surgical workflow 1: AAB CBD AB D,

[0112] Surgical workflow 2: AAB CBBABD,

[0113] Surgical workflow 3: DD AD BB AB C, and

[0114] Surgical workflow 4: D AD CBBABC.

[0115] The order of the sequences is not changed, only additional stages are added where necessary to facilitate alignment. This addition of stages is part of the unpacking process used to align each sequence.

[0116] At block 510, the entropy of each stage in the aligned workflow is calculated. The calculated entropy includes a quantifiable value of surgical variation throughout the surgery. According to one or more aspects of the present invention, the entropy at each element of the aligned sequence is calculated based on the Chao-Shen estimator of Shannon entropy. Shannon entropy is based on the probability that each stage will appear in the element. For each element, the number of occurrences of each stage is found, for the above, for stages A, B, C, D, the number of occurrences of these stages in the first element is 3, 0, 0, 2 respectively. These counts for each stage are used to calculate the entropy of each element in the sequence. Entropy is calculated using the Chao-Shen estimator, which takes the count of the frequency of each stage appearing in the sequence set, for example, for stage D and stage A, the frequency of their occurrence in the first element in the above example is 2 and 3 respectively. The probability of these stages appearing is 2 / 5 and 3 / 5 respectively. The Good Turing frequency is calculated using the following formula: n = the number of stage counts, m = the number of stage counts with a value of 1, unless all stage counts are 1, then m = n-1, C = 1-m / n. The Good Turing frequency is the initial probability of the phase multiplied by C. For the above example, m=0, since no phase count is 1, so C=1, and the Good Turing frequencies are 2 / 5 and 3 / 5. The Chao-Shen estimator is then calculated based on the Good Turing frequency (gtf). For each phase count, entropy=gtf*ln(gtf) / 1-(1-gtf)^n. The total entropy is the sum of the entropies of each phase count multiplied by -1. For the above example, the entropy of phase D (where gtf=2 / 3) is -0.3974, and the entropy of phase A (where gtf=3 / 5) is -0.3097. This gives a total entropy of 0.7071 for the first element in the sequence. The calculation is then scaled by the maximum possible entropy based on the number of sequences and the number of possible phases. For the previous example, there are 5 sequences in the set and a total of 4 possible phases (all phases that appear in any of these sequences). Maximum entropy occurs when there is an equal or as equal a distribution of counts between each possible phase. For the example above, the counts for the four phases would be 2, 1, 1, and 1. The Chao-Shen estimator is then performed on the counts that would produce the maximum entropy, which gives a value of 2.2821 for the maximum possible entropy. Dividing 0.7071 by 2.2821 gives a value of 0.31, which is the scaled entropy of the first element in this example.

[0117] Referring to this example, the entropy of the first element or segment in the aligned sequence is .31 (the first element has 2 Ds and 3 As), the entropy of the second element is .35 (4 As and 1 D), the entropy of the third element is .65 (1 D, 1 A, and 3 Bs), the entropy of the fourth element is .35 (1 D and 4 Cs), the entropy of the fifth element is .35 (1 D and 4 Bs), the entropy of the sixth element is .31 (3 Bs and 2 Ds), the entropy of the seventh element is 0 (5 As), the entropy of the eighth element is 0 (5 Bs), and the entropy of the ninth element is .35 (2 Cs and 3 Ds). These values ​​can be, for example, in any graphical form (as shown below Fig. 9 The image may be presented to the user in the graphical form shown) or in any other format.

[0118] An entropy value of 0 is assigned to elements seven and eight, where all sequences have the same phase. A value of 1 would be associated with the maximum possible entropy, given the number of sequences and possible phases. In this example, the entropy is highest in element 3, which has the most different phases throughout the surgery.

[0119] According to various aspects, Figure 5 The processing performed at blocks 506 and 508 uses aligned workflow compression, as described below with reference to Figure 10 to Figure 11 This results in an entropy value for each stage or segment in the average workflow.

[0120] According to various aspects, Figure 5 The processing shown is performed for a workflow using multidimensional data segmentation, as described below with reference to Figure 12 to Figure 13 as described.

[0121] Figure 5 The processing shown in the figure is not intended to indicate that the operations are to be performed in any particular order, nor that all operations are to be included in every case. Figure 5 All operations shown in . In addition, Figure 5 The processing shown in FIG. 5 may include any suitable number of additional operations.

[0122] Now go to Figure 6 , generally shows a flow chart of a method 600 for performing DBA according to one or more aspects. All or part of the method 600 may be performed by, for example Figure 1 CAS system 100, Figure 2 System 200, Figure 3 System 300 and / or Figure 4 The system 400 may be implemented in whole or in part.

[0123] When compared to modern DBA methods, one or more aspects of the present invention change the way DBA initializes and updates the average sequence. Modern methods of applying DBA specify that any sequence can be used for initialization, however, for categorical data (such as surgical stages), the initial average sequence selected will affect the final results. With this in mind, one or more aspects of the present invention find all center point sequences and then use each center point as the initial average sequence to run DBA. Then, at the end, the best average sequence (center point) is selected as the final average sequence.

[0124] exist Figure 6 At box 602, all center point sequences in the data set are located to serve as initial average sequences. Then, box 620 including boxes 604, 606 and 608 is performed on each of these initial average sequences. The box optimizes the initial average sequence by iteratively updating the average sequence to the mode value of the aligned sequence. Repeat this until the iteration has reached a maximum value or the change in the average sequence is minimal in the last iteration. Repeat this box for each center point sequence in the set. At box 604, the DTW of each sequence in the set to the initial average sequence is calculated. At box 606, the mode value of each element in the average sequence in the sequence is found. If there is a mode, the value of the element in the average sequence is set to the mode value. If there is more than one mode, multiple versions of the average sequence are created, where the element value is set to each mode. At box 608, if the percentage difference between the new average sequence and the previous average sequence is less than a predefined value (e.g., 0.05%, .1%, .01%, etc.) or the maximum number of iterations has been reached, the process continues at box 610. Otherwise, processing continues at block 604 to perform another iteration. At block 610, the center point average sequence from each of the initial center point average sequences is located and used as the final average sequence. In this way, the center point of these average sequences when compared to the sequence set is found and the average sequence continues to be used in the algorithm.

[0125] Figure 6 The processing shown in the figure is not intended to indicate that the operations are to be performed in any particular order, nor that all operations are to be included in every case. Figure 6 All operations shown in . In addition, Figure 6 The processing shown in FIG. 5 may include any suitable number of additional operations.

[0126] In modern methods, the average sequence is updated by finding the centroidal average sequence for each element based on the sequences in the set. However, for categorical data, this is not possible. Therefore, in one or more aspects of the present invention, the mode value of each element is found. In the case of multiple modes, different average sequences are created, one with each mode as an element value. Then, the best average sequence - the center point - is found and continued to be used.

[0127] For example, for a sequence collection:

[0128] A,A,C,D,E

[0129] A,B,B,D,E

[0130] A,B,B,D,E

[0131] A,A,C,D,E

[0132] The mode of each element is as follows: A, A or B, C or B, D, E

[0133] The average sequence combining each of these potential modes includes:

[0134] A,A,C,D,E

[0135] A, B, C, D, E

[0136] A,A,B,D,E

[0137] A,B,B,D,E

[0138] Now go to Figure 7 , generally shows a flow chart of a method 700 for performing adaptive DBA (ADBA) according to one or more aspects. All or part of the method 700 can be performed, for example, by Figure 1 CAS system 100, Figure 2 System 200, Figure 3 System 300 and / or Figure 4 The system 400 may be implemented in whole or in part.

[0139] Similar to Figure 6 Method 600, Figure 7 The method 700 shown in finds and uses all center point sequences as initial average sequences, and selects the best final average sequence and uses it as output. The optimization of the average partition includes inserting a new element or merging two elements to change the length of the partition. In modern ADBA methods, the newly inserted element has the initial value of the average sequence of the elements on either side of it. When these two elements are merged, the resulting element has the value of the average sequence of the merged elements. When using classified data (such as surgical stages), these methods of averaging values ​​are impossible. One or more aspects of the present invention adapt to classified data by creating two potential average sequences in these two cases. One is that the new element has the value of the element on its right, and the other is that the new element has the value of the element on its left. Then, find the best average sequence-the center point, and continue to use this average sequence in the algorithm.

[0140] A simple example involves the following average series:

[0141] A, B, C, D, E

[0142] If we were to insert an element between B and C, the two potential average sequences would be:

[0143] A,B,B,C,D,E

[0144] A,B,C,C,D,E

[0145] According to one or more aspects of the invention, the best version of the two sequences - the center point - will continue to be used.

[0146] Now go to Figure 7 , at box 602, all center point sequences are located for use as initial average sequences. Processing continues at box 740, which includes boxes 702 to 720 and is performed for each initial average sequence. This box performs optimization of the initial average sequence and is performed for each possible center point sequence. The average sequence is optimized by dividing it into partitions, where each partition is either too short or too long when aligned with the sequence by DTW. Each partition is then optimized and concatenated together to provide an updated average sequence. This is iterated until the maximum number of iterations has been reached or the change in the average sequence is minimized.

[0147] At box 702, the DTW of the average sequence and each sequence in the set is calculated. The average sequence is used to calculate the scaling factor. The scaling factor is the value of each element in the average sequence, which indicates whether the element on the average sequence matches multiple elements in the sequence, or vice versa. This indicates whether the average sequence is too long or too short at this time compared to the sequence. When the average sequence is aligned with a sequence using DTW, the alignment coefficient of each element in the average sequence is calculated; if the element matches one element in the sequence, it is 0; if the element matches multiple elements in the sequence, the value of -1 is assigned to the number of elements it matches, or 1 - the number of elements in the average sequence that match one element in the sequence. The scaling factor is then the sum of the alignment coefficients of each element of the average sequence when aligned with each sequence in the set. For example, if Figure 8As shown in the block diagram 800 of , the block diagram depicts the alignment of the following example for sequences ABA, AAABA and A and the average sequence ABA. The alignment coefficients that will be generated by ABA aligned with the first sequence are 0, 0 and 0, because each element in the average sequence will match an element in the sequence. For the second sequence, the first element A of the average sequence will match the first 3 elements in the sequence AAA, resulting in an alignment coefficient of 3-1=2. The following elements will match an element in the sequence, resulting in an alignment coefficient of 2, 0 and 0. For the final sequence, all 3 elements of A will match the only element of the sequence, resulting in an alignment coefficient of -2, -2 and -2. The scaling factor is the sum of the alignment coefficients of each element. These scaling factors will be 0 (0+2-2), -2 (0+0-2), -2 (0+0-2). These values ​​are then scaled by the number of total elements (3 in this case), resulting in alignment coefficients of 0, -0.66, -0.66. This shows that the average sequence is usually too long on the last two elements when compared to these sequences. Positive values ​​indicate the opposite.

[0148] Processing continues at block 704 by partitioning the average sequence and each sequence where the scaling factor changes sign. Partitioning involves separating the average sequence into segments or partitions where each segment is either too long or too short when compared to the sequence. For example, for an average sequence ABCDE with scaling factors of -1, 0, -2, 2, 3, the average sequence will be partitioned into two partitions ABC and DE, the first partition being too long and the second partition being too short when compared to the sequence. Each partition will then be optimized differently by merging elements from the too long partition and inserting elements in the too short partition.

[0149] Processing continues at box 730, repeating the process for each partition created in box 704. The box optimizes each partition within the average sequence by inserting or merging elements within the partition and then performing DBA. Repeat this process for each partition until the scaling factor of the partition has changed sign. The processing performed in box 730 includes boxes 706 to 716. At box 706, depending on the type of partition, that is, too long (scaling factor is negative), too short (scaling factor is positive) or matching (scaling factor is 0), different optimization processes will be performed. For partitions of the same size as the average sequence, processing continues at box 712. Alternatively, for partitions that are too short when compared to the sequence, box 708 is executed to insert elements in the partition. This includes creating two versions of the partition, the elements inserted in one version have the value of the element on its left, and the elements inserted in the other version have the value of the element on its right. Alternatively, for partitions that are too long when compared to the sequence, box 710 is executed to merge two elements in the average sequence with the smallest scaling partition. Two versions of the partition are created at block 710, one version in which the merged element has the value of the left element and the other version in which the merged element has the value of the right element.

[0150] At block 712, DBA is run on the partition average sequence, such as Figure 6 The DBA method 600 is used to optimize these values. This involves optimizing the value of the partition average sequence based on the mode value of the corresponding partition of the sequence. Processing continues at box 714, i.e., finding a new scaling factor, as previously explained. If the sign of the scaling factor has changed, processing continues at box 716. Otherwise, if the sign of the scaling factor has not changed, the average sequence is stored and processing returns to box 708 or 710. At box 716, the stored average sequence as the center point is used as the new average sequence for the partition. After repeating the processing in box 730 for each partition, processing continues at box 718, i.e., the partition average sequences are cascaded together to generate a new average sequence, and processing continues at box 720.

[0151] At block 720, if the percentage difference between the new average sequence and the previous average sequence is less than a predefined value (e.g., 0.05%, .1%, .01%, etc.) or the maximum number of iterations has been reached, processing continues at block 610. Otherwise, processing continues at block 702 to perform another iteration. At block 610, a center point average sequence from each of the initial center point average sequences is located and used as the final average sequence. In this way, the center points of these average sequences when compared to the sequence set are found and continue to be used in the algorithm.

[0152] According to one or more aspects, there are several times in the algorithm that the average sequence is selected from the list by selecting a center point (in DBA and ADBA and in the optimization section at the end of the algorithm). In the case where there is more than one center point, the sequence with the smallest Levenshtein distance when compared to the data set is used. This is a commonly used distance measure when comparing two lists of letters, and this is also true for phase data.

[0153] Figure 7 The processing shown in the figure is not intended to indicate that the operations are to be performed in any particular order, nor that all operations are to be included in every case. Figure 7 All operations shown in . In addition, Figure 7 The processing shown in FIG. 5 may include any suitable number of additional operations.

[0154] As described above, according to various aspects, entropy is calculated for each stage or segment in the aligned workflow to determine the entropy or change in the data at each point in the workflow. For stages that have been extended into multiple stages, there will be an entropy value for each stage in the extended stage, and multiple stages in the resulting aligned workflow may correspond to a single stage in the average workflow. This may confuse some users because some stages are extended, and potentially misleading entropy values ​​may also be generated during these extended stages.

[0155] Now go to Fig. 9 , generally illustrates an example of a visualization 900 of intraoperative changes when an aligned workflow is longer than an average workflow according to one or more aspects. Fig. 9 As shown, a set of workflows 902 are aligned with an average workflow 908 to generate an aligned workflow 904. Fig. 9 As shown, the stages in the aligned workflow 904 are extended to include all stages in the workflow 902, and so that all aligned workflows 904 have the same length so that they can be compared to each other. Fig. 9 The length of the aligned workflow 904 shown in FIG. 9 is nine, which is longer than Fig. 9 The average workflow 908 shown in FIG. 9 is seven in length. Some stages in the average workflow 908 may occupy multiple columns in the visualization of the aligned workflow 904, while other stages may occupy only one column. Fig. 9 As shown, entropy values ​​906 are calculated over the extended aligned workflow 904 , resulting in a visualization that depicts more entropy values ​​906 than the stages in the average workflow 908 .

[0156] In other aspects, aligned workflow compression is implemented to provide a single entropy value for each stage in the average workflow. When aligned workflow compression is performed, each of the additional stages (if any) added during the alignment process is associated with a stage in the average workflow. An entropy value is calculated that includes data from all stages associated with each stage in the average workflow. This may make it easier for the user to understand what the changes are at this time. Then, entropy is calculated as previously described but based on these stage counts, resulting in an entropy value for each stage in the average workflow. This may make the surgical change output smoother and / or clearer.

[0157] Now go to Fig.10 , generally shows a block diagram 1000 of compression and entropy calculation of an aligned workflow and visualization of the results according to one or more aspects. Fig.10 As shown, Fig. 9 Each stage in the aligned workflow 904 is aligned with a stage in the average workflow 908. Fig.10 As shown, the first two stages in the aligned workflow 904 are associated with the first stage in the average workflow 908. In these combined stages or segments, there are seven "A" stages and three "D" stages. This ratio is shown in the first stage of the compressed aligned workflow 1004, where 3 / 10 of the columns are shaded to indicate stages of type "D" and 7 / 10 of the columns are shaded to indicate stages of type "A". In addition, the calculation of entropy is based on the values ​​of all stages in each combined stage, and one entropy value is calculated for each stage in the average workflow 908. Entropy is calculated based on the proportion of time each stage occurs, so it is not affected by differences in the total number of stages in each set. Fig.10 A visualization of a compressed aligned workflow 1004 and a visualization of an entropy value 1002 of the compressed aligned workflow are shown in FIG.

[0158] Now go to Fig.11 , generally illustrates a comparison of visualization of intra-operative changes using an extension phase and visualization of intra-operative changes using a compression phase according to one or more aspects. Fig.11 As Fig. 9 The graphical form shown depicts Fig. 9 904 and their corresponding entropy values ​​906. In this example, the number of stages in the visualization is greater than the number of stages in the average workflow 908. Also, as shown, the entropy value 906 for each stage in the aligned workflow 904 is calculated and displayed.

[0159] Fig.11 Also depicted in graphical form Fig.10908. As previously described, the aligned workflows 904 have been compressed so that the number of stages in each aligned workflow 1004 is the same as the number of stages in the average workflow 908. Also shown in graphical form are the entropy values ​​1002 that are calculated and displayed for each stage in the average workflow 908. Thus, the visualization of the compressed aligned workflows 1004 depicts one column and one entropy value 1002 for each stage in the average workflow 908. In this way, the frequency with which each stage occurs at each point in the surgery is shown in an easily understandable format.

[0160] Now go to Fig.12 , generally shows a block diagram 1200 of a multi-dimensional workflow according to one or more aspects. Fig.12 As shown, (multiple) machine learning models (such as Figure 3 The machine learning model 330 of FIG. 1200 can characterize or segment surgical data along multiple dimensions, as shown in block 1202. Block 1202 shows data in two dimensions including surgical stages and surgical tools. The sequence of surgical stages identified in block 1202 includes stage A, stage B, and stage C. The sequence of tools identified in block 1202 includes tool A, tool B, tool C, tool B, tool C, tool A, tool C, and none. The identified surgical stages and surgical tools are temporally superimposed and used to generate Fig.12 1204. The sequence of stages shown in the two-dimensional workflow 1204 includes (stage A, tool A), (stage A, tool B), (stage A, tool C), (stage A, tool B), (stage B, tool B), (stage B, tool C), (stage B, tool A), (stage C, tool A), (stage C, tool C), and (stage C, tool none).

[0161] Now go to Fig.13 , generally illustrating a block diagram 1300 of a multidimensional workflow alignment according to one or more aspects. The block diagram 1300 includes a multidimensional workflow 1302, an average multidimensional workflow 1306, and an aligned multidimensional workflow 1304.

[0162] The aligned multidimensional workflow 1304 can be used to calculate surgical variation in the same manner as previously described. The techniques used for one-dimensional data can be extended to multiple dimensions such that an average sequence is calculated based on multiple dimensions, an alignment is performed, and entropy is calculated. According to various aspects, in addition to using counts of how often each surgical phase occurs at each point, counts of how often each tool occurs at each point are also used. When two multidimensional workflows are aligned using DTW, the distance between them is found to derive a cost value for each potential alignment. For example, in Fig.13In the example shown, if the surgical phase is different, the cost increases by .5, and if the surgical tool is different, the cost increases by .5.

[0163] The maximum possible value is calculated based on the possible values ​​of all surgical phases and tools when the entropy value is scaled by its maximum possible value. For example, for one-dimensional data, the distance between the first sequence A,B,C and the second sequence A,D,C is 1 because one phase is different. For multidimensional data, the distance between the sequence (A,D),(B,D),(C,D) and (A,D),(D,D),(C,D) is .5. This variation of the distance score between workflows can be used to calculate the average sequence and alignment.

[0164] like Fig.13 As shown, the average multidimensional workflow 1306 is dominated by the majority surgical phase and the majority surgical tool at each point or segment. Because the multidimensional workflows 1302 have different lengths, some of the differences are due to alignment.

[0165] According to aspects, surgical phase and tool variations (eg, entropy) are calculated separately to provide the user with a view of how they vary in their surgical phase workflow and how they vary in their tool operation workflow.

[0166] According to various aspects, the above reference Fig.10 The described compressed alignment process can be applied to Fig.13 Aligned multi-dimensional workflow 1304 is shown.

[0167] Now go to Fig.14 , generally illustrates an example of a visualization 1400 of workflow normalization in accordance with one or more aspects. Fig.14 The visualization 1400 shown in shows the average entropy of surgeries per month, where the shaded area corresponds to the error band of the calculation. This shows the change in workflow over time and the potential error in the entropy calculation based on the number of surgeries performed per month.

[0168] Now go to Fig.15 , generally illustrates an example of a visualization 1500 of changes during surgery according to one or more aspects. This shows the entropy throughout the surgery from the starting point 0 on the left to the end of the surgery at point 1 on the right. The entropy throughout the surgery is shown in blue for a particular surgeon, compared to pink for related departments. This allows a user to compare their changes with those of their department to determine if their changes are consistent with their peers.

[0169] Now go to Fig.16 , generally illustrates an example of a visualization 1600 of aligned sequences according to one or more aspects. Fig.16As shown in the visualization of , for each element in the aligned sequence, a graph showing the entropy value scaled between zero (no change, all workflows have the same value) and one (each workflow has a unique value) is shown. The blue shaded area corresponds to the calculated error band. The colored strips below show the aligned sequence, where each color represents a different stage in the surgery and a row corresponds to one recorded surgery. This can be used to show the amount of change in the surgical procedures in the analyzed video collection over time.

[0170] Respectively Fig.14 , Fig.15 and Fig.16 The visualizations 1400, 1500, 1600 shown in are examples of the type of data that can be generated and displayed to a user. According to various aspects, a graphical representation of the identified surgical method (or other data) is output for display on a display device. User input can be received via a user interface of the graphical representation, and a second graphical representation including, for example, a provider using one or more of the identified surgical methods can be output for display on the display device. In this manner, the visualization is interactive and can be modified based on input from the user. Those skilled in the art will appreciate that the visualization can be generated by Figure 4 The surgical change visualization module 406 provides any number of different visualizations containing different data and in different formats based on the content of the surgical workflow data for output to the Figure 4 A display device 410 is provided.

[0171] Now go to Fig.17 , generally showing a computer system 1700 according to one aspect. As described herein, the computer system 1700 may be an electronic computer framework including and / or employing any number of computing devices and networks utilizing various communication technologies and combinations thereof. The computer system 1700 may be easily expandable, scalable, and modular, and may have the ability to change to different services or reconfigure some features independently of other features. The computer system 1700 may be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smart phone. In some examples, the computer system 1700 may be a cloud computing node. The computer system 1700 may be described in the general context of computer executable instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer system 1700 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0172] like Fig.17As shown, the computer system 1700 has one or more central processing units (CPUs) 1701a, 1701b, 1701c, etc. (collectively or generally referred to as (multiple) processors 1701). The processor 1701 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The processor 1701 is also referred to as a processing circuit, which is coupled to a system memory 1703 and various other components via a system bus 1702. The system memory 1703 may include one or more memory devices, such as a read-only memory (ROM) 1704 and a random access memory (RAM) 1705. The ROM 1704 is coupled to the system bus 1702 and may include a basic input / output system (BIOS) that controls certain basic functions of the computer system 1700. The RAM is a read-write memory coupled to the system bus 1702 for use by the processor 1701. The system memory 1703 provides temporary storage space for the operation of the instructions during operation. The system memory 1703 may include a random access memory (RAM), a read-only memory, a flash memory, or any other suitable memory system.

[0173] Computer system 1700 includes an input / output (I / O) adapter 1706 and a communication adapter 1707 coupled to system bus 1702. I / O adapter 1706 may be a small computer system interface (SCSI) adapter that communicates with hard disk 1708 and / or any other similar components. I / O adapter 1706 and hard disk 1708 are collectively referred to herein as mass storage device 1710.

[0174] Software 1711 for execution on computer system 1700 may be stored in mass storage device 1710. Mass storage device 1710 is an example of a tangible storage medium readable by processor 1701, wherein software 1711 is stored as instructions executed by processor 1701 to cause computer system 1700 to operate as described below with reference to the various figures. Examples of computer program products and execution of such instructions are discussed in greater detail herein. Communications adapter 1707 interconnects system bus 1702 with network 1712, which may be an external network that enables computer system 1700 to communicate with other such systems. In one aspect, a portion of system memory 1703 and mass storage device 1710 together store an operating system, which may be a coordinating Fig.17 The various components shown function as any suitable operating system.

[0175] Additional input / output devices are shown connected to the system bus 1702 via a display adapter 1715 and an interface adapter 1716. In one aspect, adapters 1706, 1707, 1715, and 1716 may be connected to one or more I / O buses that are connected to the system bus 1702 via an intermediate bus bridge (not shown). A display 1719 (e.g., a screen or display monitor) is connected to the system bus 1702 by a display adapter 1715, which may include a graphics controller for improving the performance of graphics-intensive applications, as well as a video controller. A keyboard, mouse, touch screen, one or more buttons, speakers, and the like may be interconnected to the system bus 1702 via an interface adapter 1716, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as the Peripheral Component Interconnect (PCI). Thus, as Fig.17 As configured, the computer system 1700 includes processing capabilities in the form of a processor 1701, as well as storage capabilities including a system memory 1703 and a mass storage device 1710, input devices such as buttons and a touch screen, and output capabilities including a speaker 1723 and a display 1719.

[0176] In some aspects, the communication adapter 1707 can use any suitable interface or protocol (such as Internet Small Computer System Interface, etc.) to transmit data. The network 1712 can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. An external computing device can be connected to the computer system 1700 through the network 1712. In some examples, the external computing device can be an external network server or a cloud computing node.

[0177] It should be understood that Fig.17 The block diagram is not intended to indicate that the computer system 1700 will include Fig.17 Instead, the computer system 1700 may include Fig.17 1700 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, aspects described herein with respect to computer system 1700 may be implemented with any suitable logic, wherein, in various aspects, logic as referred to herein may include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, etc.), software (e.g., an application, etc.), firmware, or any suitable combination of hardware, software, and firmware.

[0178] The present invention may be a system, method and / or computer program product at any possible level of integrated technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon to cause a processor to perform various aspects of the present invention.

[0179] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device (such as punch card or the convex structure in the groove with instructions recorded thereon) and any suitable combination of the above. Computer readable storage medium used herein should not be interpreted as instantaneous signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses transmitted by fiber optic cables), or electrical signals transmitted by wires.

[0180] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0181] The computer-readable program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, a remote computer may be connected to a user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet, using an Internet service provider). In some respects, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized electronic circuits to execute computer-readable program instructions using the state information of the computer-readable program instructions, so as to perform various aspects of the present invention.

[0182] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to various aspects of the present invention. It will be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0183] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a way to implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other equipment to operate in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0184] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device, thereby producing a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0185] The flowcharts and block diagrams in the accompanying drawings show the architecture, functions and operations of possible implementations of systems, methods and computer program products according to various aspects of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing (multiple) specific logical functions. In some alternative embodiments, the functions marked in the box may not occur in the order marked in the accompanying drawings. For example, depending on the functions involved, the two boxes shown in succession can actually be executed substantially simultaneously, or these boxes can sometimes be executed in reverse order. It will also be noted that each box of the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a dedicated hardware-based system that performs a combination of a specified function or action or performs dedicated hardware and computer instructions.

[0186] The description of various aspects of the present invention is presented for the purpose of illustration, but is not intended to be exhaustive or limited to the disclosed aspects. Various modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described aspects. The terms used herein are selected to best explain the principles of various aspects, practical applications or technical improvements over existing technologies on the market, or to enable those of ordinary skill in the art to understand the various aspects described herein.

[0187] Various aspects of the present invention are described herein with reference to the accompanying drawings. Without departing from the scope of the present invention, alternative aspects of the present invention can be designed. Various connections and positional relationships (e.g., above, below, adjacent, etc.) between elements are set forth in the following description and the accompanying drawings. Unless otherwise indicated, these connections and / or positional relationships may be direct or indirect, and the present invention is not intended to be limited in this respect. Accordingly, the coupling of entities may refer to direct or indirect coupling, and the positional relationship between entities may be direct or indirect positional relationships. In addition, the various tasks and process steps described herein may be incorporated into a more comprehensive program or process with additional steps or functions not described in detail herein.

[0188] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms "comprises / comprising / includes / including", "has / having", or "contains / containing" or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0189] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. The terms "at least one" and "one or more" may be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "plurality" may be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connected" may include both indirect "connections" and direct "connections."

[0190] The terms "about," "substantially," "approximately," and variations thereof are intended to include the degree of error associated with measuring a particular quantity based on the equipment available at the time the application was filed. For example, "about" may include a range of ±8% or 5% or 2% of a given value.

[0191] For the sake of brevity, conventional techniques related to making and using various aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing the various technical features described herein are well known. Accordingly, for the sake of brevity, many conventional implementation details are only briefly mentioned herein or omitted entirely, without providing well-known system and / or process details.

[0192] It should be understood that the various aspects disclosed herein may be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, according to examples, certain actions or events of any process or method described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., all described actions or events may not be necessary to implement these techniques). In addition, although for clarity, certain aspects of the present disclosure are described as being performed by a single module or unit, it should be understood that the technology of the present disclosure may be performed by a combination of units or modules associated with, for example, a medical device.

[0193] In one or more examples, the described techniques can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a non-transitory computer-readable medium, which corresponds to a tangible medium such as a data storage medium (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer).

[0194] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the above structures or any other physical structures suitable for implementing the described techniques. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

Claims

1. A system comprising: a machine learning processing system comprising one or more machine learning models trained to recognize a plurality of stages in a video of a surgical procedure; as well as A data analysis system configured to quantify changes in surgical approaches in a plurality of videos capturing the same type of surgical procedures, wherein quantifying the changes in surgical approaches comprises: receiving a plurality of surgical videos, each of the plurality of surgical videos capturing a workflow of the same type of surgical procedure; Segmenting each of the plurality of surgical videos into a segmentation workflow comprising a plurality of stages, the segmentation being performed based on the stages identified by the machine learning processing system; aligning the stages in the segmented workflows with an average workflow of the segmented workflows to create a plurality of aligned workflows, wherein the number of stages of each workflow in the plurality of aligned workflows is the same as the number of stages of the average workflow; Calculating the entropy of each stage in these aligned workflows based on the Chao-Shen estimator; and The entropy of each stage in the aligned workflows is output to a display device for display, wherein the entropy represents the change in the surgical approach of the surgical procedure.

2. The system of claim 1, wherein: Calculating the entropy includes scaling the entropy by its maximum possible value to a value that falls within a specified range of values.

3. A system as claimed in claim 1 or claim 2, wherein: The quantifying the variation of the surgical method further includes calculating the average workflow based on the workflows in the plurality of surgical videos.

4. The system of claim 3, wherein: The calculating the average workflow includes analyzing all center point workflows in the surgical videos, and selecting the average workflow from the center point workflows based on the analysis.

5. The system of claim 4, wherein: The analysis involves performing Adaptive Dynamic Time Normalization-Barycentric Averaging (ADBA) using each of these center point workflows as the initial average workflow.

6. A system as claimed in claim 4 or claim 5, wherein: The analysis is optimized for categorical data and is performed based at least in part on the mode values ​​at various stages on these segmentation workflows.

7. The system of claim 6, wherein: The optimization includes modifying the length of the average workflow by adding a stage to the average workflow or removing one of the stages from the average workflow.

8. A system as described in any preceding item, wherein: The outputting further includes outputting an indication of a surgical service provider associated with one or more of the aligned workflows to the display device for display.

9. A system as described in any preceding item, wherein: Each stage identified by the one or more machine learning models comprises a value of multi-dimensional data comprising a surgical tool and a surgical stage.

10. A computer-implemented method comprising: receiving, by a processor, a plurality of surgical videos, each of the plurality of surgical videos capturing a workflow of a same type of surgical procedure, and each of the plurality of surgical videos being segmented into a segmented workflow based on a plurality of data dimensions, the segmented workflow comprising stages; analyzing a plurality of the segmented workflows to calculate a multi-dimensional average workflow; aligning the plurality of segmented workflows with the multi-dimensional average workflow; as well as Based on the alignment, the variation of surgical techniques in the plurality of segmented videos is quantified.

11. The computer-implemented method of claim 10, wherein: The analysis includes modifying the length of the multidimensional average workflow by adding a stage to the multidimensional average workflow or removing one of the stages from the multidimensional average workflow.

12. The computer-implemented method of claim 10 or claim 11, further comprising outputting, by the processor, the aligned workflows and the graphical representation of the change to a display device.

13. The computer-implemented method of claim 12, wherein: The graphical representation identifies the surgical service providers associated with each of these workflows.

14. The computer-implemented method of any one of claims 10 to 13, wherein: Scaled entropy values ​​were used to quantify the variation in these surgical approaches.

15. The computer-implemented method of any one of claims 10 to 14, wherein: Calculating the multidimensional average workflow includes analyzing all center point workflows in the surgical videos, and selecting the multidimensional average workflow from the center point workflows based on the analysis.

16. The computer-implemented method of claim 15, wherein: The analysis involves performing Adaptive Dynamic Time Normalization-Barycentric Averaging (ADBA) using each of these center point workflows as the initial average workflow.

17. A computer program product comprising a memory device having stored thereon computer executable instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising: Visualize changes in the surgical approach to performing a surgical procedure, including: receiving a plurality of surgical videos, each of the plurality of surgical videos capturing a workflow of a same type of surgical procedure, and each of the plurality of surgical videos being segmented into a segmented workflow, the segmented workflow comprising stages; quantifying the variations of the workflows in the plurality of surgical videos, the quantification comprising calculating stages of an average workflow based at least in part on the plurality of surgical videos, and calculating an entropy value for each of the stages in the average workflow based at least in part on the average workflow and the plurality of surgical videos; as well as A graphical representation of the quantified changes is output to a display device.

18. The computer program product of claim 17, wherein: The visualization further includes: receiving user input via a user interface of the graphical representation; and In response to the user input, a second graphical representation is output to the display device, the second graphical representation including additional information describing characteristics of one or both of the service provider or the patient.

19. A computer program product as claimed in claim 17 or claim 18, wherein: The graphical representation of the quantified change includes a graph of the change over the duration of the surgical procedure.

20. The computer program product of any one of claims 17 to 19, wherein: Each of these stages includes multidimensional data.

Images