Video-based surgical skill assessment using tool tracking

Abstract

Various systems and techniques for performing video-based surgeon technical skill assessment and classification are disclosed. In one aspect, a method for classifying technical skills of a surgeon while performing a surgical procedure is disclosed. During operation, the method receives a tool movement track that includes a sequence of detected tool movements by a surgeon performing a surgical procedure with the surgical tool. The method then generates a sequence of multi-channel feature matrices to mathematically represent the tool motion trajectory. Next, the method performs a one-dimensional (1D) convolution operation on the sequence of multi-channel feature matrices to generate a sequence of context-aware multi-channel feature representations of the tool motion trajectory. The sequence of context-aware multi-channel feature representations is then processed by a translator model to generate the skill classification, wherein the translator model is trained to identify and focus on a subset of the detected sequence of tool motions that is most relevant to the skill classification.

Description

[0001] Priority claims and related patent applications

[0002] This patent application claims the benefit of provisional patent application serial number 63 / 322,166, entitled “Video-based Surgical Skill Assessment Using Tool Tracking,” filed on March 21, 2022, attorney docket number VRB5157USPSP1. The foregoing patent application and provisional patent application are incorporated herein by reference as a part of this patent application. Technical Field

[0003] The disclosed embodiments generally relate to surgical tool motion tracking and surgical skill analysis. More specifically, the disclosed embodiments relate to video-based multi-tool tracking and tool motion analysis using deep learning, and video-based surgeon skill assessment and classification using tool motion trajectories generated from surgical videos. Background Art

[0004] Tracking the movement of the surgeon's instrument (e.g., surgical tool) during minimally invasive surgery (MIS) or minimally invasive procedures (MIP) is a key step in automating the measurement of the surgeon's technical skills. The tool motion metrics calculated by the tool motion path / trajectory have been shown to be relevant to surgeon experience, learning curve progression, and patient outcome measurements. Typically, it is only feasible to directly use the robot kinematics output to calculate the tool motion metrics for training simulators or robot-assisted MIS procedures. However, since most MIP cases worldwide are performed by laparoscopes rather than robots, a widely applicable solution is needed to track the instrument. Using computer vision technology to generate tool position data will enable tool motion metrics to be calculated based only on surgical case videos, thereby not relying on the data output of a specific surgical device.

[0005] The technical skills of the surgeon are usually calculated during video examination by an expert using a rating scale that assigns numerical values ​​to specific characteristics exhibited by the surgeon's movements. For example, the overall surgical evaluation (GOALS) scale of laparoscopic skills uses a five-point Likert scale to evaluate the surgeon's depth perception, two-handed dexterity, efficiency, and tissue handling. The GOALS score is sometimes also used as a baseline for other technologies that compare the surgeon's technical skill level. Various studies have explored technologies that automate technical skill assessment based on the surgeon's movement pattern, rather than relying on manual inspection. For the surgeon's movement pattern, metrics of interest, such as the path length, speed, acceleration, rotation angle, curvature, and tortuosity of the tool, can be calculated and used to determine the surgeon's skill level. In addition, motion smoothness metrics (such as jerk) have been shown to indicate a lack of mastery of the task, which can be used to measure the surgeon's progress along the learning curve. In addition, feature extraction techniques applied to instrument trajectories have been used to classify surgeons by skill level.

[0006] Existing work has shown the ability to differentiate surgeons of different technical skill levels using feature-based methods. However, these existing methods are typically performed in laboratory tasks or using kinematic robotic data from accelerometers inside the robot. However, these existing feature-based methods have been found to have very limited success in more complex and realistic surgical scenarios. Summary of the invention

[0007] The present invention discloses various video-based tool motion tracking and surgeon technical skill assessment systems and techniques based on computer vision and deep learning. More specifically, the disclosed tool motion tracking systems and techniques employ various computer vision and deep learning models that are configured to detect new tools entering a video frame and continuously generate tool position data that can be used to calculate motion trajectories and other motion metrics based specifically on a laparoscopic video feed. The disclosed video-based tool motion tracking framework utilizes various tracking techniques suitable for long sequence tracking with a minimum of identity switching. This is achieved by customizing cost criteria that consider semantic, spatial, and category similarities when assigning detections to established tracks. Because the disclosed tool motion tracking systems and techniques can track multiple surgical tools simultaneously, the disclosed tool motion tracking systems and techniques are also referred to as "multi-surgical tool tracking" (MSTT) systems and techniques.

[0008] Specifically, the disclosed MSTT system and technology adopt at least two deep learning models: a first model, which is configured to detect one or more surgical tools within each video frame; and a second model, which is configured to simultaneously track multiple surgical tools by estimating the trajectories of several tools that exist simultaneously in the video clip and assigning newly detected objects to the correct trajectory. It should be noted that the two deep learning models operate in tandem in a detection and tracking framework, which is designed to correctly link the detected objects across a sequence of frames based on the located bounding boxes of the detected objects. In addition, the second model is also configured to re-identify the tools after the tools temporarily leave the video frame and the corresponding tracks become inactive. Using the disclosed tool tracking model, the MSTT system and technology can evaluate the track / track of each of the tracked tools by continuously establishing (x, y) coordinates for the tracked tools, such as the left-hand tool and the right-hand tool used by the surgeon throughout a given surgical step or task. In some embodiments, after the trajectories are generated by the disclosed MSTT system and technology, the surgeon can calculate the derivatives of the execution metrics such as path length and displacement based on these trajectories.

[0009] The MSTT system and technique are capable of assessing the tracks / trajectories (i.e., x and y image positions) of the right and left tools used by the surgeon in each surgical task. The proposed multi-tool tracking framework adopts a detection tracking scheme, in which at each new frame, the actively used tools are detected and assigned to existing tracks. The MSTT system and technique require two deep learning models to be trained in advance: the first model is a "tool detection" model based on the Faster-RCNN model, which is trained to locate the surgical tool at each frame of the video; and the second model is a re-identification (or "re-ID") model trained to distinguish between multiple simultaneously detected tools. The re-ID model is trained to learn the feature description of each generated bounding box of the detected tool, so that if two bounding boxes belong to the same track, the feature correspondence descriptors of the two bounding boxes should be close in Euclidean space compared to the two feature descriptors of the bounding boxes corresponding to two different tracks.

[0010] The present invention also discloses various video-based surgeon technical skill (or simply "surgeon skill") evaluation and classification techniques, which use tool motion trajectories generated from surgical videos, for example, using the disclosed MSTT system. From the tool motion trajectories generated by the video, reliable motion features can be extracted and used to evaluate the surgeon's technical skills in terms of surgical efficiency and surgical proficiency. In the present disclosure, both feature-based deep learning skill evaluation techniques and converter-based deep learning skill evaluation techniques are provided. More specifically, the disclosed feature-based deep learning skill evaluation techniques utilize statistical values ​​(e.g., averages) of motion features extracted from the generated tool motion trajectories / trajectories to generate surgeon skill classifications. In contrast, the disclosed converter-based deep learning skill evaluation techniques utilize time-based multi-channel motion features extracted from the generated tool motion trajectories / trajectories to generate surgeon skill classifications. The performance of the two deep learning skill evaluation techniques is compared.

[0011] In one aspect, a process for tracking surgical tools in a surgical video is disclosed. The process begins by receiving one or more established tracks of one or more previously detected surgical tools in the surgical video. The process then processes a current frame of the surgical video to detect one or more objects using a first deep learning model. Next, for each of the one or more detected objects, the process further performs the following steps to assign the detected object to the correct track: (1) calculating a semantic similarity between the detected object and each of the one or more established tracks; (2) calculating a spatial similarity between the detected object and the nearest predicted position of each of the one or more established tracks; and (3) attempting to assign the detected object to one of the one or more established tracks based on the calculated semantic similarity and spatial similarity metrics.

[0012] In some embodiments, before processing the current frame, the process converts the frame rate of the surgical video so that the converted frame rate is greater than or equal to a predetermined frame rate, and resizes the current frame to a predetermined image size.

[0013] In some embodiments, the first deep learning model is a Faster-RCNN model trained to detect and classify a set of different types of surgical tools within a given video frame of a surgical video.

[0014] In some embodiments, the one or more previously detected surgical tools include a left-hand tool and a right-hand tool.

[0015] In some embodiments, the process calculates the spatial similarity between the detected object and the most recently predicted position of each of the one or more established trajectories by: (1) using a second deep learning model to extract a set of image features from the detected object; and (2) comparing the extracted set of image features to multiple stored sets of image features associated with the established trajectories to determine whether the detected object and previously detected images of the surgical tool associated with the established trajectories are visually similar.

[0016] In some embodiments, the set of image features forms a feature vector of 128 dimensions.

[0017] In some embodiments, multiple sets of stored image features are associated with multiple previously detected images of the surgical tool within a predetermined time period.

[0018] In some embodiments, before calculating the spatial similarity, the process receives the position of the detected object on the generated bounding box. The process then generates the most recent predicted position of the established track by applying a Kalman filter to the received position of the detected object and the last known position of the established track.

[0019] In some embodiments, the location of the bounding box is the center of the bounding box.

[0020] In some embodiments, the process attempts to assign the detected object to one of the established tracks by using data association techniques on the calculated semantic similarity and the calculated spatial similarity measures between the detected object and each of the one or more established tracks.

[0021] In some embodiments, the data association technique employs a Hungarian method configured to identify a correct track assignment for a detected object within the one or more established tracks by minimizing a cost function for the track assignment between the detected object and the one or more established tracks.

[0022] In some embodiments, the Hungarian method employs a bipartite graph to solve a cost function associated with the detected objects.

[0023] In some embodiments, the process assigns a very high weight to a track assignment to prohibit the track assignment if the corresponding calculated spatial similarity is greater than a predetermined distance threshold.

[0024] In some embodiments, the cost function additionally includes a track ID associated with the established track and a class ID assigned to the detected object.

[0025] In some embodiments, the process assigns a very high weight to a track assignment to prohibit the track assignment if the corresponding track ID associated with an established track does not match the class ID of a detected object.

[0026] In some embodiments, the process also includes a step for recovering an inactive track by: (1) receiving a detected object from one or more detected objects that cannot be assigned to any track in the one or more established tracks; and (2) determining whether the position of the unassigned object is sufficiently close to the last known position of an inactive track in the one or more established tracks, and whether the category ID of the unassigned object matches the track ID of the inactive track; and (3) if so, reassigning the unassigned object to the inactive track to reactivate a previously established track that has become inactive.

[0027] In another aspect, a method for classifying a surgeon's technical skills when performing a surgical procedure is disclosed. During operation, the method receives a tool motion trajectory comprising a sequence of detected tool motions of a surgeon performing a surgical procedure using a surgical tool. The method then generates a sequence of multi-channel feature matrices to mathematically represent the tool motion trajectory. Next, the method processes the sequence of multi-channel feature matrices using a deep learning model to generate a skill classification for the surgeon performing the surgical procedure, wherein the deep learning model has been trained to identify and focus on a subset of tool motions in the sequence of detected tool motions that are most relevant to the skill classification.

[0028] In some embodiments, the deep learning model is a transformer model, and the method processes the sequence of multi-channel feature matrices by: (1) performing a one-dimensional (1D) convolution operation on the sequence of multi-channel feature matrices by convolving each multi-channel feature matrix within the sequence of multi-channel feature matrices with a kernel of a predetermined time length to generate context-aware multi-channel feature representations of the multi-channel feature matrices; and (2) processing the sequence of context-aware multi-channel feature representations of the motion track through the transformer model to generate a skill classification.

[0029] In some embodiments, the converter model identifies and focuses on a subset of tool motions that are most relevant to the skill classification by using a self-attention technique.

[0030] In some embodiments, the process convolves each multi-channel feature matrix with the kernel by convolving each channel of the multi-channel feature matrix with the kernel separately.

[0031] In some embodiments, the 1D convolution operation compares the multi-channel feature matrix at a given time step with multiple adjacent time steps before and after the given time step. Thus, the context-aware multi-channel feature representation embeds a certain amount of learned relationships into the multiple adjacent time steps before and after the given time step.

[0032] In some embodiments, the motion trajectory is generated based on a sequence of tool positions detected within a sequence of video frames captured at a set of time steps. In addition, each multi-channel feature matrix within the sequence of multi-channel feature matrices is generated at a corresponding time step in the set of time steps.

[0033] In some embodiments, the multi-channel feature matrix is ​​composed of at least the following signal channels: (1) a time step; (2) the (X, Y) coordinates of the detected tool position within the corresponding video frame detected at the time step; and (3) the size of the bounding box of the detected tool within the corresponding video frame detected at the time step.

[0034] In some embodiments, the multi-channel feature matrix also includes a temporal mask channel indicating the presence / absence of a tool in a corresponding video frame.

[0035] In another aspect, a surgeon skill classification system is disclosed. The surgeon skill classification system includes one or more processors; and a memory coupled to the one or more processors. In addition, the memory stores instructions that, when executed by the one or more processors, cause the system to: (1) receive a motion trajectory comprising a sequence of detected tool motions of a surgeon performing a surgical procedure using a surgical tool; (2) generate a sequence of multi-channel feature matrices to mathematically represent the motion trajectory; and (3) process the sequence of multi-channel feature matrices using a deep learning model to generate a skill classification for the surgeon performing the surgical procedure, wherein the deep learning model has been trained to identify and focus on a subset of tool motions in the sequence of detected tool motions that are most relevant to the skill classification.

[0036] In some embodiments, the deep learning model is a transformer model, and the memory further stores instructions that, when executed by one or more processors, cause the system to: (1) perform a one-dimensional (1D) convolution operation on a sequence of multi-channel feature matrices by convolving each multi-channel feature matrix within the sequence of multi-channel feature matrices with a kernel of a predetermined time length to generate context-aware multi-channel feature representations of the multi-channel feature matrices; and (2) process the sequence of context-aware multi-channel feature representations of motion trajectories using the transformer model to generate a skill classification.

[0037] In yet another aspect, a method for classifying the technical skills of a surgeon performing one or more surgical tasks is disclosed. During operation, the method receives a motion trajectory comprising a sequence of detected tool motions generated based on a sequence of video frames capturing a surgeon performing a surgical task using a surgical tool. Next, the method decomposes the motion trajectory into a sequence of 2D coordinates in the 2D pixel space of the video frames. The method then calculates a set of tool motion metrics characterizing the detected tool motions based on the 2D coordinate sequence and the changes in the 2D coordinate sequence over time. The method then processes the set of calculated tool motion metrics using a deep learning model to generate a skill classification for a surgeon performing a surgical task, wherein the deep learning model has been trained.

[0038] In some embodiments, the deep learning model is a random forest model, and using the random forest model to process the set of tool motion metrics includes: (1) applying multiple trained decision tree models to the calculated set of tool motion metrics to generate multiple class predictions for a high technical skill class or a low technical skill class for the surgeon; and (2) generating an overall technical skill classification for the surgeon based on a majority vote of the high technical skill class or the low technical skill class in the multiple predictions.

[0039] In some embodiments, each decision tree model in the plurality of trained decision tree models generates a corresponding class prediction based on a randomly selected subset of the set of computed tool motion metrics.

[0040] In some embodiments, the set of calculated tool motion metrics includes the following metrics: (1) distance; (2) velocity; (3) acceleration; (4) jerk; (5) curvature; (6) tortuosity; (7) rotation angle; and (8) motion ratio.

[0041] In some embodiments, the method calculates a tool motion metric in the set of tool motion metrics by the following steps: (1) calculating a value of the tool motion metric at each time step corresponding to each video frame in a sequence of video frames; and (2) calculating the tool motion metric by averaging the set of calculated values ​​of the tool motion metric. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The structure and operation of the present disclosure will be understood by reviewing the following detailed description and accompanying drawings, in which like reference numerals refer to like components, and in which:

[0043] Figure 1 A block diagram illustrating an exemplary multi-surgical tool tracking (MSTT) system for implementing the disclosed user in-place / out-of-place identification technology according to some embodiments described herein is shown.

[0044] Figure 2A An exemplary current frame processed by the disclosed tool detection module according to some embodiments described herein is shown, the current frame including three newly detected objects.

[0045] Figure 2B An exemplary output of the disclosed tool tracking module is shown after assigning two newly detected objects to corresponding left and right tools at a given timestamp in accordance with some embodiments described herein.

[0046] Figure 3 Two exemplary established tracks generated by the disclosed MSTT system tracking a right-hand tool and a left-hand tool are shown according to some embodiments described herein.

[0047] Figure 4 A flowchart illustrating an exemplary process for tracking one or more surgical tools throughout a sequence of video frames of a surgical video according to some embodiments described herein is presented.

[0048] Figure 5 A flow chart illustrating an exemplary process for attempting to recover a previously established inactive track according to some embodiments described herein is presented.

[0049] Figure 6 An algorithm is shown that summarizes the disclosed multi-tool / object tracking and the disclosed tracking recovery / re-ID techniques based on the concept of detection-based tracking according to some embodiments described herein.

[0050] Figure 7 A flow chart illustrating an exemplary process for performing feature-based surgeon skill assessment using tool motion trajectories generated based on surgical videos according to some embodiments described herein is presented.

[0051] Figure 8 A block diagram illustrating a learning-based skill assessment system according to some embodiments described herein that performs surgeon skill classification by directly processing generated trajectories but without relying on extracted motion features is shown.

[0052] Fig. 9 A flow chart illustrating an exemplary process for performing learning-based surgeon skill assessment using tool motion trajectories generated based on surgical videos according to some embodiments described herein is presented.

[0053] Fig.10 A computer system is conceptually illustrated that can be used to implement some embodiments of the subject technology. DETAILED DESCRIPTION

[0054] The specific embodiments listed below are intended to be descriptions of various configurations of the subject technology, and are not intended to represent the only configuration that can practice the subject technology. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details, and their purpose is to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details listed herein, and can be practiced without these specific details. In some cases, structures and components are shown in block diagrams to avoid blurring the concept of the subject technology.

[0055] Various video-based tool motion tracking systems and techniques based on computer vision and deep learning are disclosed. More specifically, the disclosed tool motion tracking systems and techniques employ various computer vision and deep learning models that are configured to detect new tools entering a video frame and to continuously generate tool position data that can be used to calculate motion trajectories and other motion metrics based specifically on a laparoscopic video feed. Because the disclosed tool motion tracking systems and techniques can track multiple surgical tools simultaneously, the disclosed tool motion tracking systems and techniques are also referred to as "multiple surgical tool tracking" (MSTT) systems and techniques.

[0056] Specifically, the disclosed MSTT system and technology adopt at least two deep learning models: a first model, which is configured to detect one or more surgical tools within each video frame; and a second model, which is configured to track multiple surgical tools simultaneously by estimating the trajectories of several tools that exist simultaneously in the video clip and assigning newly detected objects to the correct trajectory. It should be noted that the two deep learning models operate in a detection and tracking framework in tandem, and the detection and tracking framework is designed to correctly link the detected objects across the frame sequence based on the located bounding box of the detected objects. In addition, the second model is also configured to re-identify these tools after the tools temporarily leave the video frame and the corresponding tracks become inactive. Using the disclosed tool tracking model, the MSTT system and technology can estimate the trajectory of each of the tracked tools by continuously establishing (x, y) coordinates for the tracked tools, such as the left-hand tool and the right-hand tool used by the surgeon throughout a given surgical step or task. In some embodiments, after the trajectory is generated by the disclosed MSTT system and technology, the surgeon can calculate the derivatives of the execution metrics such as path length and displacement based on these trajectories.

[0057] The MSTT system and technique are capable of estimating the trajectories (i.e., x and y image positions) of the right and left tools used by the surgeon in each surgical task. The proposed multi-tool tracking framework adopts a detection tracking scheme, in which at each new frame, the actively used tools are detected and assigned to existing tracks. The MSTT system and technique require two deep learning models trained in advance: the first model is a "tool detection" model based on the Faster-RCNN model, which is trained to locate the surgical tool at each frame of the video; and the second model is a re-identification (or "re-ID") model trained to distinguish between multiple simultaneously detected tools. The re-ID model is trained to learn the feature description of each generated bounding box of the detected tool, so that if two bounding boxes belong to the same track, the feature correspondence descriptors of the two bounding boxes should be close in Euclidean space compared to the two feature descriptors of the bounding boxes corresponding to two different tracks. The disclosed MSTT system and technique allow tracking of more than one surgical tool in a video feed for an extended duration (e.g., from several minutes to tens of minutes).

[0058] Figure 1 is a block diagram illustrating an exemplary multi-surgical tool tracking (MSTT) system 100 for implementing the disclosed user in-place / out-of-place identification techniques according to some embodiments described herein. Figure 1 As shown, the robotic surgical system 100 may include at least the following functional modules: a video preprocessing module 102, a tool detection module 104, a tool tracking module 106, and a tool re-identification module 108, which are coupled to each other in the order shown. Specifically, the tool detection module 104 also includes a deep learning-based tool detection model 114, which has been trained to perform surgical tool positioning on video images. In addition, the tool re-identification module 108 includes a deep learning-based tool re-identification model 116, which has been trained to re-identify surgical tools that temporarily disappear from the video frame in the received video frame.

[0059] During operation, the disclosed MSTT system 100 may begin when the video pre-processing module 102 receives a sequence of raw video frames 120. In some embodiments, the video pre-processing module 102 includes a function or sub-module to increase the received raw video frames 120 to a frame rate at least equal to a predetermined frame rate (e.g., 30 frames per second (30fps)). In other words, if the incoming frame rate of the raw video frames 120 is lower than the predetermined frame rate (e.g., incoming frame rate=15fps), the video pre-processing module 102 up-converts the video frames 120 to at least the predetermined frame rate (e.g., to 30FPS). For example, the video pre-processing module 102 may perform the up-conversion operation using one of linear interpolation and optical flow techniques. In addition, the video pre-processing module 102 may also include a function or sub-module for resizing each higher resolution image of the received raw video frames 120 to a quarter high definition at 960p×540p. It should be noted that the two functions / submodules for frame rate conversion and image resizing can be sequentially executed / applied in any given order. In addition, one or both of these two functions or submodules can be omitted. When both functions or submodules are omitted, the video preprocessing module 102 can be omitted from the MSTT system 100. Otherwise, the video preprocessing module 102 outputs a preprocessed video frame 122.

[0060] For each of the received pre-processed video frames 122, the tool detection module 104 is configured to apply a deep learning tool detection model 114 on the video frame and generate one or more detected objects 124 (in the form of localized and classified bounding boxes) as output. In some embodiments, the deep learning tool detection model 114 is implemented based on a Faster-RCNN (regional convolutional neural network) architecture. Specifically, this Faster-RCNN-based detection model has been trained to locate each surgical tool within each frame of the video, such as in each pre-processed video frame 122. In some embodiments, locating the detected surgical tool within the processed video frame includes generating a bounding box around the end effector portion of the detected surgical tool.

[0061] In some embodiments, the tool detection model 114 is trained on a training data set constructed to include various situations of tool configuration / appearance of a given surgical tool (such as a pair of graspers) during surgery. It should be noted that a given surgical tool may present different configurations / appearances during surgery. For example, the jaws of a given tool may be in an open state (the open state may further vary in the degree of opening) or in a closed state. In addition, a given tool may change in camera viewing angle in different frames (e.g., at different spatial angles, or at different distances to the camera lens). In addition, a given tool may also appear to be partially blocked by another tool or tissue, for example. In order to be able to detect the same surgical tool in such a wide variety of possible situations when the surgical tool is in use, the training data set should also include a variety of different sample image sets covering all actual situations of surgical configuration / appearance.

[0062] In some embodiments, a set of training surgical videos are collected from gastric bypass and sleeve gastrectomy surgeries. Next, the images of surgical tools captured in each training video are cropped and labeled by one or more human annotators to provide true value labels to these surgical tool images. It should be noted that the constructed training data set may include multiple subsets of labeled training images of multiple surgical tools, wherein each subset of labeled training images is specially constructed for unique surgical tools of a wide variety of configurations / appearances. After constructing such a multi-tool detection training data set, the training data set is split into a training set (e.g., 80% of the total training data set) and a validation set (e.g., 20% of the total training data set). When the tool detection model 114 has been trained and verified on such a training data set, the trained tool detection model 114 can be applied to a real-time video feed to detect multiple surgical tools. For each of the multiple surgical tools, the trained tool detection model 114 is capable of detecting surgical tools with different learning configurations / appearances. During deployment, the trained tool detection model 114 continues to generate a located bounding box for a given tool, wherein the location and size (in terms of width and height) of the generated bounding box may continue to change. In some embodiments, the detection model 114 is configured such that each initial tool detection is only verified and accepted if the confidence score associated with the detection is above a predetermined threshold.

[0063] After the new tool is detected by the tool detection module 104 within the newly processed frame 122, the tool tracking module 106 in the MSTT system 100 is configured to begin tracking the newly detected tool through subsequent video frames 122 based on the corresponding positioned bounding box and the associated tool ID that continues to be generated by the tool detection module 104. It should be noted that the tool tracking module 106 includes a deep learning-based re-ID model 116 that is configured to process the cropped image within the generated bounding box of the detected tool to determine whether the detected tool belongs to one of the established tracks. In some embodiments, the re-ID model 116 has been trained to generate a feature vector of a predetermined dimension (e.g., 128) of the input image to represent the detected tool in the cropped image. The newly generated feature vector can then be compared with the previously generated feature vector of the previously identified tool to determine whether the detected tool belongs to one of the previously identified tools.

[0064] In some embodiments, the re-ID model 116 is trained using positive samples / positive training data consisting of a large number of positive training image pairs, where each pair of positive training images belongs to the same object. For example, a subset of the training image pairs in the positive training data may belong to the same object / tool, but each given image pair in the subset of training image pairs may consist of two different configurations and / or appearances of the same object / surgical tool during a surgical procedure. For example, two images of the same object / surgical tool may differ in one or more of the following: the viewing angle of the camera, the distance to the camera lens, and the state of the jaws of a given tool (e.g., whether the jaws of a given tool are in an open state or in a closed state, and if in an open state, the different degrees of opening of the jaws).

[0065] In addition, the re-ID model 116 can also be trained with negative samples / negative training data consisting of a large number of negative training image pairs, where each pair of negative training images belongs to two different objects / surgical tools. For example, a subset of the training image pairs in the negative training data may belong to left-hand tools and right-hand tools in different configurations that appear simultaneously in the same video frame during a surgical procedure.

[0066] In some embodiments, after the re-ID model 116 has been trained using both positive training data and negative training data, the re-ID model 116 can then be applied to and process a set of input images of learning objects. The trained re-ID model 116 then generates a feature representation for each processed image of the learning object such that the feature distance between two input images of the same object (calculated based on the two corresponding feature representations) is small, whereas the feature distance between two input images of two different objects (also calculated based on the two corresponding feature representations) is large. More specifically, the re-ID model 116 includes a deep neural network that is configured to learn a set of image features of a given object (e.g., a surgical tool) and output a feature representation / vector of the given object (e.g., a surgical tool) using a set of numbers. Training the deep neural network using the above-mentioned positive training data and negative training data ensures that when the two objects / surgical tools are the same object / surgical tool, the trained deep neural network within the re-ID model 116 generates two sets of numbers / feature vectors for the two objects / surgical tools with a small feature distance; however, when the two objects / surgical tools are different objects / surgical tools, the trained deep neural network generates two sets of numbers / feature vectors for the two objects / surgical tools with a large feature distance.

[0067] The detailed operation of the trained re-ID model 116 is described below. In some embodiments, to allow tracking of a newly detected tool, the tool tracking module 106 may explicitly generate a tracking request for the newly detected tool. However, the tool tracking module 106 may also be configured to automatically trigger a new tracking process in response to detecting a previously undetected tool in a frame (i.e., the assigned tool ID of the detected tool was not previously assigned).

[0068] In some embodiments, each new tool tracking process begins by initiating a track or track slice for each unique tool detection from an initial video frame or multiple initial frames. Specifically, the tool tracking module 106 may initiate a track or track slice with a unique track ID, which may be the same as the unique tool ID generated by the tool detection module 104 for the detected tool. It should be noted that for a multi-tool tracking process, multiple tools may or may not be initially detected in the same initial video frame. However, if the tool detection module 104 detects multiple unique tools from a single initial video frame, the tool tracking module 106 may be configured to simultaneously initiate multiple tracks / track slices corresponding to multiple uniquely detected tools (e.g., both left-hand tools and right-hand tools). Alternatively, if the tool detection module 104 sequentially detects two or more unique tools from multiple initial frames, the tool tracking module 106 may be configured to sequentially initiate two or more tracks / track slices corresponding to two or more sequentially detected tools.

[0069] It should be noted that a common situation with multiple tools in a video clip is when the surgeon simultaneously holds a first tool in the left hand (also referred to as a "left hand tool" or "left tool") and a second tool in the right hand (also referred to as a "right hand tool" or "right tool") and operates the left and right hand tools with both hands. In some embodiments, in order to initiate two tracks / tracks for the left tool and the right tool, the initial designation of the left tool or the right tool after the two tools are detected in the processed video frame 122 can be directly determined by the user through manual input. In other words, the user will view two new detections and use the input component to specify that one of the two detections is for the left tool and the other is for the right tool. Subsequently, two tracks / tracks (hereinafter always referred to as "tracks") are initialized for the left tool and the right tool based on the manual input received from the user. However, in other embodiments, the tool tracking module 106 may be configured to automatically determine and designate the left tool and the right tool, for example, based on the relative positions of the two new detections. For example, the tool tracking module 106 may automatically designate a detected tool that appears on the left side of the video frame as the left tool, and a detected tool that appears on the right side of the video frame as the right tool. Subsequently, the tool detection module 104 is configured to simultaneously track the left tool and the right tool by constructing two independent tracks. It is noted that by properly initiating the left tool and the right tool at the beginning of the multi-tracking process, the tool detection module 104 may then track the left tool and the right tool separately in subsequent video frames, even if the two tools are the same type of tool (e.g., both are graspers).

[0070] In various embodiments, the disclosed tool tracking process may begin when at least one track has been initialized or instantiated. However, when more than one track is initialized (e.g., when both the left tool track and the right tool track have been initialized), the tool tracking process becomes a multi-tool tracking process. During the multi-tool tracking process, the tool detection module 104 continues to operate as usual, i.e., processing each newly received video frame 122 and generating a located bounding box of the tool detected in the newly received video frame (also referred to as a "new frame"). Subsequently, for each newly detected object in the new frame (also referred to as a "new detection"), the tool tracking module 106 is configured to determine whether the newly detected object belongs to one of the existing / established tracks. If so, the new detection can be assigned to one of the established tracks. For example, after the left tool track and the right tool track have been established, and when the tool detection module 104 generates two located bounding boxes of two detected tools in the new frame, the tool tracking module 106 is configured to assign each of the two new detections to the corresponding left tool track or right tool track. Note that after a new detection is assigned to an established track, the established track's most recent tool position may be updated with the new detection's position.

[0071] In various embodiments, the tool tracking module 106 is configured to assign new detections to corresponding established tracks by evaluating a set of metrics. The set of metrics may include: (1) a semantic similarity metric; (2) a spatial similarity metric; and (3) a category-ID / tool-ID matching metric. Specifically, the semantic similarity metric, or simply "semantic similarity," compares the image feature similarity between the newly detected object and the stored tool image associated with the existing track. The spatial similarity metric, or simply "spatial similarity," compares the position of the newly detected object with the predicted position of the tracked tool of the established track in the current video frame. The tool ID matching metric simply compares the tool ID assigned to the newly detected object with the tool ID associated with the existing / established track. Note that the tool tracking module 106 may evaluate the set of metrics individually or collectively for each new detection. In some embodiments, when the set of metrics is evaluated collectively, the tool tracking module 106 may use a loss function that combines the set of metrics using a set of weighted loss terms. Each metric in the set of metrics used to assign new detections to existing tracks is now described in more detail.

[0072] Note that the semantic similarity metric is applied by the tool tracking module 106 to evaluate whether the newly detected object from the newly processed video frame / image is sufficiently similar in appearance to the tracked tool associated with the established track. In some embodiments, in order to perform the semantic similarity evaluation, the corresponding area of ​​the video frame is cropped using the generated bounding box of the newly detected object. The cropped image is then fed to the re-ID model 116 within the tool tracking module 106 to generate a feature vector of a predetermined dimension (e.g., 128 dimensions / numbers) or a 128-dimensional feature vector. Although a 128-dimensional feature vector (or simply "128 features") can be used to describe semantic similarity, other embodiments of the re-ID model 116 can be trained to generate feature vectors of other predetermined dimensions (e.g., 64 dimensions or 256 dimensions) without departing from the scope of the present technology. Because the features / numbers in a given extracted feature vector encode image features such as shape and texture of the cropped image, these features / numbers can be referred to as "semantic features", and the process of generating these feature vectors can be referred to as "semantic feature embedding". Because the bounding boxes generated by the tool detection module 104 may have different sizes, in some embodiments, the tool tracking module 106 is configured to resize each received cropped image to a common input image size before passing each received cropped image through the re-ID model 116.

[0073] For semantic similarity evaluation of new detections, it is assumed that at least one track has been initialized or instantiated. Because each established track is constructed from a sequence of previous detections of the tracked tool and cropped images, a set of semantic feature vectors associated with the set of previous detections of the tracked tool has been extracted and becomes part of the established track, which is stored in memory. In various embodiments, each established track may have the following properties: (1) an array of semantic feature vectors corresponding to the set of previous detections of the tracked tool; and (2) an array of predicted positions of the tracked tool corresponding to the set of previous detections of the tracked tool. In the initial frame, the array of semantic feature vectors may be initialized by semantic feature embedding performed by the re-ID model 116, and the center of the assigned bounding box is used for initial track position prediction. In some embodiments, each established track may have the following mathematical expression: {"pred" or "P": [(x 0 ,y 0 ),(x 1 ,y 1 ),(x 2 ,y 2 ),…]; "semantic_feature" or "F": [f 0, f 1 ,f 2 ,…]}, where “P” is the prediction position array and “F” is the semantic feature vector array. Note that the two arrays “P” and “F” have an element-to-element correspondence. In addition, both arrays of the established track also correspond to the same timing array “T”: [t 0, t 1 ,t 2 ,…], where the set of time points in “T” corresponds to the set of timestamps when the previous set of detections was made and the positions in “P” and the feature vectors in “F” were calculated.

[0074] Therefore, the newly calculated semantic feature vector of the newly detected object can be compared with these historical / stored semantic feature vectors associated with each established track to determine whether the newly detected object and the previous detection image of the tracked tool are visually similar. In other words, in order to determine whether the newly detected object belongs to one of the established tracks (e.g., the left tool track and the right tool track), a semantic feature vector-based comparison is performed between the newly calculated semantic feature vector and the stored semantic feature vector associated with the established track, which is called a "semantic similarity comparison" or a "semantic similarity measure". It should be noted that the semantic similarity measure or semantic similarity comparison is one of the key decision making mechanisms in the proposed multi-tool tracking system. It should be noted that this can be very effective in such scenarios when multiple object detections are performed in the middle or similar positions of a given frame so that the spatial similarity measure cannot reliably assign the multiple detections to the corresponding established tracks. In addition, the semantic similarity measure plays a highly important role in the tool re-ID part of the multi-tool tracking system. For example, after a tracked tool temporarily moves out of a camera's field of view (FOV), and then the same tool re-enters the camera's FOV, a semantic similarity metric may be used to ensure that new detections of the re-appearing tool belong to a previously established track of the same tool.

[0075] In some embodiments, the Euclidean distance between the newly computed semantic feature vector and the historical semantic feature vector associated with the established track can be calculated. In a specific embodiment, given a new detection d at timestamp t and an existing track k, f d It is also assumed that the existing track k is associated with an array F of extracted semantic feature vectors of the detected tool from timestamp 0 to timestamp t-1, where F = [f 0 ,f 1 ,…,f t-1 ]. First calculate f d and the Euclidean distance between each of the feature vectors in the array F. The calculated set of semantic distances may then be sorted in increasing or decreasing order, with larger semantic distances representing smaller / weaker similarities. Next, the average of a predetermined portion / subset (e.g., 10%) of the similarity data (i.e., the calculated Euclidean distances) may be used as the semantic distance / similarity between track k and the new detection d.

[0076] In some embodiments, the calculated semantic distance / similarity is further compared to a predetermined semantic similarity threshold value, based on a threshold distance value. If the calculated semantic distance is greater than the semantic similarity threshold value, it is a strong indicator that the newly detected object does not belong to a given track. However, if the calculated semantic distance is less than the semantic similarity threshold value, it may still not be enough to assign the newly detected object to a given track. For example, it is possible that two simultaneously tracked tools are of the same type, so that they have a very similar appearance (and therefore the calculated semantic similarity is small). Therefore, it is necessary to also compare the spatial similarity between the newly detected object and the given track.

[0077] In some embodiments, when calculating the semantic similarity measure between the newly detected object and the established track from the current / newly processed frame, the tool tracking module 106 is configured to compare the newly detected object with the previous detection of the tracked tool associated with the established track in the last N processed frames immediately before the current frame, rather than comparing only with the last frame immediately before the current frame, where N is an integer. In some embodiments, the N processed frames correspond to between 1 second and 10 seconds of the processed frame, for example, for a video frame rate of 25fps to 30fps, N can be about 100. Therefore, the last N processed frames can be referred to as "short-term memory". One reason why the tool tracking module 106 considers short-term memory rather than only the last frame in the established track is that the actively tracked tool may have blockages from tissue, other tools, and other artifacts. Therefore, there is a risk of encountering a fully or partially blocked tool by comparing only with the last frame in the established track or even the last few frames in the established track, which results in a loss of tracking accuracy. Furthermore, in the case where both the left and right tools are of the same tool type, e.g., both are grippers, it becomes more challenging for the re-ID model to distinguish between the two tools when only the semantic similarity comparison is performed with the last frame of the established track. However, when the semantic similarity comparison is performed to include a longer history of processed frames, the detection accuracy can be significantly improved.

[0078] Note that a "detection gap" may exist in the active track when the tracked tool is temporarily out of view and "lost" in the frame, for example, due to one of the above-mentioned blockages. During such a detection gap, no meaningful historical data can be generated and stored for further comparison (i.e., no corresponding bounding box is generated). Such a detection gap may be as long as several seconds of the video. In some embodiments, the tool tracking model 106 is configured so that a semantic similarity comparison is performed using the short-term memory of the stored semantic feature vectors while avoiding such detection gaps. To this end, each detection gap that is longer than a predetermined time interval (e.g., longer than 1 second) is tracked and recorded / marked as part of the established track. Therefore, when performing a semantic similarity comparison with the short-term memory, the tool tracking module 106 may be configured to identify such a marked detection gap located immediately before or near the current frame. If such a detection gap is identified, the tool tracking module 106 may further track back to the timestamp / time position just before the detection gap, and start a semantic similarity comparison with the N previous detections of the tracked tool before the detection gap when the tracked tool has been positively detected. In other words, in these embodiments, when an actual / positive detection of a tracked tool is made, a semantic similarity comparison is performed using the most recent N processed frames (the actual / positive detection is represented by a cropped image and indicated by the generated / stored semantic feature vector and the location of the corresponding bounding box).

[0079] Back to Figure 1 In the tool tracking module 106, in some embodiments, assigning a newly detected object to one of a plurality of established tracks requires combining the evaluation of a semantic similarity metric and a spatial similarity metric. When considered separately, the spatial similarity metric is used to determine whether the position of the newly detected object matches the most recently predicted position of the tool tracked in the established track. From the perspective of the tool track, the spatial similarity metric is used to ensure that the new detection is close enough to the expected / predicted position of the tracked tool based on the established track. For example, if the established track is tracking a right-hand tool, a new detection of the same tool in the current frame will most likely appear on the right side of the frame. When there are multiple established tracks, the spatial similarity metric can be applied to each of the established tracks. For example, when there are left and right tool tracks, the first newly detected object on the left side of the video image / frame is more likely to be associated with the left tool track, while another newly detected object on the right side of the same video image / frame is more likely to be associated with the right tool track. In other words, it is generally assumed that there is no significant jump / sudden change in the position of the tracked tool from the immediately previous frame to the current frame or between two consecutive frames.

[0080] In some embodiments, the tool tracking module 106 may apply a predetermined spatial-distance threshold to the spatial similarity metric to determine whether the position of a newly detected object (i.e., a new detection) is sufficiently similar / close to the expected / predicted position of the tracked tool based on the established tracks. In some embodiments, if the spatial distance between the position of the new detection and the expected / predicted position of a given established track in the plurality of established tracks is greater than the predetermined spatial distance threshold, a very large weight will be given to the potential assignment of the new detection to the given established track.

[0081] It should be noted that the position of the tool 124 detected by the tool detection module 104 is already known from the associated bounding box. For example, the tool tracking module 106 can use the center of the generated bounding box as the position of the detected tool. In some embodiments, the tool tracking module 106 can use the center of the right edge of the generated bounding box as the position of the detected tool. Separately, the tool tracking module 106 can use a Kalman filter to predict the new position / orientation of the tracked tool in the current frame. It should be noted that the size of the bounding box of the tracked tool tends to change from frame to frame due to multiple dynamic factors, such as changes in the perspective of the tool caused by tool movement, changes in the perspective of the camera, changes in the jaw action (i.e., open jaws versus closed jaws), and combinations of the above. Therefore, it has been found that using the center of the bounding box as the position of the detected tool can achieve better tracking performance than using the edge points of the bounding box. Therefore, the center of the bounding box of the newly detected object can be used to compare with the expected / predicted position of the tracked tool in the corresponding track.

[0082] In some embodiments, in order to simultaneously track multiple tools throughout a sequence of video frames using multiple established tracks (e.g., a left track and a right track) and based on multiple assignment metrics (including a semantic similarity metric and a spatial similarity metric), the tool tracking module 106 uses a data association technique to attempt to assign each newly detected tool within a new frame to an established track in the multiple established tracks. In some embodiments, the data association technique includes a Hungarian method configured to assign newly detected tools to their corresponding tracks in the multiple established tracks. In some embodiments, the purpose of the proposed Hungarian method is to minimize the total cost of all new detections within the current / newly processed frame, rather than minimizing the element-by-element cost (i.e., for each new detection within the current / newly processed frame). In these embodiments, a total cost function for all detections and all multiple established tracks within the current / newly processed frame is constructed and minimized. For example, if there are 3 detected objects and 3 existing tracks in the current frame, a single cost function can be constructed for the 3 detected objects and the 3 existing tracks, so that the proposed Hungarian method will try to assign each of the 3 detected objects to the correct track among the 3 existing tracks simultaneously by minimizing the single / total cost function.

[0083] Mathematically, in order to be able to associate multiple detected objects with multiple established tracks, a cost matrix between each active track t and each new detection d can be first constructed. Next, the Hungarian algorithm is used to minimize this cost matrix. The cost function is defined in Equation 1, which is a combination of (1) a semantic similarity measure, (2) a spatial distance similarity measure, and (3) detection ID and track ID matching.

[0084]

[0085] where t represents the track and d represents the detection. In Equation 1, the first term D on the right hand side feat (.,.) represents the semantic dissimilarity between the image features associated with track t and the image features of detection d. As described above, in order to achieve real-time performance and adapt to changes in the appearance of tracks across a sequence of frames, short-term memory is considered, where a predetermined number of the N most recent frames for a given track are used to compare feature distances with new detections d in the semantic feature embedding space. The second term in Equation 1 represents the spatial distance between the center of detection d and the most recently predicted position of track t (e.g., by using Kalman filtering techniques). It is assumed here that the surgical tool moves only slightly between consecutive frames, which is a reasonably safe assumption at sufficiently high frame rates (e.g., 25fps to 30fps). The expression in the second term also indicates that if the spatial distance is greater than a predetermined distance threshold λ sp, then the corresponding element in the cost matrix is ​​given a very high weight to prohibit such an assignment from occurring. The third term in Equation 1 factors in the class label of the track (or "track ID") and the class label of the detected tool (or "tool ID"), and can therefore be referred to as a "class ID" matching term. In a specific expression, the third term in Equation 1 adds a very high deviation, such as M=1000, to the total cost in the event that the detected class label does not match the class label of a given track, thereby prohibiting such an assignment from occurring.

[0086] To solve the cost function of Equation 1 using the Hungarian method, a bipartite graph may be constructed where the nodes of the bipartite graph are new detections and the closest predictions of existing tracks. In the graph, there is an edge connecting each new detection and each existing track. Each edge of the graph is also assigned a weight calculated based on the calculated semantic similarity and spatial similarity measures between the new detection and the existing track. After constructing the bipartite graph based on the new detection, the closest predictions of the existing track, and the calculated semantic similarity and spatial similarity measures, a bipartite matching is performed to generate an accurate assignment of the new detection in the newly processed frame to the corresponding track.

[0087] Further references Figure 1 In the MSTT system 100 of FIG. 1 , it is noted that the tool tracking module 106 is followed by a tool re-identification module 108, which is configured for track recovery, i.e., attempting to assign new detections to inactive tracks. In some embodiments, the tool re-identification module 108 receives any new detections (referred to as unassigned detections 126) that cannot be assigned to any active track by the tool tracking module 106 at the end of the data association operation. The re-identification module 108 is configured to determine whether the received unassigned detection is associated with a tool that was previously tracked in an established track but became blocked or moved off the screen (which also caused the associated established track to become an inactive track). If so, the re-identification module 108 attempts to recover such previously established track that has become inactive. In some embodiments, the re-identification module 108 may use a modified cost function based on equation (1) to assign unassigned detections d to any inactive track t, which is represented below.

[0088]

[0089] Note that a modified cost function equation (2) can be constructed between each unassigned detection d to each inactive track t. The intuition behind the modified cost function equation (2) is that if a tool emerges from an obstructed organ, the detection module 102 will generally not be able to correctly identify the tool as a "missing" tool, and therefore will not be able to assign the original tool ID associated with the missing tool. However, it is reasonable to assume that the position of the new detection of the missing tool within the frame is close to the last known position of the inactive track. Therefore, a small spatial distance calculated from the last known position can be reliably used to reassign the unassigned detection to the inactive track. In general, the modified cost function equation (2) that assigns new detections to any inactive track is more stringent than the cost function equation (1) that assigns new detections to active tracks. This is achieved by generating very high cost values ​​(which naturally prohibit assignment) when the new detection category ID does not match the category ID of the inactive track, or the last known positions of the new detection and the inactive track are spatially far away from each other. However, if the new detection can pass both of the above two similarity tests, the new detection can be reassigned to the inactive track.

[0090] Figure 2A 2 shows an exemplary current frame 202 including three newly detected objects processed by the tool detection module 104 according to some embodiments described herein. Figure 2A As can be seen in FIG. 2 , the processed frame 202 includes three generated bounding boxes 204 , 206 , and 208 that enclose the end effector portions of three detected surgical tool objects. Figure 2B 2 shows an exemplary output of the tool tracking module 106 after assigning two newly detected objects 212 and 214 to corresponding left tool 222 and right tool 224 at a given timestamp according to some embodiments described herein. Figure 2B As can be seen in FIG. 2 , the first detected object / bounding box 212 is successfully assigned to the left tool 212 , and the second detected object / bounding box 214 is successfully assigned to the right tool 224 .

[0091] Figure 3 Two exemplary established tracks 302 and 304 generated by the disclosed MSTT system 100 tracking a right-hand tool 306 and a left-hand tool 308 are shown according to some embodiments described herein. Figure 3 As shown, track 302 is built for a right-hand tool 306 , while track 304 is built for a left-hand tool 308 . Figure 3 It is also shown that the disclosed MSTT system 100 is able to correctly distinguish between the left-hand tool and the right-hand tool in three instances when the two tools are in close proximity to each other, and assign the two new detections in each of these instances to the correct track. Figure 3The exemplary tracking process also demonstrates that the disclosed MSTT system 100 is capable of tracking multiple tools for extended periods of time (eg, >10 minutes in this example).

[0092] Figure 4 A flowchart illustrating an exemplary process 400 for tracking one or more surgical tools throughout a sequence of video frames of a surgical video according to some embodiments described herein is presented. In one or more embodiments, the steps may be omitted, repeated, and / or performed in a different order. Figure 4 One or more of the steps in the steps. Therefore, Figure 4 The particular arrangement of steps shown should not be construed as limiting the scope of the present technology.

[0093] Process 400 may begin by receiving one or more established tracks of one or more previously detected surgical tools (step 402). In some embodiments, the one or more previously detected surgical tools are detected by a Faster-RCNN model that is trained to locate and classify a set of different types of surgical tools within a given video frame of a surgical video. In some embodiments, the Fast-RCNN detection model is trained to simultaneously detect and classify two surgical tools of the same type (e.g., a left-hand grasper and a right-hand grasper) within a given video frame.

[0094] The process 400 then receives a current frame within the sequence of video frames (step 404) and processes the current frame to detect one or more objects using the same deep learning model (step 406). Next, the process 400 then calculates a semantic similarity measure between each of the one or more detected objects and each of the established tracks (step 408). In some embodiments, the process 400 calculates a semantic similarity measure by first using a re-identification (re-ID) model to extract an image feature set from the one or more detected objects. The process 400 then calculates a semantic similarity measure by comparing the extracted image feature set with a plurality of historical image feature sets associated with the established tracks to determine whether the newly detected object and the detected tool associated with the established track are visually similar to each other. In some embodiments, the image feature set forms a feature vector having 128 dimensions.

[0095] The process 400 additionally calculates a spatial similarity measure between each of the detected one or more objects and the nearest predicted position of each of the established tracks (step 410). In some embodiments, before calculating the spatial similarity measure, the process 400 first receives the center position of the generated bounding box of the detected object. The process 400 further calculates the nearest predicted position of the established track by applying a Kalman filter to the received center position of the detected object and the last known position of the established track. Subsequently, the process 400 attempts to assign each of the detected one or more objects to one of the one or more established tracks based on the calculated semantic similarity measure and spatial similarity measure of the detected object, the tool ID of the established track, and the tool ID of the detected object (step 412). In some embodiments, in order to assign a given detected object to one of the established tracks, the process 400 uses data association techniques for the calculated semantic similarity measure and spatial similarity measure between the given detected object and the one or more established tracks. Specifically, the data association technique employs the Hungarian method, which is configured to identify the correct track assignment for a detected object by minimizing a cost function for the assignment between a given detected object and one or more established tracks. In some embodiments, process 400 uses a bipartite graph to solve the cost function associated with each detected object.

[0096] Figure 5 A flowchart illustrating an exemplary process 500 for attempting to recover a previously established inactive track according to some embodiments described herein is presented. In one or more embodiments, the steps may be omitted, repeated, and / or performed in a different order. Figure 5 One or more of the steps in the steps. Therefore, Figure 5 The particular arrangement of steps shown should not be construed as limiting the scope of the present technology.

[0097] Process 500 may begin by receiving a detected object that cannot be assigned to any of the active tracks in one or more established tracks from the end of process 400 (step 502). It should be noted that an established track may become an inactive track when the associated surgical tool is temporarily blocked by an organ or temporarily moved out of the video frame. Next, process 500 determines whether the position of the unassigned object is close enough to the last known position of the inactive track (step 504). If not, process 500 terminates. However, if the position of the unassigned object is close enough to the last known position of the inactive track, the process further determines whether the category ID associated with the unassigned object matches the track / tool ​​ID of the inactive track (step 506). If not, process 500 terminates. However, if the category ID of the unassigned object matches the track / tool ​​ID of the inactive track, process 500 reassigns the unassigned object to the inactive track to restore the previously established track that has become inactive (step 508).

[0098] Figure 6 An algorithm 600 is shown that summarizes the disclosed multi-tool / object tracking and disclosed track recovery / re-ID techniques based on the concept of detection-based tracking according to some embodiments described herein. As can be seen in the first part of the algorithm 600, at each frame, an object / tool ​​is detected in that frame independently of other frames. Next, each detected object / tool ​​is passed through a re-identification (i.e., re-ID) network to extract appearance features. In addition, for each active track, a Kalman filter is used to predict the position of the corresponding tool. All of the above information is used to construct a cost matrix, which is fed to the Hungarian algorithm to assign new detections to active tracks. Therefore, one or more tracks are created by linking corresponding detections within a certain period of time. It should be noted that the second part of the algorithm 600 is used for track recovery, which is used to match the unassigned detections from the first part of the algorithm 600 with some inactive tracks (if it is possible to do so).

[0099] Feature-based assessment of surgeon skills

[0100] After using the MSTT system 100 to generate tracks / trajectories (also referred to as "paths") for one or more tracked surgical tools in a surgical video that captures the surgeon's surgical movements, each generated tool motion track / trajectory (hereinafter also referred to as a "motion track") can be decomposed into (x, y) coordinates in the two-dimensional (2D) pixel space of the video frame. Next, these 2D coordinates / positions of the tool trajectory and the changes of these 2D positions over time can be used to calculate a set of tool motion characteristics / metrics, which characterize the path / trajectory of the tracked tool and thus characterize the surgical movement. In some embodiments, the set of tool motion metrics (or simply "motion metrics") may include the following metrics: distance (i.e., path length), speed, acceleration, jerk, curvature, tortuosity, rotation angle, and motion ratio. In addition, a set of motion metrics can be calculated at each time step / time step. Generally speaking, a distance metric can first be calculated as the distance between a pair of 2D coordinates corresponding to the current time step and the immediately previous time step. Next, velocity, acceleration, and jerk metrics that measure tool velocity, tool acceleration, and tool jerk, respectively, may be calculated in sequence based on the newly calculated distance metric.

[0101] Separately, after the new 2D coordinates of the track / tool ​​position have been determined for the current time step / time step, curvature, tortuosity, rotation angle, and motion ratio metrics can be calculated at each time step. Specifically, the curvature metric measures the straightness of the path / trajectory, which can be calculated at each time step. The tortuosity metric measures how much the path / trajectory deviates from a straight line, which is typically calculated as the ratio of the length of the path / trajectory between the two endpoints of the path / trajectory to the straight line connecting the same two endpoints. The rotation angle metric measures the direction of tool motion relative to both the previous time step and the next time step. Finally, the motion ratio metric measures the ratio of the amount of time the tool is in motion to the total time of the surgical task.

[0102] It should be noted that after the set of metrics is calculated at each new time step, a set of corresponding statistics (or "statistics") can be determined and / or updated based on the newly calculated set of motion metrics at the new time step. For example, for distance metrics, the corresponding statistic can be the cumulative sum of the total distance captured by the motion trajectory between the initial time step and the current time step. However, for velocity, acceleration, jerk, curvature, tortuosity, and angle of rotation metrics, the corresponding statistics can be continuously determined or updated as the median or average of the corresponding metrics on the entire path / trajectory between the initial time step and the current time step. Table I summarizes the above-mentioned motion metric set, its simple definition, and the manner in which the corresponding statistics are calculated according to some embodiments described herein. After extracting the motion metric set and determining the corresponding statistics, feature-based skill assessment techniques can be applied to the statistical value set to generate a surgeon's skill assessment.

[0103] In some embodiments, the proposed feature-based skill assessment is configured to classify the surgeon's skill based on the corresponding tool trajectory by applying a random forest model to the above-mentioned set of motion metrics extracted from the tool motion trajectory. Note that the random forest model is trained to take as input the set of calculated motion metrics from the trajectory, and predicts a "high skill" class or a "low skill" class for the surgeon based on the input motion metric set of the trajectory.

[0104] More specifically, the random forest model also creates a large number of decision trees, such as hundreds to more than one thousand, each of which is configured to randomly sample a subset of input motion metrics / features. In some embodiments, two of the above-mentioned motion metrics / feature sets will be randomly selected as inputs to each of a large number of decision trees (or "a set of decision trees"). Then, each decision tree will do its best to learn from the received metrics / features to classify the input into one of the high-skill category and the low-skill category. In other words, each decision tree in the decision tree set operates independently of other decision trees using a small subset of the extracted motion metrics / features to make a high-skill decision or a low-skill decision to the best of its ability. Next, the random forest model makes a collective decision on the decision set generated by the set of individual decision trees. For example, a collective decision can be made based on the percentage of high-skill category votes in the decision set and the percentage of low-skill category votes in the decision set, wherein the two calculated percentage values ​​can be used as confidence levels for classifying the surgeon's skills into high-skill categories or low-skill categories.

[0105] It is noted that the disclosed feature-based deep learning skill assessment technique (which may also be referred to as a "feature-based comparator model") can be used to provide a baseline prediction output for comparison with the transformer-based deep learning skill assessment technique described below. It is noted that existing feature-based skill assessment techniques perform skill assessment / prediction based on tool motion metrics / features calculated from kinematic data generated by sensors (e.g., accelerometers) inside the robot. In contrast, the disclosed feature-based comparator model performs skill assessment / prediction based on surgical videos, and more specifically, the above-mentioned tool trajectory / trajectory is constructed from a sequence of video frames without using kinematic data from the robot.

[0106] Figure 7 A flowchart illustrating an exemplary process 700 for performing feature-based surgeon skill assessment using tool motion trajectories generated based on surgical videos according to some embodiments described herein is presented. In one or more embodiments, the steps may be omitted, repeated, and / or performed in a different order. Figure 7 One or more of the steps in the steps. Therefore, Figure 7 The particular arrangement of steps shown should not be construed as limiting the scope of the present technology.

[0107] During operation, the process 700 receives a tool motion trajectory generated based on a sequence of video frames capturing a surgeon performing a surgical task using one or more surgical tools (step 702). In various embodiments, the received tool motion trajectory is generated by the disclosed MSTT system 100. Next, the process 700 decomposes the motion trajectory into (x, y) coordinates in the 2D pixel space of the video frame (step 704). Next, the process 700 calculates a set of tool motion metrics characterizing the tool motion based on the changes in the 2D coordinate sequence and the 2D position sequence of the motion trajectory over time (step 706). As described above, the set of tool motion metrics may include the following metrics: distance (i.e., path length), speed, acceleration, jerk, curvature, tortuosity, rotation angle, and motion ratio. In addition, the set of motion metrics may be calculated at each time step corresponding to each video frame in the sequence of video frames. Next, the process 700 applies a large number (e.g., 100 or more) of trained decision trees to the calculated set of motion metrics to generate a large number of predictions for the surgeon's high skill category or low skill category (step 708). Specifically, each decision tree in the plurality of trained decision trees makes a corresponding skill class prediction based on a randomly selected subset of the calculated motion metric set (e.g., two randomly selected metrics from the motion metric set as input to the decision tree). The process 700 then generates an overall technical skill classification for the surgeon based on a majority vote of two predicted classes (i.e., high skill class or low skill class) in the plurality of predictions (step 710).

[0108] Learning-based assessment of surgeon skills

[0109] In some embodiments, each track generated by the MSTT system 100 can be mathematically represented by a time array comprising a sequence of feature matrices. More specifically, one dimension of the time array is a sequence of time steps corresponding to a set of predicted positions of the tool in the track. At each time step, the time array also includes a feature matrix consisting of at least the following information: (1) a timestamp; (2) the 2D position (i.e., X and Y coordinates) of the tool detected in the corresponding video frame determined at the time step; and (3) the size / area of ​​the bounding box of the tool detected at the time step. Therefore, the sequence of feature matrices representing the track is a multi-channel one-dimensional (1D) signal. In some embodiments, the feature matrix may also include: (4) a time mask indicating the presence / absence of the tool. Note that this time mask parameter may be used because there may be time periods where the tool is not present on the screen but is later restored, which results in empty points or empty spaces in the generated track with undefined feature matrices. Note also that it is not desirable to feed tracks containing empty / undefined feature matrices into a feature-based skill assessment model. The temporal mask parameter in the feature matrix can be used to indicate to the skill assessment model whether the tool is present on the screen at each given time step or for a sequence of time steps. More specifically, when the tool is present on the screen, the temporal mask parameter of a given track may have a "1" or "true" value for those time steps, and when the tool is not present on the screen, the temporal mask parameter of a given track may have a "0" or "false" value for those time steps. It is noted that the temporal mask parameter can be used as an additional input channel of the disclosed deep learning-based skill assessment model.

[0110] Figure 8 A block diagram illustrating a learning-based skill assessment system 800 that performs surgeon skill classification by directly processing the generated trajectories but without relying on extracted motion features according to some embodiments described herein is shown. Figure 8 As can be seen in FIG, the time series of feature matrices 810 representing the generated tracks is the input of the proposed deep learning based skill assessment system 800, which is composed of at least a unidirectional (1D) convolution module 802 and a transformer model 804. The 1D convolution module 802 receives the feature matrix sequence 810 as input, which is then convolved by the 1D convolution module 802 with a kernel of a predetermined time length to generate a learned feature representation of the input track.

[0111] More specifically, the 1D convolution module 802 is configured to compare the multi-channel feature matrix of each given time step with the multi-channel feature matrices associated with multiple adjacent / neighboring time steps of the given time step through a 1D convolution operation, wherein the adjacent time steps may include adjacent time steps before and after the given time step. It should be noted that the convolution operation generates a context-aware multi-channel feature representation / matrix (or "context-aware feature representation" below) for the given time step, which is based on both the feature matrix of the given time step and the feature matrix associated with multiple time-connected time points. Therefore, the 1D convolution module 802 generates a larger context-aware feature representation for each original feature matrix in the input feature matrix 810. It should be noted that the 1D convolution module 802 essentially acts as a projection layer to project each multi-channel feature matrix at a given time step into a larger context-aware feature representation that embeds a certain amount of learned relationships into the neighboring time steps both before and after the given time step. Thus, the 1D convolution module 802 is used to learn the short-range relationship of the tool position from the tool motion trajectory, which represents the atomic motion of the tool. In some embodiments, the 1D convolution module 802 is configured to have a single convolution layer, wherein the weights in the convolution layer are learned by optimizing a cross entropy loss function to classify surgeon skills using a predefined skill criterion.

[0112] In the proposed learning-based skill assessment system 800, the sequence of context-aware feature representations 812 generated by the 1D convolution module 802 is fed into a converter model 804, which is configured to process the sequence of context-aware feature representations 812 to make a decision on the surgeon's skill classification. It is noted that surgical motion and therefore the surgeon's skill information is now embedded / encoded in the sequence of context-aware feature representations 812. However, it should be understood that each generated motion track that has been encoded in the feature representation 812 may include both: (1) surgical motion that is highly relevant to the surgeon's skill and indicative of the surgeon's skill and (2) other motions and / or time periods that are less relevant or irrelevant to the surgeon's skill. It is noted that the types of motion that are considered important and highly relevant to the surgeon's skill encoded in the sequence of feature representations 812 may include specific ways in which the tool is being moved. The proposed converter model 804 is configured to infer the surgeon's skill based at least on the smoothness of the tool being moved (e.g., the smoothness of the motion). For example, the converter model 804 may be configured to identify when a surgeon moves a tool with a fluid and smooth motion as an indication of good skill. The converter model 804 is also configured to identify when a surgeon moves a tool with a jerking and slipping motion and / or with pauses and hesitations as an indication of poor skill.

[0113] However, portions of the input feature representation 812 may be associated with events in the trajectory that are less relevant or irrelevant to the surgeon's skills. For example, such events may include idle time periods when tool motion is minimal or has stopped. Note that these idle time periods may be captured in the motion track / trajectory when the surgeon pauses / stops surgical motion for specific reasons, such as when indecisive in making a decision or waiting for another person's project. However, treating these idle time periods of the input feature representation 812 with equal or substantially equal weights as those highly relevant and important portions of the input feature representation 812 will obviously result in a reduction in the overall accuracy of the predicted skill classification of the skill assessment system 800.

[0114] Because the input to the skill assessment system 800 is a time series of extended duration (e.g., several minutes or longer), and the input time series may include data that is highly relevant and important to skill classification as well as data that is irrelevant or unimportant to skill classification, the transformer model 804 is configured to utilize a self-attention technique to process the input feature representation 812. Specifically, the transformer model 804 in the skill assessment system 800 includes a self-attention module 806, which is configured to process the received feature representation 812 to (1) identify and filter out portions of the sequence of feature representations 812 that are irrelevant or unimportant to the surgeon's skills and are used for skill prediction; and (2) focus the attention of the transformer model 804 on those identified portions of the sequence of feature representations 812 that are highly relevant and important to the surgeon's skills for skill prediction.

[0115] Note that in the same manner that a natural language processing (NLP) converter focuses its attention on the most important parts of the input text for category classification, the disclosed self-attention module 806 is configured to focus attention on those parts of the motion trajectory that are most relevant and important for skill assessment and classification. Therefore, the input sequence of the feature matrix 812 can be analogized to the word sequence of the input sentence / paragraph / article to the translator model. In some embodiments, the converter model 804 and the self-attention module 806 are configured to process the input sequence of the feature matrix 812 one small segment at a time. Therefore, as the converter model 804 moves down the input sequence, the generated attention weights are continuously updated to refocus the attention of the converter model 804 for different segments of the input sequence of the feature matrix 812. Ultimately, the converter model 804 generates a skill classification vector 814 as an output of the learning-based skill assessment system 800, wherein the skill classification vector 814 may include a calculated probability value for a high skill classification and a calculated probability value for a low skill classification. Note that because the input to the skill assessment system 800 is a sequence of feature matrices and the output of the skill assessment system 800 is a skill classification vector (e.g., a skill classification vector for two categories), the converter model 804 can be configured based on a sequence-vector model, where the vector is the output skill classification.

[0116] Even though the skill assessment system 800 is configured to generate a single skill classification decision for the input time series of the feature matrix 810, the input time series of the feature matrix 810 does not have to correspond to the entire track. Instead, the input time series of the feature matrix 810 can be customized to correspond to any portion of the generated track, and therefore can vary in both duration and position in the track. Therefore, the learning-based skill assessment system 800 can be used to generate a more targeted skill assessment / classification for any portion of the generated motion track. In other words, the learning-based skill assessment system 800 is able to handle any length of the input time series of the feature matrix 810, and therefore is able to handle any duration in the corresponding motion track.

[0117] Fig. 9 A flowchart illustrating an exemplary process 900 for performing learning-based surgeon skill assessment using tool motion trajectories generated based on surgical videos according to some embodiments described herein is presented. In one or more embodiments, the steps may be omitted, repeated, and / or performed in a different order. Fig. 9 One or more of the steps in the steps. Therefore, Fig. 9 The particular arrangement of steps shown should not be construed as limiting the scope of the present technology.

[0118] During operation, the process 900 receives a tool motion trajectory generated based on a sequence of video frames capturing a surgeon performing a surgical task using one or more surgical tools (step 902). In various embodiments, the received tool motion trajectory is generated by the disclosed MSTT system 100. Next, the process 900 generates a time series of feature matrices to mathematically represent the motion trajectory (step 904). Specifically, each time step in the motion trajectory can be represented by a feature matrix within the time series of generated feature matrices, which is composed of the following set of parameters: (1) a timestamp; (2) the (x, y) coordinates of the tool detected in the video frame at the time step; (3) the size / area of ​​the bounding box of the tool detected at the time step; and optionally (4) a temporal mask indicating the presence / absence of the tool in the video frame. Note that the mathematical representation of the motion trajectory is a multi-channel 1D signal.

[0119] Next, process 900 performs a 1D convolution operation on the time series of feature matrices by convolving each feature matrix at a given time step with a kernel of a predetermined time length to generate a context-aware multi-channel feature representation of each feature matrix of the input time series (step 906). Thus, the input multi-channel feature matrix at a given time step is projected into a larger context-aware feature representation that embeds a certain amount of learned relations into the neighboring time steps before and after the given time step. Note that the 1D convolution operation of step 906 is used to learn the short-range relationship of the tool position from the tool motion trajectory, which represents the atomic motion of the tool.

[0120] Process 900 then processes the sequence of context-aware feature representations of the motion trajectory using a transformer model configured with a self-attention module to generate a surgeon skill classification for the input motion trajectory (step 908). In some embodiments, the transformer model is configured to infer the surgeon skill level based at least on the smoothness of the tool motion embedded in the sequence of context-aware feature representations. More specifically, the transformer model is configured to both identify the surgeon moving the tool with a fluid and smooth motion as an indicator of good skill and identify the surgeon moving the tool with a jerking and slipping motion and / or with pauses and hesitations as an indication of poor skill. In some embodiments, the self-attention module within the transformer model is configured to process the sequence of context-aware feature representations to (1) filter out portions of the feature representation that are not relevant to the surgeon's skill and (2) focus the transformer model's attention on those portions of the feature representation that are highly relevant to the surgeon's skill for skill prediction.

[0121] Please note that the combination Figure 7 Describe the trait-based skills assessment techniques and combine them Figures 8 to 9The described learning-based skill assessment techniques all have the ability to classify the surgeon's skills directly from tool motion tracking information. However, feature-based skill assessment techniques generate motion metrics by calculating a single average across the entire track. This means that if a surgeon generally performs a task well over a long period of time, but has one or two short "bad" moments, these short "bad" moments will be averaged out by the long-term good performance. Therefore, if identifying such "bad" moments is important for making an accurate skill assessment, the feature-based skill assessment method does not have the ability to focus on those important events (e.g., short bad moments) by using a single average metric across the entire track. In contrast, the disclosed learning-based skill assessment technique is able to learn and determine what types of motion are important / relevant motions for the model to focus on. The disclosed learning-based skill assessment technique is also able to create representations of specific types of motion or motion sequences that are important to predefined skill standards. The learning-based skill assessment technique can then focus on a portion of the track corresponding to such motion or motion sequence, regardless of the length of the motion or motion sequence. It has been observed that learning-based skill assessment techniques generally outperform feature-based skill assessment techniques, suggesting that learning-based skill assessment techniques are able to extract motion features that are more meaningful and relevant to skill assessment than conventional motion features used by feature-based skill assessment techniques.

[0122] Fig.10 Conceptually illustrated are computer systems that can be used to implement some embodiments of the subject technology. Computer system 1000 can be a client, server, computer, smart phone, PDA, laptop or tablet computer, or any other type of computing device in which one or more processors are embedded or coupled. Such computer systems include various types of computer-readable media and interfaces for various other types of computer-readable media. Computer system 1000 includes bus 1002, processing unit 1012, system memory 1004, read-only memory (ROM) 1010, permanent storage device 1008, input device interface 1014, output device interface 1006, and network interface 1016. In some embodiments, computer system 1000 is part of a robotic surgical system.

[0123] Bus 1002 collectively represents all system buses, peripheral buses, and chipset buses that communicatively connect the various internal devices of computer system 1000. For example, bus 1002 communicatively connects processing unit 1012 with ROM 1010, system memory 1004, and permanent storage device 1008.

[0124] Processing unit 1012 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the various processes described in this patent disclosure, including in conjunction with Figures 1 to 6 The above-mentioned multiple surgical tool detection and tracking technology and the combination Figures 7 to 9 The above-mentioned video-based surgeon skill assessment technology. The processing unit 1012 may include any type of processor, including but not limited to a microprocessor, a graphics processing unit (GPU), a tensor processing unit (TPU), an intelligent processor unit (IPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). In different specific implementations, the processing unit 1012 can be a single processor or a multi-core processor.

[0125] ROM 1010 stores static data and instructions required by processing unit 1012 and other modules of the computer system. On the other hand, permanent storage 1008 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system 1000 is turned off. Some specific implementations of the subject disclosure use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage 1008.

[0126] Other implementations use removable storage devices (such as floppy disks, flash drives and their corresponding disk drives) as permanent storage devices 1008. Like permanent storage devices 1008, system memory 1004 is a read-write memory device. However, unlike storage devices 1008, system memory 1004 is a volatile read-write memory, such as random access memory. System memory 1004 stores some of the instructions and data that the processor needs at run time. In some implementations, the various processes described in this patent disclosure, including in conjunction with Figures 1 to 6 The above-mentioned multiple surgical tool detection and tracking technology and the combination Figures 7 to 9 The above-described video-based surgeon skill assessment techniques are stored in the system memory 1004, the permanent storage device 1008, and / or the ROM 1010. The processing unit 1012 retrieves instructions to be executed and data to be processed from these various memory units in order to perform some specific implementation process.

[0127] The bus 1002 is also connected to an input device interface 1014 and an output device interface 1006. The input device interface 1014 enables a user to communicate information to the computer system and select commands for the computer system. Input devices used with the input device interface 1014 include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 1006 enables, for example, the display of images generated by the computer system 1000. Output devices used with the output device interface 1006 include, for example, a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some specific implementations include devices that are used as both input devices and output devices, such as a touch screen.

[0128] Finally, if Fig.10 As shown, bus 1002 also couples computer system 1000 to a network (not shown) via network interface 1016. Thus, the computer may be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN"), an intranet, or a network of networks, such as the Internet. Any or all components of computer system 1000 may be used in conjunction with the subject disclosure.

[0129] Various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed in this patent disclosure may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0130] Hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic components, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of receiver devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuits specific to a given function.

[0131] In one or more exemplary aspects, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-transient computer-readable storage medium or a non-transient processor-readable storage medium. The steps of the method or algorithm disclosed herein may be embodied in processor executable instructions that may reside on a non-transient computer-readable or processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium that can be accessed by a computer or processor. By way of example but not limitation, such non-transient computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage device, disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks typically reproduce data magnetically, and optical disks reproduce data optically using lasers. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable storage medium and / or a computer-readable storage medium, thereby being incorporated into a computer program product.

[0132] Although this patent document contains many details, these details should not be understood as limitations on the scope of any disclosed technology or content that may be protected by the claims, but should be understood as descriptions of features of specific embodiments that may be specific to specific technologies. Certain features described in the context of separate embodiments in this patent document may also be implemented in a single embodiment in combination. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any sub-combination. In addition, although features may be described above as working in certain combinations and even initially protected by the claims, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0133] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0134] Only a few implementations and examples are described, but other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

[0135]

[0136]

[0137] Table 1

Claims

1. A computer-implemented method for classifying the technical skills of a surgeon performing a surgical procedure, the method comprising: receiving a tool motion trajectory comprising a sequence of detected tool motions of a surgeon performing a surgical procedure using a surgical tool; generating a sequence of multi-channel feature matrices to mathematically represent the motion trajectory; as well as The sequence of multi-channel feature matrices is processed using a deep learning model to generate a skill classification for the surgeon performing the surgical procedure, wherein the deep learning model has been trained to identify and focus on a subset of tool motions in the sequence of detected tool motions that are most relevant to the skill classification.

2. The computer-implemented method of claim 1 , wherein: The deep learning model is a transformer model, and wherein using the deep learning model to process the sequence of multi-channel feature matrices comprises: performing a one-dimensional (1D) convolution operation on the sequence of multi-channel feature matrices by convolving each multi-channel feature matrix within the sequence of multi-channel feature matrices with a kernel of a predetermined time length to generate a context-aware multi-channel feature representation of the multi-channel feature matrix; and The sequence of context-aware multi-channel feature representations of the motion trajectory is processed by the transformer model to generate the skill classification.

3. The computer-implemented method of claim 2, wherein: The converter model identifies and focuses on the subset of tool motions that are most relevant to the skill classification by using a self-attention technique.

4. The computer-implemented method of claim 2, wherein: Convolving each multi-channel feature matrix with the kernel involves convolving each channel of the multi-channel feature matrix with the kernel respectively.

5. The computer-implemented method of claim 2, wherein the 1D convolution operation compares the multi-channel feature matrix at a given time step with a plurality of adjacent time steps before and after the given time step; and The context-aware multi-channel feature representation embeds a certain amount of learned relations into the multiple adjacent time steps before and after the given time step.

6. The computer-implemented method of claim 1, wherein the motion trajectory is generated based on a sequence of positions of the tool detected within a sequence of video frames captured at a set of time steps; and wherein each multi-channel feature matrix within the sequence of multi-channel feature matrices is generated at a corresponding time step in the set of time steps.

7. The computer-implemented method of claim 6, wherein: The multi-channel feature matrix is ​​composed of at least the following signal channels: Time step; the (X, Y) coordinates of the detected tool position within the corresponding video frame detected at the time step; and A size of a bounding box of the detected tool within the corresponding video frame detected at the time step.

8. The computer-implemented method of claim 7, wherein: The multi-channel feature matrix also includes a temporal mask channel indicating presence / absence of the tool in the corresponding video frame.

9. A surgeon skill classification system comprising: one or more processors; a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the system to: receiving a motion trajectory comprising a sequence of detected tool motions of a surgeon performing a surgical procedure using a surgical tool; generating a sequence of multi-channel feature matrices to mathematically represent the motion trajectory; as well as The sequence of multi-channel feature matrices is processed using a deep learning model to generate a skill classification for the surgeon performing the surgical procedure, wherein the deep learning model has been trained to identify and focus on a subset of tool motions in the sequence of detected tool motions that are most relevant to the skill classification.

10. The surgeon skill classification system according to claim 9, wherein: The deep learning model is a transformer model, and wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to use the deep learning model by: performing a one-dimensional (1D) convolution operation on the sequence of multi-channel feature matrices by convolving each multi-channel feature matrix within the sequence of multi-channel feature matrices with a kernel of a predetermined time length to generate a context-aware multi-channel feature representation of the multi-channel feature matrix; and The sequence of context-aware multi-channel feature representations of the motion trajectory is processed using the transformer model to generate the skill classification.

11. A surgeon skill classification system according to claim 10, wherein the memory also stores instructions that, when executed by the one or more processors, cause the system to use the converter model to identify and focus on the subset of tool motions by using a self-attention technique.

12. The surgeon skill classification system according to claim 10, wherein: The memory also stores instructions that, when executed by the one or more processors, cause the system to convolve each multi-channel feature matrix with the kernel by respectively convolving each channel of the multi-channel feature matrix with the kernel.

13. The surgeon skill classification system according to claim 10, wherein the 1D convolution operation compares the multi-channel feature matrix at a given time step with a plurality of adjacent time steps before and after the given time step; and The context-aware multi-channel feature representation embeds a certain amount of learned relations into the multiple adjacent time steps before and after the given time step.

14. The surgeon skill classification system according to claim 9, wherein the motion trajectory is generated based on a sequence of positions of the tool detected within a sequence of video frames captured at a set of time steps; and wherein each multi-channel feature matrix within the sequence of multi-channel feature matrices is generated at a corresponding time step in the set of time steps.

15. The surgeon skill classification system of claim 14, wherein: The multi-channel feature matrix is ​​composed of some or all of the following signal channels: Time step; the (X, Y) coordinates of the detected tool position within the corresponding video frame detected at the time step; a size of a bounding box of the detected tool within the corresponding video frame detected at the time step; as well as A temporal mask indicating the presence / absence of the tool in the corresponding video frame.

16. A computer-implemented method for classifying the technical skills of a surgeon performing one or more surgical tasks, the method comprising: receiving a motion trajectory comprising a sequence of detected tool motions generated based on a sequence of video frames capturing a surgeon performing a surgical task with a surgical tool; Decomposing the motion trajectory into a 2D coordinate sequence in the 2D pixel space of the video frame; calculating a set of tool motion metrics characterizing the detected tool motion based on the sequence of 2D coordinates and changes in the sequence of 2D coordinates over time; as well as The set of calculated tool motion metrics is processed using a deep learning model to generate a skill classification for the surgeon performing the surgical task, wherein the deep learning model has been trained.

17. The computer-implemented method of claim 16, wherein: The deep learning model is a random forest model, and wherein using the random forest model to process the set of tool motion metrics comprises: applying a plurality of trained decision tree models to the set of calculated tool motion metrics to generate a plurality of class predictions for a high technical skill class or a low technical skill class for the surgeon; and An overall skill classification for the surgeon is generated based on a majority vote of the high-skill category or the low-skill category in the plurality of predictions.

18. The computer-implemented method of claim 17, wherein: Each decision tree model in the plurality of trained decision tree models generates a corresponding class prediction based on a randomly selected subset of the set of calculated tool motion metrics.

19. The computer-implemented method of claim 16, wherein: The set of calculated tool motion metrics includes some or all of the following metrics: distance; speed; Acceleration; Jerk; Curvature; Tortuosity; Rotation angle; Movement ratio.

20. The computer-implemented method of claim 16, wherein: Calculating a tool motion metric in the set of tool motion metrics comprises: calculating a value of the tool motion metric at each time step corresponding to each video frame in the sequence of video frames; and The tool motion metric is calculated by averaging a set of calculated values ​​of the tool motion metric.

Images