Surgical procedure identification methods, media, and surgical robots

Abstract

This application provides a surgical procedure recognition method, medium, and surgical robot. The method includes: acquiring multiple target video frames and multiple target kinematic data corresponding to each of the multiple target video frames, wherein the multiple target video frames include a current video frame and at least one historical video frame; the multiple target video frames include video frames of surgery performed using surgical instruments, and the kinematic data corresponding to each target video frame includes kinematic data of a robotic arm holding the surgical instruments when acquiring the target video frame; inputting the multiple target video frames and the multiple target kinematic data into a pre-trained surgical procedure recognition model, so that the surgical procedure recognition model can recognize the surgical procedure corresponding to the current video frame based on the multiple target video frames and the multiple target kinematic data.

Description

Technical Field

[0001] This application relates to the field of robotic systems, and more particularly to a surgical procedure recognition method, medium, and surgical robot. Background Technology

[0002] Automated surgical procedure recognition refers to the automated identification and monitoring of each step and stage of a surgical procedure. Related technologies utilize surgical procedure recognition models to identify the surgical process within surgical videos. However, surgical scenarios are often complex, with similar visual features across different surgical stages. Furthermore, obstructions are frequently present during surgery, which can lead to lower accuracy in identifying the surgical procedure within surgical videos using these models. Summary of the Invention

[0003] In a first aspect, this application provides a surgical procedure recognition method, the method comprising: acquiring multiple target video frames and multiple target kinematic data corresponding to the multiple target video frames, wherein the multiple target video frames include a current video frame and at least one historical video frame; the multiple target video frames include video frames of surgery performed using surgical instruments, and the kinematic data corresponding to the target video frames includes kinematic data of a robotic arm holding the surgical instruments when acquiring the target video frame; inputting the multiple target video frames and the multiple target kinematic data into a pre-trained surgical procedure recognition model, so that the surgical procedure recognition model can recognize the surgical procedure corresponding to the current video frame based on the multiple target video frames and the multiple target kinematic data.

[0004] Secondly, this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect of this application.

[0005] Thirdly, this application provides a surgical robot, the surgical robot comprising: a first robotic arm for holding surgical instruments; a second robotic arm for holding an endoscope, the endoscope being used to acquire multiple target video frames; the multiple target video frames including video frames of surgery performed through surgical instruments; and a processor for executing the method described in the first aspect of this application.

[0006] In this embodiment of the application, the input data of the surgical procedure recognition model includes target video frames and target kinematic data. The target video frames can provide the visual information required for surgical procedure recognition, and the target kinematic data can provide the motion information of the robotic arm during the operation. The above two types of information can provide a more comprehensive view of the surgical procedure, enabling the surgical procedure recognition model to obtain richer features to recognize the surgical procedure, thereby improving the accuracy of surgical procedure recognition.

[0007] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0009] Figure 1 This is a schematic diagram of the surgical robot according to an embodiment of this application.

[0010] Figure 2 This is a flowchart of the surgical procedure identification method according to an embodiment of this application.

[0011] Figure 3 This is a schematic diagram of the surgical recognition model according to an embodiment of this application.

[0012] Figure 4 This is a general flowchart of an embodiment of this application.

[0013] Figure 5 This is a schematic diagram illustrating the visualization effect of an embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0016] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0017] Figure 1 A schematic diagram of a surgical robot 10 is shown. The surgical robot 10 includes a console 101, a robotic arm system 102, and an imaging system 103. The console 101 includes a display unit for showing the surgical instrument environment, an operation control mechanism, and armrests. The display unit has an observation window (also called a stereoscopic display) for the surgeon to observe. The movements of the operation control mechanism correspond to the movements of the surgical instruments; by controlling the operation control mechanism, the position and posture of the surgical instruments can be adjusted. The armrests are used to support the surgeon's arms. In addition, the console 101 includes other control switches that are easily touched or pressed by hand or foot for various functional operations and human-machine interaction. The console 101 may also include a processor for implementing various control functions during the surgical procedure.

[0018] The robotic arm system 102 includes several robotic arms 102a, each comprising several connecting arms. Adjacent connecting arms move relative to each other with specific degrees of freedom, allowing the end effector of the robotic arm 102a to achieve multiple degrees of freedom (e.g., 7 degrees of freedom, depending on the instrument). Surgical instruments or image acquisition devices are mounted on the end effector of the robotic arm 102a. For ease of distinction, the robotic arm 102a holding surgical instruments can be referred to as the first robotic arm, and the robotic arm 102a holding image acquisition devices such as endoscopes can be referred to as the second robotic arm. The type of surgical instrument can be determined based on the type of surgery in the actual scenario. The image acquisition device can be an endoscope (e.g., a 3D endoscope) used to acquire video during the surgical process. By controlling the robotic arm 102a, the pose of the surgical instruments can be changed to control the surgical instruments to perform specific surgical operations.

[0019] The imaging system 103 includes a display screen, an endoscope controller, system electronics, and an image processor. The endoscope controller can work in conjunction with the operating control mechanisms on the console to control the position and orientation of the endoscope. The endoscope can acquire images (such as surgical images) and send them to the system electronics, whereby the system electronics perform image processing (e.g., decoding). The processed images can be sent to the image processor for further image processing (e.g., noise reduction, contrast enhancement, and sharpness improvement). The image processed by the image processor can be displayed on the display screen and display unit.

[0020] There is a need for surgical procedure identification in many application scenarios. For example, in surgical training, identifying surgical procedures allows for the annotation of surgical videos in a database, facilitating faster video review for surgeons and trainees on specific surgical procedures. Furthermore, in medical billing and surgical report generation, identifying surgical procedures helps achieve highly transparent and accurate billing, as well as the rapid generation of more complete and reliable surgical reports. In addition, automated surgical procedure detection technology can support operating room logistics and operations. For instance, surgeons and administrators can examine the average duration of each surgical step, rather than measuring the total surgical time, allowing for more precise determination of surgical time and prediction of estimated completion times. This enables more accurate and data-driven staffing and resource allocation for the operating room and surgical team compared to existing tools.

[0021] In related technologies, endoscopically captured video can be input into a surgical recognition model to identify the surgical procedures within the video. However, relying solely on video data to identify surgical workflows presents several challenges. For instance, surgical scenarios are often complex, visual features at different stages of the procedure are similar, and obstructions are frequently present during surgery. Therefore, the temporal dependencies and transitions between surgical procedures cannot always be clearly discerned through visual cues alone, especially during lengthy and complex surgeries. These factors may lead to lower accuracy in identifying surgical procedures within surgical videos using surgical workflow recognition models.

[0022] Based on this, this application proposes a surgical procedure identification method, see [link to relevant documentation]. Figure 2 The method includes:

[0023] Step S11: Acquire multiple target video frames and the kinematic data of the target frames corresponding to the multiple target video frames. The multiple target video frames include the current video frame and at least one historical video frame. The multiple target video frames include video frames of surgery performed with surgical instruments. The kinematic data corresponding to the target video frames includes the kinematic data of the robotic arm holding the surgical instruments when acquiring the target video frame.

[0024] Step S12: Input the multi-frame target video frames and the multi-frame target kinematic data into the pre-trained surgical procedure recognition model, so that the surgical procedure recognition model can identify the surgical procedure corresponding to the current video frame based on the multi-frame target video frames and the multi-frame target kinematic data.

[0025] This application embodiment uses both target video frames and target kinematic data as input data for the surgical procedure recognition model, thereby providing a more comprehensive view of the surgical procedure and enabling the model to acquire richer features to identify the surgical procedure, thus improving the accuracy of surgical procedure recognition. The specific implementation of this application embodiment is illustrated below.

[0026] The methods described in this application embodiment can be executed by a processor. In some embodiments, the processor can be integrated into the console 101. The surgical procedure recognition model of this application can be deployed in the console 101 or in the cloud. The processor can acquire target video frames and target kinematic data, and invoke the surgical procedure recognition model to perform surgical procedure recognition.

[0027] In step S11, a surgical instrument can be held by a robotic arm 102a. By controlling the robotic arm 102a, the position of the surgical instrument can be changed, thereby enabling the surgical instrument to perform surgical operations. Additionally, an image acquisition device (such as an endoscope) can be held by another robotic arm 102a. This image acquisition device can acquire multiple target video frames during the surgical operation. In some embodiments, each acquired target video frame is a stereoscopic image, which can be acquired by an image acquisition device (such as a three-dimensional endoscope) including multiple lenses. The three-dimensional endoscope may include a left camera and a right camera, wherein the image acquired by the left camera is called the left-eye image, and the image acquired by the right camera is called the right-eye image. Each acquired target video frame includes one left-eye image and one right-eye image. In other embodiments, each acquired target video frame may also be a two-dimensional graphic, which can be acquired by an image acquisition device (such as a two-dimensional endoscope) including a single lens.

[0028] The multi-frame target video sequence may include the current video frame and at least one historical video frame. The current video frame refers to the video frame from which the surgical procedure needs to be identified. In some embodiments, the current video frame may be a video frame captured in real-time by an image acquisition device. Each of the at least one historical video frame is a video frame acquired before the current video frame.

[0029] In addition, target kinematic data corresponding to each target video frame can be acquired. The target kinematic data corresponding to the target video frame is the kinematic data of the robotic arm 102a holding the surgical instrument when acquiring the target video frame. For example, sensors can be set on the robotic arm 102a (such as on the joints of the robotic arm 102a) to acquire the target kinematic data of the robotic arm 102a. The target kinematic data may include the position data and posture data of the robotic arm 102a. In some embodiments, the target kinematic data acquired by the sensors is the kinematic data of the robotic arm 102a itself. During surgical operations, it is necessary to acquire the kinematic data of the surgical instrument. Therefore, the kinematic data acquired by the sensors can be converted into the kinematic data of the surgical instrument based on the pose mapping relationship between the robotic arm 102a and the surgical instrument.

[0030] In step S12, multiple target video frames and their corresponding target kinematic data can be input into the surgical procedure recognition model to identify the surgical procedure corresponding to the current video frame.

[0031] In some embodiments, see Figure 3 and Figure 4 The surgical procedure recognition model can extract features from multiple target video frames to obtain the visual features of the multiple target video frames, and extract features from multiple target kinematic data to obtain the kinematic features of the multiple target kinematic data.

[0032] In some embodiments, see Figure 3 and Figure 4 The surgical procedure recognition model may include a visual encoder, which can extract the visual features of multiple target video frames. The visual encoder may include a linear mapping layer, which may be a fully connected layer, used to map the raw pixel values ​​of the target video frames into high-dimensional features.

[0033] In other embodiments, see also Figure 3 The surgical procedure recognition model may also include a kinematic encoder, which can extract the kinematic features of the target kinematic data corresponding to multiple target video frames from the target kinematic data corresponding to multiple target video frames through the kinematic encoder.

[0034] In related technologies, surgical procedures in sample video frames need to be manually labeled, and then a visual encoder in the surgical procedure recognition model is trained based on the labeled sample video frames. This method is costly in terms of time and manpower. To reduce training costs, this application proposes a programmatic basic model with a self-supervised learning paradigm, which can be scaled to large and diverse unlabeled datasets. Unlike traditional methods that directly train the visual encoder using supervised learning on the target dataset, the method in this application enhances the model's ability to learn from various unlabeled data and then generalizes its understanding to new, unseen data.

[0035] Specifically, this application employs a training method using a visual occlusion autoencoder (MAE). This is a self-supervised learning technique for computer vision tasks, involving training an autoencoder to reconstruct the original input data from a version of the input data that has been partially occluded. Its goal is to enable the model to learn robust and useful visual data representations by predicting missing pixels.

[0036] The visual encoder of this application can be trained in the following way: acquiring sample video frames, encoding the sample video frames using the visual encoder to obtain the features of the sample video frames, partially masking the features of the sample video frames, and reconstructing the sample video frames based on the masked features using the visual decoder corresponding to the visual encoder, and training the visual encoder and visual decoder based on the differences between the video frames reconstructed by the visual decoder and the sample video frames. See also Figure 4 The sample video frames can include large-scale samples obtained from multiple data sources, including video frames from hospitals, professional communities, institutions, companies and websites.

[0037] To improve efficiency and prevent overfitting on domain-specific datasets, this application employs a Swin-based transformer encoder for visual encoding, and correspondingly, a Swin-based transformer decoder for visual decoding. The Swin transformer utilizes a hierarchical structure, introducing inductive bias to reduce dependence on different datasets. Furthermore, its window-based attention mechanism offers computational advantages, improving the training and inference efficiency of the surgical procedure recognition model.

[0038] After acquiring visual and kinematic features, the visual features of each target video frame and the kinematic features of the corresponding target kinematic data can be fused to obtain the first fused feature for each target video frame. The surgical procedure in the current video frame is then identified based on the first fused features corresponding to multiple target video frames. This process is also known as multimodal semantic alignment.

[0039] For example, the visual features of the first target video frame and the kinematic features of the target kinematic data corresponding to the first target video frame can be fused to obtain the first fused feature corresponding to the first target video frame. Similarly, the visual features of the second target video frame and the kinematic features of the target kinematic data corresponding to the second target video frame can be fused to obtain the first fused feature corresponding to the second target video frame. And so on. Then, based on the first fused feature corresponding to the first target video frame, the first fused feature corresponding to the second target video frame, ..., the first fused feature corresponding to the Nth (N is the total number of target video frames) target video frame, the surgical procedure in the current video frame is identified together.

[0040] After obtaining visual features, this application enriches the acquired visual features by integrating kinematic features, thereby forming multimodal features with richer information. In some embodiments, the visual features of the target video frame and the kinematic features of the target kinematic data corresponding to the target video frame can be mapped to a unified feature space. Then, the mapped visual features and the mapped kinematic features are averaged or weighted to obtain the first fused feature corresponding to the target video frame.

[0041] In other embodiments, the visual features of the target video frame and the kinematic features of the target kinematic data corresponding to the target video frame can be fused based on an attention mechanism to obtain a first intermediate feature. The first intermediate feature and the visual features of the target video frame are then fused to obtain a second intermediate feature. Finally, the first intermediate feature, the second intermediate feature, and the visual features of the target video frame are fused to obtain a first fused feature corresponding to the target video frame. This fusion method, based on an attention mechanism to fuse visual and kinematic features, can fully utilize the information of local visual regions and the physical correlation of kinematic data.

[0042] In an example of fusing visual and kinematic features based on an attention mechanism, the kinematic features obtained by linearly processing the kinematic features of the target kinematic data corresponding to the target video frame can be used as the query, and the features obtained by linearly processing the visual features of the target video frame can be used as the key and value. The correlation between the query and the key is determined, the correlation is normalized, and the value is weighted based on the normalized correlation to obtain the first intermediate feature. The specific implementation is as follows:

[0043]

[0044] E t =Linear(S t +K t )+S t +Kt

[0045] Where t is the timestamp, V t K represents the visual features of the target video frame with timestamp t. t This represents the kinematic features of the target kinematic data corresponding to the target video frame with timestamp t, where T represents matrix transpose, linear represents linear processing, softmax represents normalization processing, and S... t This is the first intermediate feature.

[0046] In some embodiments, before fusing the visual features of the target video frame and the kinematic features of the target kinematic data corresponding to the target video frame based on the attention mechanism to obtain the first intermediate features, the visual features of each image block in the target video frame can be determined, and the visual features of each image block can be weighted based on the weights corresponding to each image block to obtain the visual features of the target video frame.

[0047] The weights corresponding to image patches represent the relevance of that image patch to the surgical procedure. A larger weight indicates a greater relevance (i.e., the importance of the corresponding image patch) to the surgical procedure. By weighting each image patch in the target video frame, the surgical procedure recognition model can focus on key image patches that are highly relevant to the surgical procedure, reducing the interference of image patches with lower relevance to the surgical procedure on the recognition results, thereby improving recognition accuracy.

[0048] In some embodiments, global visual features of multiple target video frames can be obtained, and the first fusion feature corresponding to each target video frame and the global visual feature can be fused to obtain the second fusion feature corresponding to each target video frame. The surgical procedure in the current video frame can be identified based on the second fusion feature corresponding to multiple target video frames.

[0049] The visual features fused with motion features (i.e., visual features generated from each frame of the target video) represent the visual features most directly related to kinematics, such as the type, position, and movement of surgical instruments. Global visual features, on the other hand, refer to the visual features encompassing a video sequence composed of multiple target video frames. They are typically used to characterize the overall scene, long-term dynamics, or overall context of the video sequence, rather than just the details of a single frame or local area. Compared to the visual features of each frame of the target video (i.e., local visual features), global visual features focus more on the macroscopic features of the video sequence, reflecting continuous changes or global trends in visual features. By further fusing global visual features with the first fusion feature, important visual and kinematic features can be preserved as much as possible simultaneously, thereby improving the accuracy of surgical procedure recognition.

[0050] Visual features and kinematic features may originate from different feature spaces; therefore, the surgical procedure recognition model may further include a feature encoder. When fusing global visual features and the first fused feature, the global visual features and the first fused feature corresponding to each target video frame can be input into the feature encoder. The feature encoder can map the global visual features and the first fused feature corresponding to each target video frame to a unified feature space. In some embodiments, the global visual features output by the visual encoder can first undergo pooling processing before being input into the feature encoder. The feature encoder can map the pooled global visual features and the first fused feature output by the kinematic encoder to a unified feature space, and then fuse them to obtain the second fused feature. The mapped global visual features are shown as gray squares in the figure, and the mapped first fused feature is shown as black squares in the figure. By performing pooling processing, the dimensionality of the global visual features can be effectively reduced while retaining important global visual features, thereby reducing the computational load and resource consumption of subsequent processing.

[0051] After obtaining the second fusion features corresponding to each target video frame, the surgical procedure recognition model can identify the surgical procedure in the current video frame based on the second fusion features corresponding to each target video frame.

[0052] In some embodiments, see continue to see Figure 3 and Figure 4The surgical procedure recognition model includes a time-series inference module, which comprises a transformer network and a recurrent neural network. When recognizing the surgical procedure in the current video frame, the second fusion features corresponding to multiple target video frames (where t, t-1, t-2, ..., tl in the figure represent the timestamps of the target video frames to which the second fusion features belong) are input into the transformer network. The transformer network determines the weights corresponding to the multiple target video frames, and then weights the second fusion features corresponding to the multiple target video frames based on these weights, resulting in weighted features corresponding to the multiple target video frames. These weighted features are then input into the recurrent neural network to extract temporal features from the weighted features. Based on the temporal features extracted from the weighted features of the multiple target video frames by the recurrent neural network, the surgical procedure in the current video frame can be recognized. Surgical procedure recognition typically relies on identifying key cues, such as the presence of specific actions or anatomical structures. However, during lengthy and complex surgeries, these crucial clues may be obscured or intermittent, resulting in ambiguous periods in the data stream. This application leverages the sequence reasoning capabilities of the transformer network architecture and the memory processing capabilities of recurrent neural networks for time-series reasoning, enabling the analysis of features from historical video frames preceding the current video frame to accurately predict the surgical procedure within the current video frame.

[0053] The second fusion feature input to the transformer network can be the second fusion feature corresponding to the target video frames within a preset time window. In some embodiments, the length of the preset time window can be the duration corresponding to 128 target video frames. The transformer network has sequence reasoning capabilities and can determine the weights corresponding to multiple target video frames based on the second fusion features corresponding to multiple target video frames. These weights can reflect the importance of each target video frame to the surgical procedure recognition. The larger the weight, the higher the importance of the corresponding target video frame to the surgical procedure recognition. In some embodiments, the transformer network can be a GPT (Generative Pre-trained Transformer) based transformer network. GPT-based transformer networks can effectively handle long-term sequence data. For video data, surgical procedures typically involve multiple stages, steps, and complex dynamic changes, and the relationship between each video frame is not independent. GPT-based transformer networks can effectively capture the temporal relationship between multiple target video frames, helping the surgical procedure recognition model understand information spanning long periods during the surgical process.

[0054] In some embodiments, the importance of each image patch can be visualized based on its corresponding weight. This visualization allows for interpretable surgical procedure recognition results, demonstrating how the surgical procedure recognition model uses image patches from target video frames to infer the surgical procedure in the current video frame. In some embodiments, the importance of each target video frame can also be visualized based on its corresponding weight. This visualization allows for interpretable surgical procedure recognition results, demonstrating how the surgical procedure recognition model uses multiple target video frames to infer the surgical procedure in the current video frame. Figure 5 Visualizations of some embodiments are shown. The figure illustrates three surgical cases, each visualizing the current video frame (timestamp t) and historical video frames (timestamps t-16, t-32, etc.), highlighting the frames with the highest, second-highest, and lowest temporal attention scores (i.e., weights, which can be values ​​between 0 and 1). Color-coded blocks represent attention regions in each image, focusing on key visual cues related to surgical instruments and anatomical structures; blue blocks correspond to the lowest attention scores, and red blocks to the highest. It also demonstrates the effective use of keyframes by this surgical procedure recognition model, which focus on capturing the interaction between surgical instruments and tissue, thereby enabling accurate reasoning about the current surgical procedure.

[0055] Recurrent neural networks (RNNs) possess memory processing capabilities, enabling them to store information from much earlier periods. This allows for the extraction of temporal features spanning longer timeframes from multiple target video frames, which can then be used for surgical procedure identification. For surgeries with long timeframes and complex procedures, employing RNNs can effectively extract temporal features during the surgical process, accurately identifying the surgical steps. In some embodiments, the RNN is a Long Short-Term Memory Network (LSTM).

[0056] See also Figure 3 After the recurrent neural network extracts the temporal features, these features can be input into a fully connected layer to predict the probability p of the surgical procedure in the current video frame being any of the candidate surgical procedures. t The surgical procedure with the highest probability is then selected as the surgical procedure y in the current video frame. t It should be noted that, in Figure 3 In the middle, L c This represents the loss function used when training the surgical procedure recognition model. In this embodiment, the loss function may be the cross-entropy loss function. t kt-1 k t-2 ... k t-l The intermediate hidden features generated after inputting features from different time points into the LSTM are shown. This application embodiment can generate and save the intermediate hidden features corresponding to each video frame. Although multiple LSTMs are shown, they all represent the same LSTM model. The multiple LSTMs in the figure represent the process of inputting features from different time points into the same LSTM model one by one. This application can use a two-stage training method to train the surgical procedure recognition model. In the first stage of training, a large-scale unlabeled data can be used to train the visual encoder in the surgical procedure recognition model. In the second stage of training, sample data labeled with the true values ​​of the surgical procedure can be used to train the remaining parts of the surgical procedure recognition model (such as the kinematic encoder and the visual kinematic alignment module). Figure 3 y in t This represents the true value of the surgical procedure during the second phase of training. In related technologies, visual encoders also require labeled sample data for training, while the visual encoder in this application can be trained based on unlabeled data. Therefore, this application can reduce the amount of labeled data required during training, thereby reducing the cost of the training process.

[0057] This application has the following technical advantages:

[0058] (1) This application pre-trains a visual encoder on a large-scale video dataset to extract reliable visual features, and then performs multimodal fusion and temporal aggregation to achieve surgical procedure recognition. The training method is self-supervised, requires no surgical procedure annotation, and can learn visual features from diverse surgical videos.

[0059] (2) This application proposes a multimodal feature fusion method that uses an attention mechanism to fuse kinematic features and visual features in order to make full use of their complementary information and improve the recognition accuracy of surgical procedures.

[0060] (3) This application proposes to use a GPT-based transformer architecture and LSTM to aggregate short-term and long-term information of historical video frames, thereby improving the time series reasoning ability in the surgical procedure prediction process.

[0061] This application also provides a non-transitory computer-readable storage medium including computer instructions, such as a memory including instructions that can be executed by a processor of a device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0062] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A surgical procedure recognition method, characterized by, The method includes: Acquire multiple target video frames and corresponding kinematic data for each of the multiple target video frames. The multiple target video frames include the current video frame and at least one historical video frame. The multiple target video frames include video frames of surgery performed with surgical instruments. The kinematic data corresponding to each target video frame includes the kinematic data of the robotic arm holding the surgical instruments when acquiring the target video frame. The multi-frame target video frame and the multi-frame target kinematic data are input into a pre-trained surgical procedure recognition model, so that the surgical procedure recognition model can identify the surgical procedure corresponding to the current video frame based on the multi-frame target video frame and the multi-frame target kinematic data.

2. The method of claim 1, wherein, The surgical procedure recognition model identifies the surgical procedure in the current video frame based on the following method: Feature extraction is performed on the multiple target video frames to obtain the visual features of the multiple target video frames; Feature extraction is performed on the multi-frame target kinematic data to obtain the kinematic features of the multi-frame target kinematic data; The visual features of each target video frame and the kinematic features of the target kinematic data corresponding to that target video frame are fused to obtain the first fused feature corresponding to each target video frame. The surgical procedure in the current video frame is identified based on the first fusion feature corresponding to the multi-frame target video frame.

3. The method of claim 2, wherein, The process of fusing the visual features of each target video frame with the kinematic features of the target kinematic data corresponding to that target video frame to obtain the first fused feature corresponding to each target video frame includes: The visual features of the target video frame and the kinematic features of the target kinematic data corresponding to the target video frame are fused based on the attention mechanism to obtain the first intermediate feature; The first intermediate feature and the visual features of the target video frame are fused to obtain the second intermediate feature; The first intermediate feature, the second intermediate feature, and the visual features of the target video frame are fused to obtain the first fused feature corresponding to the target video frame.

4. The method of claim 3, wherein, The first intermediate feature is obtained by fusing the visual features of the target video frame and the kinematic features of the target kinematic data corresponding to the target video frame based on the attention mechanism, including: The features obtained by linearly processing the kinematic features of the target kinematic data corresponding to the target video frame are used as the query, and the features obtained by linearly processing the visual features of the target video frame are used as the key and value. The correlation between the query and the key is determined, the correlation is normalized, and the value is weighted based on the normalized correlation to obtain the first intermediate feature.

5. The method of claim 2, wherein, Before fusing the visual features of the target video frame and the kinematic features of the target kinematic data corresponding to the target video frame based on an attention mechanism to obtain the first intermediate feature, the method further includes: Determine the visual features of each image block in the target video frame; Based on the weights corresponding to each image block, the visual features of each image block are weighted to obtain the visual features of the target video frame.

6. The method of claim 2, wherein, The step of identifying the surgical procedure in the current video frame based on the first fusion features corresponding to the multiple target video frames includes: Obtain the global visual features of the multi-frame target video; The first fusion feature corresponding to each target video frame and the global visual feature are fused to obtain the second fusion feature corresponding to each target video frame. The surgical procedure in the current video frame is identified based on the second fusion feature corresponding to the multi-frame target video frame.

7. The method of claim 6, wherein, The surgical procedure recognition model includes a transformer network and a recurrent neural network; the step of recognizing the surgical procedure in the current video frame based on the second fusion features corresponding to the multi-frame target video frames includes: The second fusion feature corresponding to the multi-frame target video frame is input into the transformer network to determine the weights corresponding to the multi-frame target video frame through the transformer network, and the second fusion feature corresponding to the multi-frame target video frame is weighted based on the weights corresponding to the multi-frame target video frame to obtain the weighted feature corresponding to the multi-frame target video frame. The weighted features corresponding to the multi-frame target video frame are input into the recurrent neural network to extract temporal features from the weighted features corresponding to the multi-frame target video frame through the recurrent neural network. Based on the temporal features extracted from the weighted features corresponding to the multi-frame target video frame by the recurrent neural network, the surgical procedure in the current video frame is identified.

8. The method of claim 7, wherein, The transformer network is a GPT-based transformer network.

9. The method of claim 2, wherein, The surgical procedure recognition model includes a visual encoder, and the visual features of the multi-frame target video are extracted from the multi-frame target video by the visual encoder; wherein, the visual encoder is trained in the following manner: Acquire sample video frames, encode the sample video frames using the visual encoder, and obtain the features of the sample video frames; The features of the sample video frame are partially masked, and the sample video frame is reconstructed based on the masked features by the visual decoder corresponding to the visual encoder. The visual encoder and the visual decoder are trained based on the differences between the video frames reconstructed by the visual decoder and the sample video frames.

10. The method of claim 9, wherein, The visual encoder is a transformer encoder based on the Swing architecture, and the visual decoder is a transformer decoder based on the Swing architecture.

11. A computer readable storage medium having stored thereon computer instructions, wherein, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 10.

12. A surgical robot, characterized by The surgical robot includes: The first robotic arm is used to hold surgical instruments; The second robotic arm is used to hold the endoscope, which is used to acquire multiple target video frames, including video frames of surgery performed with surgical instruments. A processor for performing the method according to any one of claims 1 to 10.

Images