Operation video behavior triple recognition method and system based on multi-task query

Abstract

The invention provides an operation video behavior triple recognition method based on multi-task query, and the method comprises the steps: extracting the image features of a current video frame; constructing a plurality of groups of task-oriented learnable query vectors which respectively correspond to three sub-tasks; based on an original prediction vector of a previous frame of main task, screening a highest confidence vector and converting the highest confidence vector into a prior query vector; fusing the prior query vector with the current frame subtask query vector to obtain a previous frame guided subtask query vector; through a multi-head attention mechanism, enabling the subtask query vectors guided by the previous frame to interact with the image features of the current frame, and generating task specificity vectors; and inputting the task specificity vectors into a decoder and a classification head to obtain classification results of the main task and the sub-tasks. Therefore, the operation video triple recognition precision and the time sequence stability are improved, class imbalance is relieved, the model interpretability is enhanced, and clinical requirements are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of surgical behavior recognition, and in particular to a surgical video behavior triple recognition method and system based on multi-task query. BACKGROUND

[0002] Surgical video action triple recognition refers to accurately recognizing the combination mode of "instrument-action-target" in the surgical process, and is a core technology for intelligent surgical analysis and intraoperative decision support. By analyzing the visual features in the video frame, the structured understanding of the surgical operation is realized, which provides key technical support for clinical teaching, surgical quality evaluation and other scenes.

[0003] With the popularity of minimally invasive surgery and the increasing demand for medical intelligence, surgical video data has shown explosive growth, and higher requirements have been put forward for the accuracy, interpretability and timing stability of triple recognition. The existing technology is mostly based on the Transformer architecture, and focuses on key areas with the help of attention mechanism, but in complex surgical scenes, it still faces the challenge of balancing recognition performance and actual application needs.

[0004] Specifically, the existing technology has two significant defects: first, the black box nature of the attention mechanism makes the model decision-making process opaque, making it difficult to visualize key attention areas and reducing the credibility in clinical applications; second, the action category distribution in the surgical video is imbalanced, the recognition accuracy of the few-sample category is low, and the inter-frame action semantic correlation modeling is insufficient, which easily leads to recognition result jumping and affects the timing stability. These problems restrict the landing application of surgical video triple recognition technology in actual medical scenes, and urgent technological breakthroughs are needed. SUMMARY

[0005] To solve the above problems, the present application provides a surgical video behavior triple recognition method and system based on multi-task query, which realizes surgical behavior understanding through multi-object collaborative recognition.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme: In a first aspect, the present application provides a surgical video behavior triple recognition method based on multi-task query, comprising: extracting current video frame image features; constructing multiple sets of task-oriented learnable query vectors, each corresponding to three sub-tasks; the sub-tasks include surgical instrument category recognition, surgical action category recognition and surgical target category recognition; based on the main task original prediction vector obtained from the previous frame, filtering the highest confidence vector and converting it into a prior query vector; fusing the prior query vector with the current frame sub-task query vector to obtain the sub-task query vector guided by the previous frame; The feature interaction module is configured to make the sub-task query vectors guided by the previous frame interact with the image features of the current frame through a multi-head attention mechanism to generate task-specific vectors. The classification module is configured to input the task-specific vectors into a decoder and a classification head to obtain classification results of the main task and the sub-tasks.

[0007] In a second aspect, the present application provides a surgical video behavior triple identification system based on multi-task query, comprising: The original feature acquisition module is configured to extract image features of a current video frame. The query vector construction module is configured to construct multiple sets of task-oriented learnable query vectors corresponding to three sub-tasks, respectively. The previous frame guiding module is configured to filter the highest confidence vector based on the main task original prediction vector obtained from the previous frame and convert it into a prior query vector. The feature interaction module is configured to make the sub-task query vectors guided by the previous frame interact with the image features of the current frame through a multi-head attention mechanism to generate task-specific vectors. The classification module is configured to input the task-specific vectors into a decoder and a classification head to obtain classification results of the main task and the sub-tasks.

[0008] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the surgical video behavior triple identification method based on multi-task query of the first aspect.

[0009] In a fourth aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the surgical video behavior triple identification method based on multi-task query of the first aspect when executing the program.

[0010] Compared with the prior art, the present application has the following advantages: (1) The present application introduces three small-class task guiding mechanisms that are structurally separated, and uses learnable task-oriented query design to improve the model's interpretability. The original surgical action triple label is disassembled into three sub-tasks, and the classification and query mechanisms are established respectively to alleviate the class imbalance problem, and a lightweight inter-frame guiding mechanism is combined to improve the model's ability to model the time sequence structure and classification stability. The method proposed in the present application improves the model's performance while also improving the model's interpretability and credibility.

[0011] (2) The model and method provided by the application have strong advantages in modeling the structure of video frame images in the Transformer model, and through the task-guided query mechanism and inter-frame guided fusion design, the problems of triplet class imbalance and attention distribution opacity can be effectively alleviated, and the method has high precision, high interpretability and certain temporal stability, and is suitable for multi-label surgical action recognition and analysis tasks.

[0012] Advantages of the additional aspects of the application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0013] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the application, and together with the description of the exemplary embodiments of the application and their description serve to explain the application, and do not constitute limitations on the application.

[0014] Figure 1 The main flowchart of the surgical video behavior triplet recognition method based on multi-task query provided for the embodiment of the application is shown in the figure. Figure 2 The model block diagram of the surgical video behavior triplet recognition method based on multi-task query provided for the embodiment of the application is shown in the figure. Figure 3 The attention weight visualization diagram when only the main task query module is added for the embodiment of the application is shown in the figure, (a) is the original video frame image, and (b) is the attention heat map when only the main task query module is added. Figure 4 The attention weight visualization diagram when all the query modules are provided for the embodiment of the application is shown in the figure, (a) is the original video frame image, (b) is the attention heat map when all the query modules are provided, (c) is the attention heat map of the surgical instrument sub-task oriented query module, (d) is the attention heat map of the surgical action sub-task oriented query module, and (e) is the attention heat map of the surgical target sub-task oriented query module. DETAILED DESCRIPTION

[0015] The application will be further described below in combination with the drawings and embodiments.

[0016] Explanation of technical terms: Multi-task query refers to introducing multiple query vector groups with different semantic guiding effects in the Transformer structure, and each group of queries corresponds to a task, and can perform attention aggregation on image features in parallel.

[0017] Embodiment one As Figure 1As shown, the embodiment discloses a surgical video behavior triple recognition method based on multi-task query, comprising the following steps: S1: extracting current video frame image features; S2: constructing multiple sets of task-oriented learnable query vectors, respectively corresponding to three sub-tasks; the sub-tasks include surgical instrument class recognition, surgical action class recognition, and surgical target class recognition; S3: based on the main task original prediction vector obtained from the previous frame, screening the highest confidence vector and converting it into a prior query vector; fusing the prior query vector with the current frame sub-task query vector to obtain a previous frame guided sub-task query vector; S4: through the multi-head attention mechanism, the previous frame guided sub-task query vector is respectively interacted with the current frame image features to generate task-specific vectors; S5: inputting the task-specific vectors into the decoder and classification head to obtain the classification results of the main task and the sub-tasks.

[0018] Next, combined with Figure 1 , a surgical video behavior triple recognition method based on multi-task query is disclosed in detail.

[0019] The embodiment is based on the multi-task query mechanism and introduces inter-frame guidance to realize surgical video triple recognition and classification. The model adopted includes Swin Transformer-Tiny, decoder, and classification head. Among them, Swin Transformer-Tiny is used as the backbone network to encode the video frame image, and then multiple sets of task-oriented learnable query vectors are introduced and applied to the parallel modeling of the main task (original action triple class recognition task) and three sub-tasks (surgical instrument, surgical action, and surgical target class recognition task). By splitting the large class task into small class sub-tasks to assist the main task, the class imbalance problem is alleviated.

[0020] I. Image feature extraction The model architecture diagram of the embodiment is as Figure 2 shown. Swin Transformer-Tiny (Swin-T) is used as the backbone network, which is responsible for encoding the input video frame image and converting the original image into patch-level features. The specific process is as follows: After the input video frame is extracted by the multi-layer features of Swin-T, the image features with a dimension of are generated . Among them, the video frame includes the current frame and the previous frame ; is the batch size, is the patch number, To feature dimension.

[0021] The image features include global and local semantic information of the image, which is the basis for the subsequent task-oriented query mechanism and provides unified feature support for multi-task parallel modeling.

[0022] II. Task-oriented query mechanism This embodiment realizes parallel modeling of main tasks and subtasks by introducing multiple sets of task-oriented learnable query vectors, while enhancing the semantic focusing ability and interpretability of the model on key areas.

[0023] Specifically, three sets of learnable query vectors are introduced, corresponding to three subtasks: surgical instrument, surgical action, and surgical target classification identification tasks. The category information of surgical instruments, actions, and targets is the core component of the main task of surgical action triplets.

[0024] As an implementation, multiple task-oriented learnable query vectors are applied to the main task of original 100 triplet category classification and the independent modeling and classification of three types of small sub-class tasks split from the main task, i.e., 6 types of surgical instrument categories, 10 types of surgical action categories, and 15 types of surgical target categories.

[0025] Each set of query vectors has task specificity and automatically focuses on relevant areas in image features according to its task semantics (e.g., instrument query focuses on instrument position, action query focuses on action area).

[0026] It should be understood that the learnable query vector is essentially a set of predefined trainable parameters, which are randomly assigned during initialization but learn the task focus during training, such as instrument query learning instrument feature pattern and action query learning action feature pattern. It does not directly define the physical position, but through attention mechanism, it automatically finds the relevant areas in the image and finally realizes task specificity, i.e., different queries adapt to different subtask requirements.

[0027] Among them, the main task is the overall goal, and the subtask is the local analysis dimension after decomposition, i.e., the main task ensures overall recognition, and the subtask assists understanding from the local, both grasping the overall situation and details, making the model understand the complex surgical scene more comprehensively and accurately, and realizing multi-dimensional complementation.

[0028] III. Pre-frame guidance To enhance the continuity of action semantics between video frames, a lightweight pre-frame guidance mechanism is introduced to assist the initialization of the current frame main task query based on the prediction results of the previous frame, and a lightweight Top-K guided query update module (Top-K Guided Query Update, abbreviated as TKQ) is proposed.

[0029] Specifically, the previous video frame is input into the backbone network and the decoder to obtain the original predicted vector logits vector of its classification output, with a dimension of . This vector is the original predicted output (without passing through the softmax activation) of the previous frame's main task query after passing through the classification head (MLP), directly reflecting the prediction confidence of the previous frame's triple category.

[0030] Sort the logits vector of the previous frame, extract the top K category indices with the highest confidence, and extract the corresponding query vector set from the query results retained in the previous frame, which is projected to the dimension required by the current decoder through a linear transformation ; Among them, the top K main task category indices with the highest confidence in the logits vector of the previous frame naturally carry the semantic association of the sub-task dimension. This association is determined by the triple logic of video action semantics, making the high-confidence predictions of the top K main task queries necessarily correspond to the key semantic fragments of the sub-task queries in the instrument, verb, and target branches, achieving the forced binding of the main task query and the sub-task query.

[0031] Through residual weighted fusion, fuse the query vectors of the instrument, verb, and target task branches in the previous frame with the query vectors of the corresponding branches in the current frame to complete the initialization, thereby generating a query vector guided by the previous frame with semantic continuity and enhancing the inter-frame semantic consistency: ; Among them, represents the query after completing the initialization, represents the query vector of the current frame, represents the query of the previous frame, represents the fusion weight set empirically.

[0032] It should be noted that these sub-task branches such as instrument, verb, and target do not exist independently of the main task. They are the structural decomposition of the main task semantics. The action semantics of the main task are inherently composed of the triple logic of instrument, verb, and target, and the query vectors of the sub-task branches naturally carry the semantic fragments of the main task. When the semantic information of the previous frame's sub-task branches is transmitted to the corresponding branches of the current frame through residual weighted fusion, it is essentially piecing together the semantic puzzle of the main task block by block. The precise alignment and fusion of the sub-task dimension can ensure that when the main task semantics are inherited between frames, both the integrity of the triple logic is retained, and through fine-grained branch collaboration, the main task query vector inherits the semantic inertia of the previous frame.

[0033] Such as Figure 2As shown in the figure, the query vectors of each task in the previous frame and the corresponding task query vectors of the current frame are input into the TKQ module for previous frame guidance. Using the task features learned in the previous frame, the initial state of the corresponding task query in the current frame is adjusted, that is, analyzing the current frame with historical experience to make the inter-frame semantics more coherent, so as to obtain the main task query vector guided by the previous frame.

[0034] In this embodiment, first, the logits vector predicted by the previous frame is used to extract high-confidence class indices and match the corresponding query vectors, and the dimensions are adapted through linear transformation to provide high-quality priors for the initialization of each task query in the current frame; then, through residual weighted fusion, the semantic experience of the previous frame is incorporated into the current frame query, which not only strengthens the coherence of the inter-frame action semantics and effectively avoids jumps caused by single-frame independent recognition, but also optimizes the initial state of the current frame query with historical task features, enabling the model to focus more on the key action regions with inter-frame continuity, greatly improving the temporal stability and accuracy of action recognition in complex surgical scenarios, laying a coherent semantic foundation for multi-task parallel recognition, and helping surgical video analysis to better conform to the actual process logic.

[0035] IV. Attention Interaction and Feature Generation The learnable query vector and the image feature interact through the multi-head attention mechanism (Multi-head Attention).

[0036] Specifically, the query vectors guided by the previous frame , [[ID=​​​​​​​​​​​​​​​​In this embodiment, task-oriented query vectors are independently processed with image features. Dot product similarity is used to accurately capture the semantic association between the task and the image patch, enabling targeted localization of key areas such as surgical instruments, actions, and targets, resulting in more precise feature focusing. The interactively generated specific task representation output, integrating task semantics and image features, provides intermediate features rich in task information for subsequent decoding and classification, ensuring recognition accuracy. Furthermore, attention weights can be used to generate heatmaps, visually presenting the model's focus areas and effectively improving the interpretability of surgical scene recognition. This helps clinicians better understand the recognition process, achieving a balance between accurate recognition and transparent interpretation in complex surgical video analysis.

[0040] V. Multi-task decoder and classification head This embodiment is responsible for decoding and classifying the query outputs of each task to generate the final prediction result. Specifically: The query outputs of the subtasks and the outputs of the main task (benefiting from the supervision signals and feature regularization provided by the three sets of subtask query branches) enter independent decoder layers. After operations such as residual connection (Add & Norm) and feedforward network (FFN), they are input into the corresponding multilayer perceptron (MLP) classification head.

[0041] The main task classification head outputs a logits vector of triplet action recognition results, with dimensions of [missing information]. The subtask classification heads output the classification results for equipment, actions, and targets, respectively, with corresponding dimensions as follows: , , .

[0042] The model employs end-to-end training, using binary cross-entropy as the loss function, and incorporates KL divergence loss terms between consecutive frames to enhance temporal stability. The final output consists of main task results for triplet recognition and subtask results for local classification display.

[0043] VI. Experimental Verification The model and method proposed in this embodiment, based on the strong advantages of the Transformer model in modeling the structure of video frame images, can effectively alleviate the problems of triple class imbalance and opaque attention distribution through the design of task-guided query mechanism and inter-frame guided fusion. It has high accuracy, high interpretability and certain temporal stability, and is suitable for multi-label surgical action recognition and analysis tasks.

[0044] Table 1 Ablation Experiment Results

[0045] This embodiment conducted a systematic comparative experiment on the CholecT45 surgical video triplet action classification dataset. As shown in Table 1 of the results obtained from the ablation experiment, compared with the baseline model, this embodiment improved the accuracy of the triplet classification task by 3.43%.

[0046] The ablation experiments in this embodiment primarily demonstrate the effectiveness of various model variants by assessing the accuracy of prediction results. Five-fold cross-validation is used to obtain the mean of five sets of results as the final evaluation result. This verifies the independent contribution and combined effect of each submodule in the proposed Multi-Query Decoding Head (MQ-DH) module. Table 1 shows the classification accuracy performance on four tasks—Instrument, Verb, Target, and Triplet—under different task-specific query vector combinations, and also shows the TKQ pre-frame guidance mechanism module when all query mechanisms are enabled. The first row shows the baseline model results, which only uses the Swin Transformer backbone network to extract image features, without any query mechanism, directly inputting visual features into the classifier for recognition. Subsequent rows introduce instrument query vectors, verb query vectors, and target query vectors respectively, analyzing their respective improvements to classification performance. The sixth row shows the complete MQ-DH design, where all three task-specific queries are introduced. The seventh row adds an additional fused triplet query vector as an auxiliary supervision structure to evaluate its impact on overall performance. Experimental results show that introducing the fused query did not bring further performance improvement, but rather slightly reduced accuracy and average precision. Therefore, we no longer use this fused query in the final model structure. In addition, all configurations are evaluated using five-fold cross-validation. The last column of the table shows the average results of the triplet classification task in the five-fold experiment, which is more representative and has stronger reference value, especially when facing severe class imbalance.

[0047] By presenting the accuracy results for the triad classification task and its sub-classes, the effectiveness and role of each module in the surgical triad behavior classification task are analyzed through result comparison. The results showing a gradual increase with the addition of the query module indicate that the query mechanism, compared to model variants that directly classify extracted features, better assists the main task in completing the detection and understanding of surgical behaviors. This verifies the shortcomings of the method of directly using visual features to connect the classifier (without the query mechanism) in modeling high-dimensional semantic relationships, and demonstrates the ability of the query mechanism to effectively capture more comprehensive global and semantic information.

[0048] This embodiment improves the accuracy of each task by introducing a task query mechanism and a previous frame guidance module, verifying the effectiveness of this embodiment in handling the class imbalance problem. It also proves that by adding these three subcategories to form a multi-task learning structure, it can, to a certain extent, help the main task acquire semantic shared information to better understand the information of each subcategory in the triad. Furthermore, by using subcategory query assistance, it adjusts and alleviates the problem of prediction bias for the larger category in the original triad main task.

[0049] Meanwhile, such as attention weight visualization analysis Figure 3 and Figure 4 As shown, Figure 3 The subclass task query module has not been added. Figure 4 After adding the sub-task query module, comparing the attention positions of the sub-task queries with the original image reveals that each task-oriented query module has the ability to focus on task-related locations. This verifies that by dividing the main task's triadic classification into three sub-tasks based on the segmentation of surgical instruments, surgical actions, and surgical targets, and creating task-oriented query modules accordingly, the sub-task query modules can effectively focus on the specific information of instruments, actions, and targets, providing effective auxiliary information for the main task. Furthermore, this embodiment uses a visualization heatmap of attention weights to show the detailed attention points of the constructed query modules within the overall model, thus providing a degree of interpretability for the target detection and classification model. Therefore, the task-guided query module can guide attention to key areas such as surgical instruments and surgical targets, thereby improving the model's interpretability.

[0050] This specific embodiment significantly improves the performance of surgical video action triplet recognition through the synergistic effect of multi-task query and previous frame guidance. Specifically, three sets of task-oriented query vectors interact with image features in parallel, accurately focusing on key areas such as instruments, actions, and targets. This not only alleviates the imbalance of the main task categories with the help of sub-tasks but also enhances the model's interpretability through attention heatmaps. The previous frame guidance mechanism integrates historical semantic experience, strengthens the continuity of actions between frames, and avoids recognition jumps. It effectively balances recognition accuracy, temporal stability, and clinical interpretability, adapting to the analytical needs of complex surgical scenarios.

[0051] Example 2 This embodiment provides a surgical video behavior triplet recognition system based on multi-task query, including: The original feature acquisition module is configured to extract image features of the current video frame. The query vector construction module is configured to construct multiple sets of task-oriented learnable query vectors, each corresponding to one of three sub-tasks; the sub-tasks include surgical instrument category recognition, surgical action category recognition, and surgical target category recognition. The previous frame guidance module is configured to, based on the original prediction vector of the main task obtained in the previous frame, filter the vector with the highest confidence and convert it into a prior query vector; and fuse the prior query vector with the sub-task query vector of the current frame to obtain the sub-task query vector guided by the previous frame. The feature interaction module is configured to use a multi-head attention mechanism to enable the subtask query vector guided by the previous frame to interact with the image features of the current frame, thereby generating a task-specific vector. The classification module is configured to input task-specific vectors into the decoder and classification head to obtain classification results for the main task and sub-tasks.

[0052] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the surgical video behavior triplet recognition method based on multi-task query as described in Embodiment 1 above.

[0053] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the surgical video behavior triplet recognition method based on multi-task query as described in Embodiment 1 above.

[0054] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying surgical video behavior triples based on multi-task query, characterized in that, include: Extract the image features of the current video frame; Construct multiple sets of task-oriented learnable query vectors, each corresponding to one of the three subtasks; The sub-tasks include surgical instrument category recognition, surgical action category recognition, and surgical target category recognition; Based on the original prediction vector of the main task obtained from the previous frame, the vector with the highest confidence is selected and converted into a priori query vector; the priori query vector is fused with the subtask query vector of the current frame to obtain the subtask query vector guided by the previous frame. Through a multi-head attention mechanism, the sub-task query vector guided by the previous frame interacts with the image features of the current frame to generate a task-specific vector. The task-specific vector is input into the decoder and the classification head to obtain the classification results of the main task and sub-tasks.

2. The surgical video behavior triplet recognition method based on multi-task query as described in claim 1, characterized in that, The extraction of current video frame image features specifically involves using Swing Transformer-Tiny as the backbone network to encode the input video frame into patch-level features.

3. The surgical video behavior triplet recognition method based on multi-task query as described in claim 1, characterized in that, The step of filtering the highest confidence vectors based on the original prediction vectors of the main task obtained from the previous frame is as follows: sort the original prediction vectors output by the main task in the previous frame and extract the top K category indices with the highest confidence; where K≥1.

4. The surgical video behavior triplet recognition method based on multi-task query as described in claim 1, characterized in that, The fusion specifically employs residual weighted fusion: the prior query vector is weighted and added to the subtask query vector initialized in the current frame according to a preset ratio.

5. The surgical video behavior triplet recognition method based on multi-task query as described in claim 1, characterized in that, The multi-head attention mechanism is a multi-head cross-attention mechanism; The interaction process includes: calculating the dot product similarity between each query vector and the current frame image features independently, and generating attention weights to focus on task-related regions.

6. A surgical video behavior triplet recognition system based on multi-task query, characterized in that, include: The original feature acquisition module is configured to extract image features of the current video frame. The query vector construction module is configured to construct multiple sets of task-oriented learnable query vectors, each corresponding to one of three sub-tasks; the sub-tasks include surgical instrument category recognition, surgical action category recognition, and surgical target category recognition. The previous frame guidance module is configured to, based on the original prediction vector of the main task obtained in the previous frame, filter the vector with the highest confidence and convert it into a prior query vector; and fuse the prior query vector with the sub-task query vector of the current frame to obtain the sub-task query vector guided by the previous frame. The feature interaction module is configured to use a multi-head attention mechanism to enable the subtask query vector guided by the previous frame to interact with the image features of the current frame, thereby generating a task-specific vector. The classification module is configured to input task-specific vectors into the decoder and classification head to obtain classification results for the main task and sub-tasks.

7. The surgical video behavior triplet recognition system based on multi-task query as described in claim 6, characterized in that, The extraction of current video frame image features specifically involves using Swing Transformer-Tiny as the backbone network to encode the input video frame into patch-level features.

8. The surgical video behavior triplet recognition system based on multi-task query as described in claim 6, characterized in that, The step of filtering the highest confidence vectors based on the original prediction vectors of the main task obtained from the previous frame is as follows: sort the original prediction vectors output by the main task in the previous frame and extract the top K category indices with the highest confidence; where K≥1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the surgical video behavior triplet recognition method based on multi-task query as described in any one of claims 1-5.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the surgical video behavior triplet recognition method based on multi-task query as described in any one of claims 1-5.

Images