Online operation stage identification method and system based on double-branch query network

Abstract

The invention provides an online operation stage identification method and system based on a double-branch query network, and belongs to the technical field of computer vision and medical treatment. The method comprises the following steps: acquiring and preprocessing an operation video frame; inputting the preprocessed operation video frames into a trained double-branch query network, wherein the trained double-branch query network extracts initial frame-level features of the operation video frames by using a frame branch module; inputting the initial frame-level features into a dual-scale selector for stage query initialization, and generating stage query of the current video frame; the stage query is input into a state space query module for historical information retrieval, and the stage query is optimized; and according to the optimized stage query and frame-level features, generating an identification result of the operation stage category and the corresponding frame range. The method greatly improves the recognition precision and reasoning speed of the online operation stage, and achieves the precise recognition of the type of the operation stage and the corresponding frame range.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and medical technology, and particularly relates to an online surgical phase recognition method and system based on a dual-branch query network. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Surgical Phase Recognition (SPR) aims to perform temporal segmentation on a complete surgical video and label predefined phase tags for each segment. In recent years, as an important research direction in surgical process analysis, surgical phase recognition has shown significant value in improving surgical quality and optimizing department management efficiency. Offline surgical phase recognition methods mainly rely on the context information of the entire video sequence for postoperative analysis, but this paradigm is difficult to meet the real-time requirements of the surgical process. Therefore, online phase recognition has attracted extensive attention in the research community because it can provide real-time decision support during the surgical process.

[0004] The core challenge of online phase recognition is to predict the surgical phase using only the current and historical frames without access to future information. Existing methods mostly use recurrent neural networks or convolutional neural networks to model temporal information, but they have limitations in capturing long-term dependency relationships. When faced with complex surgical scenarios such as surgical technique differences and frequent noise artifacts, the adaptability of these methods faces severe challenges. Recently, Transformer-based models have shown significant advantages in video content analysis. Given their advantages in capturing long-term dependency relationships and enhancing feature representations, researchers have started to explore applying this paradigm to the online phase recognition task.

[0005] However, directly applying Transformer to the Online Phase Recognition (OPR) task is difficult to achieve ideal results. This is mainly due to the computational burden and inference latency generated by the Transformer network when processing long surgical video sequences. To alleviate this problem, existing research usually divides the long input video sequence into short segments for local temporal modeling, and then captures the long-range dependencies between local information through a global temporal module. However, the local temporal module only focuses on a limited number of frames and lacks a feedback mechanism, resulting in weakened correlations between different local frames and information loss prone to occur when establishing long-term dependencies. Summary of the Invention

[0006] To overcome the deficiencies of the above-mentioned existing technologies, the present invention provides an online surgical stage recognition method and system based on a dual-branch query network. The bidirectional branch query network (B2Q-Net) redefines the OPR task by establishing a bidirectional query mechanism between stage-level features and frame-level features. B2Q-Net integrates historical information during the stage query initialization phase, enabling two-way information flow between the two-level feature maps of the stage and the frame during the iterative optimization process. In addition, by introducing a dual-scale selector (DSS), high-quality stage queries are generated for the current video frame. These stage queries retrieve historical information through the proposed state space query (SSQ) module, which uses learnable tokens as the historical state space to store historical information, greatly improving the recognition accuracy and inference speed.

[0007] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of the present invention provides an online surgical stage recognition method based on a dual-branch query network;

[0009] An online surgical stage recognition method based on a dual-branch query network, comprising:

[0010] Obtain surgical video frames and perform preprocessing;

[0011] Input the preprocessed surgical video frames into a trained dual-branch query network to optimize the stage query;

[0012] Among them, the trained dual-branch query network uses a branch module to extract the initial frame-level features of the surgical video frames; input the initial frame-level features into a dual-scale selector for stage query initialization to generate stage queries for the current video frame; input the stage queries into a state space query module for historical information retrieval to optimize the stage queries;

[0013] Generate recognition results of surgical stage categories and corresponding frame ranges based on the optimized stage queries and frame-level features.

[0014] As a further technical solution, the preprocessing process includes data augmentation of the surgical video frames, and the data augmentation includes image translation, scaling, random cropping, RGB channel offset, and brightness and contrast adjustment for each frame of the video.

[0015] As a further technical solution, the branch module includes: a frame branch module, a stage branch module, and a cross-branch communication module; the process of using the branch module to extract the initial frame-level features of the surgical video frames is:

[0016] The spatial features and frame-level temporal features of video frames are extracted using a frame branch module. The LSTM network is used to link the independent spatial features of each frame, and the gated mechanism is used to fuse the historical temporal information and the current frame features to obtain hidden state features. The frame-level temporal features are generated by aggregating the hidden state features.

[0017] The stage branch module captures the inter-stage dependencies through learnable queries that fuse historical and current information, and generates stage features.

[0018] A cross-branch communication module is used to establish bidirectional information interaction between the frame-level temporal features and the stage features, and generate the initial frame-level features of the surgical video frames.

[0019] As a further technical solution, the dual-scale selector consists of a global head and a local head, and each head contains multiple layers of temporal convolutions.

[0020] As a further technical solution, the process of inputting the initial frame-level features into the dual-scale selector for stage query initialization to generate the stage query of the current video frame is as follows:

[0021] Obtain the outputs of the global head and the local head. The output of the global head is used to evaluate the global response value of the frame-level features to all stages, and the output of the local head is used to obtain the local response value of the frame-level features within the temporal window centered on the current frame.

[0022] Fuse the global response value and the local response value, calculate the final score sequence of each window and sort it:

[0023] According to the sorted score sequence, extract the corresponding features from the initial frame-level features to generate the stage query of the current video frame.

[0024] As a further technical solution, the process of inputting the stage query into the state space query module for historical information retrieval to optimize the stage query is as follows:

[0025] Define a state space set initialized to all zeros, and use learnable embeddings as position encoding;

[0026] Use the self-attention layer to capture the internal correlations of the updated state space of the video frames in the (b - 1)th batch and the stage query of the video frames in the bth batch respectively, and then use the cross-attention layer to enable the current stage query to retrieve the key values of the state space.

[0027] Generate attention weights to enhance the stage query feature space and the state space, and finally generate the optimized state space and stage query through a feed-forward neural network.

[0028] As a further technical solution, the method also includes the design of a loss function, and the loss function includes frame-level cross-entropy loss, frame-level smooth loss, stage-level cross-entropy loss, and attention map loss.

[0029] The second aspect of the present invention provides an online surgical stage recognition system based on a dual-branch query network.

[0030] An online surgical stage recognition system based on a dual-branch query network includes:

[0031] A preprocessing module, configured to: obtain surgical video frames and perform preprocessing;

[0032] A stage query optimization module, configured to: input the preprocessed surgical video frames into a trained dual-branch query network to optimize the stage query;

[0033] Wherein, the trained dual-branch query network extracts initial frame-level features of the surgical video frames by using a branch module; inputs the initial frame-level features into a dual-scale selector for stage query initialization to generate a stage query for the current video frame; inputs the stage query into a state space query module for historical information retrieval to optimize the stage query;

[0034] A surgical stage recognition module, configured to: generate an identification result of the surgical stage category and the corresponding frame range according to the optimized stage query and the frame-level features.

[0035] The third aspect of the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in an online surgical stage recognition method based on a dual-branch query network as described in the first aspect of the present invention are implemented.

[0036] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in an online surgical stage recognition method based on a dual-branch query network as described in the first aspect of the present invention are implemented.

[0037] The above one or more technical solutions have the following beneficial effects:

[0038] The present invention proposes a bidirectional branch query network (B2Q-Net), which reconstructs the frame-by-frame classification in the online stage recognition task into a bidirectional query mechanism between stage-level features and frame-level features, and realizes the collaborative enhancement of the two through iterative optimization.

[0039] The present invention adopts a dual-scale selector (DSS) to generate high-quality stage queries through multi-scale feature selection, balancing global and local contexts. By using a state space query (SSQ) module, historical information is stored through learnable tokens and long-term dependencies are captured, enhancing the adaptability of the model in complex surgical scenarios, and greatly improving the recognition accuracy and inference speed for online surgical stages.

[0040] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0042] Figure 1 It is a flowchart of the method for the first embodiment.

[0043] Figure 2 It is a schematic diagram of the two-way branch query network architecture for the first embodiment.

[0044] Figure 3 It is a schematic diagram of the dual-scale selector (DSS) for the first embodiment.

[0045] Figure 4 It is a schematic diagram of the state space query module (SSQ) for the first embodiment.

[0046] Figure 5 It is the comparison result of the method adopted by the present invention with other methods on the dataset in the first embodiment.

[0047] Figure 6 It is the comparison result of the inference time of the method adopted by the present invention with other methods in the first embodiment.

[0048] Figure 7 It is a system structure diagram for the second embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0051] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0052] The present invention first extracts initial frame-level features through a branch module, then a dual-scale selector extracts specific features therefrom as the stage query corresponding to the current video frame, and retrieves historical information from the historical state space through a state space query module. Finally, through frame-level and stage-level temporal modeling, the dual-level feature representation and its correlation are iteratively enhanced to achieve accurate identification of surgical stage categories and corresponding frame ranges.

[0053] Embodiment 1

[0054] This embodiment discloses an online surgical stage recognition method based on a dual-branch query network. For a given batch of surgical video frame inputs (where T is the number of frames), the goal of the online stage recognition (OPR) task is to classify the video frames according to the current batch X b and historical batches into where each frame label corresponds to a predefined surgical stage category.

[0055] As Figure 1 and Figure 2 shown, an online surgical stage recognition method based on a dual-branch query network includes:

[0056] Step S1, obtaining surgical video frames and performing preprocessing.

[0057] The preprocessing process includes performing data augmentation on the obtained surgical video frames where the data augmentation includes performing image translation, scaling, random cropping, RGB channel offset, and brightness contrast adjustment on each frame of the surgical video.

[0058] Step S2, inputting the preprocessed surgical video frames into a trained dual-branch query network to optimize the stage query.

[0059] Step S21, the trained dual-branch query network uses a branch module to extract initial frame-level features of the surgical video frames; the branch module includes a frame branch module, a stage branch module, and a cross-branch communication module.

[0060] (1) Among them, the frame branch module is responsible for modeling spatial features and capturing frame-level temporal correlations. First, the input surgical video frames are encoded into high-dimensional features:

[0061] F S =[E(x1), E(x2), …, E(x T )] T ;

[0062] In the formula is the single-frame feature embedding; F SIs the encoded high-dimensional feature. Considering the potential negative impact of Batch Normalization (BN) in time series modeling, the ConvNeXt network without BN is used as the spatial feature extraction backbone in this embodiment.

[0063] To enhance the time series perception ability and integrate historical batch information, the LSTM network is used to link the independent spatial features of each frame. The historical time series information and the current frame features are fused through the gating mechanism as follows:

[0064]

[0065] Where i t , f t and o t represent the input gate, forget gate, and output gate respectively, Λ is the sigmoid activation function, h t-1 is the hidden state at the previous moment; W i , W f , W o are the weight matrices corresponding to the input gate, forget gate, and output gate respectively ; U i , U f , U o are the weight matrices corresponding to h t-1 in the input gate, forget gate, and output gate respectively; b i , b f , b o are the bias matrices in the input gate, forget gate, and output gate respectively; is the feature encoded by ConvNeXt.

[0066] The cell state C t and the hidden state h t are updated as follows:

[0067]

[0068] h t = o t ⊙ tanh(C t );

[0069] In the formula, is the intermediate term of the cell state; U c is the weight matrix of h t ; W c is the weight matrix of ; b c is the bias matrix; tanh is the hyperbolic tangent activation function, and ⊙ represents element-wise multiplication.

[0070] Finally, all hidden states are aggregated to obtain the enhanced frame-level feature:

[0071] F f = [h1, h2, …, h T T ;

[0072] where Initialize the stage query of the input dual-scale selector.

[0073] The frame-level features interact with the stage-level features through the cross-branch communication module B(·, ·, ·). To reduce the computational complexity, temporal downsampling D(·) is performed before the module, and the features are reconstructed through temporal upsampling U(·) after processing:

[0074]

[0075] In the formula, is the frame-level feature after passing through the cross-branch communication module; A p2f is the attention score output by the cross-branch module; U is the upsampling operation; B is the cross-branch communication module; D is the downsampling operation; F p is the stage-level feature; F f1 is the frame-level feature after passing through the projection layer; is the downsampled feature, and the attention weight A p2f is used to enhance the frame-level feature representation.

[0076] (2) The stage branch module captures the inter-stage dependencies through a learnable query that fuses historical and current information. The input query P I is encoded by the L-layer Transformer:

[0077]

[0078] In the formula, is the stage feature after the self-attention operation of the stage feature of the (l-1)-th layer; is the stage feature after the cross-attention operation; is the Bernoulli sampling binary mask, LN represents layer normalization, MSA and MCA represent multi-head self-attention and cross-attention respectively; FFN is the feed-forward neural network.

[0079] The stage features interact with the frame features through the cross-branch communication module, and the attention weight characterizes the frame range corresponding to each stage feature. Finally, the stage feature representation is gradually optimized through multi-layer stacking.

[0080] (3) The cross-branch communication module (CBC) realizes the information interaction between the two branches through cross-attention. Taking the CBC module in the frame branch as an example:

[0081] ​

[0082] In the formula, σ is the softmax function; W K is the weight matrix corresponding to the key value; W Q is the weight matrix corresponding to the query; d k is the scaling factor; W V is the weight matrix corresponding to the value; W1 is the weight matrix corresponding to the linear layer; b1 is the bias matrix corresponding to the linear layer; Cat is the concatenation operation; the CBC module in the phase branch realizes the reverse information transmission in a symmetric manner.

[0083] Step S22, input the initial frame-level feature into the dual-scale selector for stage query initialization to generate the stage query of the current video frame;

[0084] Since the learnable queries in the Transformer-based detection model are mainly used to provide position constraints for feature pooling, some detection model variants initialize the object queries by selecting the positions corresponding to the Top-K encoded features with the highest confidence scores. This method aims to associate each query with a specific spatial position (the confidence score represents the probability that the position contains a foreground object). In the OPR task, directly selecting the Top-K frame-level features with the highest classification confidence to initialize the stage query may cause the queries to concentrate on similar frame-level features with continuously high confidence, thus ignoring the temporal diversity of the surgical video sequence. Therefore, in this embodiment, a dual-scale selector that balances global and local contexts is adopted, and frame-level features are selected through relative probability modeling - this mechanism determines the possibility of each frame feature as a stage query by evaluating the mutual relationship between features within a certain period.

[0085] Combined with Figure 3 , the dual-scale selector consists of a global head and a local head, and each head contains multiple layers of temporal convolutions. Given the frame-level feature First, obtain the output by the global head and the (N w is the number of intervals of the local window). Among them, F g describes the global response value of each frame feature to all stages, and F l is used to obtain the conditional probability response P(N w |t), which represents the response value of the frame-level feature within the temporal window centered on time t). Calculate the final score of each window by fusing the global and local response values:

[0086]

[0087] In the formula, represents the global feature from the th frame to the th frame, Denote the local feature centered at t with a window size of N w +1. Denote the relative probability of the feature within the window centered at t as the stage query, and the selection score S at time t t is the average probability in all windows containing this time; for the score sequence after sorting in descending order, select TOP-N q score corresponding index is:

[0088]

[0089] In the formula, Top-N q (·) represents obtaining the index corresponding to the top N q scores; Satisfy is the i-th Nq selection score; finally, extract the corresponding feature from the frame-level feature F f to generate the stage query of the current video frame:

[0090]

[0091] In the formula, P f is the stage query; is the selected frame branch temporal feature; this query will input the state space query module for historical information retrieval.

[0092] Step S23, input the stage query into the state space query module for historical information retrieval to optimize the stage query;

[0093] Due to the frequent artifacts and noises during the surgical process, it is unreliable to construct the stage query only based on the current video frame - its limited time span lacks a broader temporal context and is difficult to capture the overall information of the surgical progress. Inspired by the RNN's use of the state space to capture long temporal continuous dynamic changes, in this embodiment, a state space query module that dynamically integrates historical information is adopted. This module represents the state space of historical information through predefined learnable tokens, and these tokens will be continuously updated during the interaction with the stage query of the current video frame.

[0094] Combine Figure 4 , specifically, first define a set of state space tokens S0 initialized to all zeros, and use learnable embeddings as the position encoding (N s is the number of tokens). As the surgical video progresses, the update process of the state space tokens is as follows: the state space tokens S after the (b - 1)-th update b-1 and the stage query of the b-th batch of video frames Separate inputs are fed into the self-attention layer to capture internal correlations, and then the cross-attention layer is used to enable the query of the current stage to retrieve the key values in the state space. The generated attention weights are used to enhance both the stage query feature space and the state space. Finally, the updated state space token S is generated through the feed-forward neural network FFN respectively. b-1 and the stage query

[0095] Step S3: Generate the recognition results of the surgical stage category and the corresponding frame range according to the optimized stage query and the frame-level features.

[0096] Furthermore, in this embodiment, a loss function is also introduced in the frame branch and the stage branch for joint supervision.

[0097] In the frame branch, frame-level labels similar to traditional OPR tasks are used as supervision signals. The cross-entropy loss is used to narrow the distance between the prediction and the true label, and a smoothing loss is introduced to promote the continuity of adjacent frame predictions:

[0098]

[0099] where is the cross-entropy loss corresponding to the output of the frame branch; is the smoothing loss corresponding to the output of the frame branch; T is the number of video frames, C is the number of categories, w c is the category weight, is the one-hot encoding of the frame-level label , is the predicted probability of the t-th frame for category c.

[0100] In the stage branch, the original frame-level labels are converted into stage-level supervision signals. The stage-level labels include the stage category label and the stage position label During training, the cross-entropy loss is calculated by matching the output of the stage branch and the stage labels:

[0101]

[0102] where is the cross-entropy loss; M is the total number of stages included in the current video frame, is the one-hot encoding of the stage category label; w n is the predefined loss weight for different stages; is the model output.

[0103] Given that the cross-attention graph of the CBC module represents the frame position range corresponding to the stage features, its physical meaning is the same as that of . Therefore, supervision is carried out by calculating the cross-entropy loss between the attention map and the position label.

[0104]

[0105] Wherein, is the loss corresponding to the attention matrix in the cross-branch communication module; is the one-hot encoding of the stage position label, and A t,n is the element of the cross-attention map generated by the CBC module.

[0106] The overall loss function is the weighted sum of the above loss terms:

[0107]

[0108] Wherein and correspond to the cross-attention map losses generated by the CBC module in the frame branch and the stage branch respectively, and λ i is the preset weight coefficient.

[0109] It is also verified by experiments on three public surgical video datasets:

[0110] (1) Cholec80: It contains 80 endoscopic videos of cholecystectomy completed by 13 surgeons, with a frame rate of 25 fps and 7 surgical stages annotated. Following previous studies, the dataset is divided into 40 training videos and 40 test videos.

[0111] (2) M2CAI16: It includes 41 videos of laparoscopic cholecystectomy, with 8 surgical stages annotated. According to the division method of previous work, 27 videos are used for training and 14 for testing.

[0112] (3) AutoLaparo: A laparoscopic hysterectomy dataset, containing 21 videos (7 surgical stages), with an average duration of over 60 minutes and a frame rate of 25 fps. According to the division: 10 for training, 4 for validation, and 7 for testing videos.

[0113] Four widely recognized metrics are used in the experiment to evaluate the online surgical stage recognition performance:

[0114] · Accuracy: It reflects the overall performance of the video sequence and is calculated by averaging the accuracy of single videos

[0115] · Precision: Based on the single-stage prediction results, it measures the accuracy of stage recognition

[0116] · Recall: It evaluates the coverage ability of the model for specific surgical stages

[0117] · Jaccard Index: Calculate the overlap degree between the prediction stage and the ground truth stage

[0118] For fair comparison, all metrics follow the standardized calculation protocol of previous studies. Among them, accuracy reflects the global performance, and the latter three metrics focus on the fine-grained evaluation of specific surgical stages.

[0119] By training B2Q-Net in an end-to-end manner on a single 32G VRAM Tesla V100 GPU, the single-sequence batch size is set to 1×256. The AdamW optimizer is adopted, and the initial learning rate is uniformly set to 1×10-4 (applicable to the Cholec80, M2CAI16, and AutoLaparo datasets). The model is trained for 100 epochs, and a weight decay of 0.01 is applied in each optimization step. To simulate the real online surgical scenario, a sliding window input sequence of 256 frames is adopted in both the training and inference stages. After each video is processed, the state space tokens of the stage branch and the LSTM hidden states of the frame branch will be reset.

[0120] As Figure 5 shown, B2Q-Net has achieved significant improvements on three public surgical datasets: 95.8% (+2.1%) accuracy on Cholec80, 92.6% (+3.4%) accuracy on M2CAI16, and 89.3% (+5.2%) accuracy on AutoLaparo. The continuous improvement in stage-level metrics (precision, recall, Jaccard Index) validates the effectiveness of the model in stage transition modeling and boundary optimization. This advantage stems from the fact that B2Q-Net synchronously perceives stage-level and frame-level features through a dual-branch collaborative mechanism, enhancing the discriminative ability of stage classification.

[0121] Figure 6 Shows the comparison results of the inference speed between the B2Q-Net of the present invention and other methods. It can be seen that the inference speed of B2Q-Net is significantly better than the existing optimal method: it is about 1.7 - 3.2 times faster than LoViT, SKiT, and DACAT. This benefits from using stage queries (instead of video frames) as the input of the Transformer, effectively reducing the computational complexity. The middle three rows show the performance at different input frame numbers without using temporal downsampling: when the input frames increase from 128 to 512, the inference speed only drops from 100 frames per second to 87 frames per second, verifying the improvement of the stage query mechanism on computational efficiency. The bottom three rows show that after using temporal downsampling, the inference speed of 256-frame input increases from 95 frames per second to 106 frames per second.

[0122] Embodiment 2

[0123] This embodiment discloses an online surgical stage recognition system based on a dual-branch query network;

[0124] As Figure 7 shown, an online surgical stage recognition system based on a dual-branch query network includes:

[0125] A preprocessing module, configured to: obtain surgical video frames and perform preprocessing;

[0126] A stage query optimization module, configured to: input the preprocessed surgical video frames into a trained dual-branch query network to optimize stage queries;

[0127] Among them, the trained dual-branch query network uses a branch module to extract initial frame-level features of surgical video frames; inputs the initial frame-level features into a dual-scale selector for stage query initialization to generate stage queries for the current video frame; inputs the stage queries into a state space query module for historical information retrieval to optimize stage queries;

[0128] A surgical stage recognition module, configured to: generate recognition results of surgical stage categories and corresponding frame ranges according to the optimized stage queries and frame-level features.

[0129] Embodiment III

[0130] The purpose of this embodiment is to provide a computer-readable storage medium.

[0131] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in an online surgical stage recognition method based on a dual-branch query network as described in Embodiment 1.

[0132] Embodiment IV

[0133] The purpose of this embodiment is to provide an electronic device.

[0134] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in an online surgical stage recognition method based on a dual-branch query network as described in Embodiment 1.

[0135] The steps involved in the devices in the above Embodiments II, III, and IV correspond to those in Method Embodiment 1. For specific implementation manners, reference may be made to the relevant description part of Embodiment 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0136] Those skilled in the art should understand that the various modules or steps of the present invention described above can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0137] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that, based on the technical solutions of the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts are still within the protection scope of the present invention.

Claims

1. An online surgical stage recognition method based on a dual-branch query network, characterized in that including: Obtain surgical video frames and perform preprocessing; Input the preprocessed surgical video frames into a trained dual-branch query network to obtain the recognition result of the online surgical stage; Among them, the trained dual-branch query network uses a branch module to extract the initial frame-level features of the surgical video frames; input the initial frame-level features into a dual-scale selector for stage query initialization to generate the stage query of the current video frame; input the stage query into a state space query module for historical information retrieval to optimize the stage query; Generate the recognition result of the surgical stage category and the corresponding frame range according to the optimized stage query and frame-level features.

2. The online surgical stage recognition method based on a dual-branch query network according to claim 1, wherein, The preprocessing process includes data augmentation for the surgical video frames, and the data augmentation includes image translation, scaling, random cropping, RGB channel offset, and brightness and contrast adjustment for each frame of the video.

3. The online surgical stage recognition method based on a dual-branch query network according to claim 1, characterized in that, The branch module includes: a frame branch module, a stage branch module, and a cross-branch communication module; the process of using the branch module to extract the initial frame-level features of the surgical video frames is as follows: Use the frame branch module to extract the spatial features and frame-level temporal features of the video frames, use the LSTM network to link the independent spatial features of each frame, fuse the historical temporal information and the current frame features through a gating mechanism to obtain the hidden state features, and generate the frame-level temporal features by aggregating the hidden state features; The stage branch module captures the inter-stage dependencies through a learnable query that fuses historical and current information to generate stage features; Use the cross-branch communication module to establish two-way information interaction between the frame-level temporal features and the stage features to generate the initial frame-level features of the surgical video frames.

4. The online surgical stage recognition method based on a dual-branch query network according to claim 1, wherein, The dual-scale selector consists of a global head and a local head, and each head contains multiple layers of temporal convolution.

5. The online surgical stage recognition method based on a dual-branch query network according to claim 1 or 4, characterized in that, The process of inputting the initial frame-level features into the dual-scale selector for stage query initialization to generate the stage query of the current video frame is as follows: Obtain the outputs of the global head and the local head. The output of the global head is used to evaluate the global response value of the frame-level features to all stages, and the output of the local head is used to obtain the local response value of the frame-level features within the temporal window centered on the current frame; Fuse the global response value and the local response value, calculate the final score sequence of each window and sort it: According to the sorted score sequence, extract the corresponding features from the initial frame-level features to generate the stage query of the current video frame.

6. The online surgical stage recognition method based on a dual-branch query network according to claim 1, characterized in that, The process of inputting the stage query into the state space query module for historical information retrieval to optimize the stage query is as follows: Define a state space set initialized to all zeros and use a learnable embedding as the position encoding; Use the self-attention layer to capture the internal correlations between the updated state space of the (b-1)th batch of video frames and the stage query of the bth batch of video frames respectively, and then use the cross-attention layer to make the current stage query retrieve the key values of the state space; Generate attention weights to enhance the stage query feature space and the state space, and finally generate the optimized state space and stage query through a feed-forward neural network.

7. The online surgical stage recognition method based on a dual-branch query network according to claim 1, wherein It also includes the design of a loss function, and the loss function includes frame-level cross-entropy loss, frame-level smooth loss, stage-level cross-entropy loss, and attention map loss.

8. An online surgical stage recognition system based on a dual-branch query network, characterized in that, including: A preprocessing module configured to: obtain surgical video frames and perform preprocessing; The stage query optimization module is configured to: input the preprocessed surgical video frames into the trained dual-branch query network to optimize the stage query; Among them, the trained dual-branch query network uses a branch module to extract the initial frame-level features of the surgical video frames; input the initial frame-level features into a dual-scale selector for stage query initialization to generate the stage query of the current video frame; input the stage query into a state space query module for historical information retrieval to optimize the stage query; The surgical stage recognition module is configured to: generate the recognition results of the surgical stage categories and the corresponding frame ranges according to the optimized stage query and the frame-level features.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps in an online surgical stage recognition method based on a dual-branch query network according to any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in an online surgical stage recognition method based on a dual-branch query network according to any one of claims 1-7.

Images