Surgical phase recognition method, apparatus and device based on multi-head attention mechanism

Abstract

The application provides a surgical stage recognition method, device and equipment based on a multi-head attention mechanism. The method is aimed at a surgical video, takes into account the particularity of surgery and relatively more details of the surgical video, and generally has a long surgery time and strong time sequence. Therefore, a modified SwinTransformer network is used instead of a ResNet network to extract features, more detailed features can be extracted, accurate recognition of the surgical stage is realized, and the accuracy is improved by adding a TCN time dependence module. The TCN output and the previous spatial feature matrix are input into a Transformer for encoding, analysis and fusion, the technology for text translation is realized based on video features to "translate" the video, intelligent recognition of the surgical process can be realized, automatic indexing is also realized to facilitate medical records or teaching resources storage, and the method can also be used as a way of quality evaluation of the surgical process, and even can intelligently remind the next work flow to use the instrument.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and smart medical treatment, and in particular to a surgical stage recognition method, device and equipment based on a multi-head attention mechanism. BACKGROUND

[0002] At present, the recognition of surgical stages mainly focuses on the classification of current frame features, current frame devices or short-time time dependence and shallow convolutional layers. However, surgery is a long process and has strong time and time sequence dependence. For surgical flow recognition classification of a video frame, if only the current video feature classification or the video features within 10s are considered, the reference is limited. The surgery time is generally relatively long and the surgical stages are not simply sequentially executed and may be interleaved repeatedly. Therefore, long-time video feature dependence needs to be considered to improve the accuracy of workflow recognition.

[0003] Moreover, the feature extraction network mainly used in the field at present is ResNet, which is used for global feature extraction. However, surgery videos are relatively detailed videos, and ResNet has poor perception of details. At the same time, the commonly used LSTM for time features has a limited receptive field and slow calculation, which limits the accuracy of video stage recognition. SUMMARY

[0004] The present application provides a surgical stage recognition method, device and equipment based on a multi-head attention mechanism, which is used to realize surgical stage recognition and automatic labeling of surgical video frames.

[0005] In a first aspect, the present application provides a surgical stage recognition method based on a multi-head attention mechanism, which includes:

[0006] Sampling and preprocessing the surgical video;

[0007] SwinTransformer is used as a feature extraction network. The patch feature of each frame of the surgical video is extracted, and 768-dimensional image feature information is obtained after convolution;

[0008] The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix, which is input into a feature enhancement module based on a TCN model to enhance the video time sequence feature processing and obtain a time feature matrix;

[0009] The spatial feature matrix and the time feature matrix are input into a Transformer model for fusion processing of the spatial feature matrix and the time feature matrix, and a 768-dimensional feature matrix is output;

[0010] The 768-dimensional feature matrix is ​​input into Softmax for classification and recognition to obtain the surgical video process stage corresponding to the current original surgical image frame.

[0011] The surgical video process stages are automatically labeled by matching the current original surgical image frame with the corresponding stage dictionary.

[0012] Optionally, the SwinTransformer is used as the feature extraction network. By performing patch feature extraction on each frame of the surgical video and then convolving it, 768-dimensional image feature information is obtained, specifically including:

[0013] The SwinTransformer is used as the feature extraction network. Each frame of the surgical video image (C, H, W) is input to the feature extraction module to extract patch features for each frame of the surgical video. Here, C is the number of channels, and H and W are the height and width, respectively.

[0014] Optionally, the patch feature extraction for each frame of the surgical video includes:

[0015] A single frame of surgical video image is divided into several patches, which are then processed by Patch Partition to become a 48-dimensional vector of 56*56.

[0016] The self-attention of each window is calculated through the Linear Embedding layer and the self-attention mechanism layer. Each window has a 7*7 patch, and the output dimension is 56*56*96.

[0017] After processing the 56*56*96 feature vector using patch merging, a 28*28*192 feature vector is output.

[0018] After processing the 28*28*192 feature vector again using patch merging, a 14*14*384 feature vector is output.

[0019] After processing the 14*14*384 feature vector again using patch merging, a 7*7*768 feature vector is output. Then, after global average pooling, a 1*768 dimensional image feature vector is output.

[0020] Optionally, the step of using the 768-dimensional image feature information of each frame of the surgical video as a spatial feature matrix, inputting it into the feature enhancement module based on the TCN model for enhanced video temporal feature processing, and obtaining a temporal feature matrix specifically includes:

[0021] The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix and input into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix. The TCN model uses dilated convolution or dilated convolution to expand the field of view.

[0022] Optionally, the fusion processing method for the spatial feature matrix and the temporal feature matrix includes at least one of the following: linear fusion, fusion based on multi-head attention mechanism, and fusion based on Transformer.

[0023] Optionally, the step of inputting the spatial feature matrix and the temporal feature matrix into the Transformer model, performing fusion processing of the spatial feature matrix and the temporal feature matrix, and outputting a 768-dimensional feature matrix specifically includes:

[0024] The temporal feature matrix is ​​input into the Transformer model, and after passing through a Transformer encoder layer, it becomes the key. The spatial feature matrix is ​​then input into the Transformer model again as the query for a second Transformer encoding. Finally, it is output as a 768-dimensional feature matrix through a feedforward network.

[0025] Optionally, sampling and preprocessing of the surgical video includes:

[0026] Downsampling was performed on the surgical video;

[0027] Preprocessing of the surgical video after downsampling includes cropping, grayscale conversion, median filtering, and smoothing.

[0028] Secondly, this application provides a surgical stage recognition device based on a multi-head attention mechanism, comprising:

[0029] The processing module is configured to sample and preprocess surgical videos;

[0030] The convolution module is configured to use SwinTransformer as the feature extraction network. By performing patch feature extraction on each frame of the surgical video, 768-dimensional image feature information is obtained after convolution.

[0031] The extraction module is configured to take the 768-dimensional image feature information of each frame of surgical video as a spatial feature matrix, input it into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix.

[0032] The fusion module is configured to input the spatial feature matrix and the temporal feature matrix into the Transformer model, perform fusion processing on the spatial feature matrix and the temporal feature matrix, and output a 768-dimensional feature matrix.

[0033] The recognition module is configured to input the 768-dimensional feature matrix into Softmax for classification and recognition, and obtain the surgical video process stage corresponding to the current original surgical image frame.

[0034] The annotation module is configured to automatically annotate the surgical video stages by matching the stage dictionary with the current original surgical image frame to the corresponding surgical video process stage.

[0035] Thirdly, this application provides an electronic device, comprising:

[0036] Processor; and,

[0037] Memory for storing the executable instructions of the processor;

[0038] The processor is configured to perform any of the possible methods described in the first aspect by executing the executable instructions.

[0039] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the possible methods described in the first aspect.

[0040] This application provides a surgical stage recognition method, device, and equipment based on a multi-head attention mechanism. Considering the special nature of surgery and the relatively numerous details in surgical videos, coupled with the generally long duration and strong temporal sequence of surgeries, a modified SwingTransformer network is used instead of the ResNet network for feature extraction. This allows for the extraction of more detailed features, achieving accurate identification of surgical stages. Furthermore, the accuracy is improved by adding a time-dependent TCN module. The TCN output and the previously generated spatial feature matrix are then input into the Transformer for encoding, parsing, and fusion. This approach, using techniques typically employed for text translation, achieves "translation" of the video based on video features. This enables intelligent recognition of the surgical process, automatic indexing for convenient use as medical records or teaching resources, and can also serve as a method for surgical process quality assessment. It can even provide intelligent reminders for instruments used in the next workflow. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 This is a flowchart illustrating a surgical stage identification method based on a multi-head attention mechanism according to an example embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the network architecture of SwinTransformer according to an example embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the TCN model according to an example embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of a Transformer encoder according to an example embodiment of this application;

[0046] Figure 5 This is a schematic diagram of a surgical stage recognition device based on a multi-head attention mechanism, according to an example embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] Figure 1 This is a flowchart illustrating a surgical stage identification method based on a multi-head attention mechanism, according to an example embodiment of this application. Figure 1 As shown, the method provided in this embodiment includes:

[0051] S101. Sample and preprocess the surgical video.

[0052] The sampling and preprocessing of surgical videos includes, but is not limited to, downsampling, cropping, grayscale conversion, median filtering, and smoothing.

[0053] This invention addresses the characteristics of surgical videos, such as laparoscopic surgery, where a black border is present around the image. This black border represents useless information and can affect the accuracy of feature recognition and judgment. Therefore, preprocessing such as cropping is necessary. This involves cropping the black border and resizing the image to a uniform size M*N. Thus, preprocessing includes, but is not limited to, grayscale conversion, median filtering (to smooth noise), and cropping to a standard size M*N.

[0054] In this embodiment of the invention, a surgical stage dictionary can be constructed in advance before the surgical stage is identified. For example, taking a surgical procedure that includes 7 stages as an example, a surgical stage dictionary as shown in Table 1 below can be constructed.

[0055] Table 1. Dictionary of Surgical Stages

[0056] Number Corresponding stage 1 P1 2 P2 3 P3 4 P4 5 P5 6 P6 7 P7

[0057] S102. Using SwinTransformer as the feature extraction network, patch features are extracted from each frame of the surgical video, and 768-dimensional image feature information is obtained after convolution.

[0058] The SwinTransformer is used as the feature extraction network. Each frame of the surgical video image (C, H, W) is input to the feature extraction module to extract patch features for each frame. Here, C represents the number of channels, and H and W represent the height and width, respectively. (Reference...) Figure 2 As shown, a PatchEmbedding is constructed by extracting patch features from each frame of video, and the image is sliced ​​into patches and embedded into the Embedding. Each Stage of the SwinTransformer consists of Patch Merging and multiple Blocks. SwinTransformer Structure Figure 2 As shown, after finally outputting 7×7×768 dimensional features, convolution is used to generate 768 dimensional image feature information.

[0059] Specifically, a single frame of surgical video image is divided into several patches (taking a 224*224*3 image as an example, the number of patches is 4). After Patch Partition processing, it becomes a 48-dimensional vector of 56*56. The self-attention mechanism layer calculates the self-attention of each window, and each window has 7*7 patches (i.e., the sequence length is 49), with an output dimension of 56*56*96. After processing the 56*56*96 feature vector with patch merging, a 28*28*192 feature vector is output. After processing the 28*28*192 feature vector with patch merging again, a 14*14*384 feature vector is output. After processing the 14*14*384 feature vector with patch merging again, a 7*7*768 feature vector is output. Finally, after Global Average Pooling (GAP), a 1*768-dimensional image feature vector is output.

[0060] The training database for the SwinTransformer network includes a training sample set, a validation sample set, and a test sample set. Each sample set includes surgical video frames from different stages and their corresponding annotations. With the goal of minimizing the total loss function, the SwinTransformer network is trained using the training sample set as an image classification network. The trained SwinTransformer network is then obtained, and the video frames to be segmented are input into the trained SwinTransformer network to obtain image feature attributes with 768 feature dimensions.

[0061] S103. The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix and input into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and a temporal feature matrix is ​​obtained.

[0062] The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix and input into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix. The TCN model uses dilated convolution or dilated convolution to expand the field of view.

[0063] Temporal Convolutional Networks (TCNs) are often used to model sequential data, and their dilated causal convolutions can obtain features over a longer period while reducing the number of convolutional layers. TCN models provide more accurate temporal references, thus enabling long-term feature enhancement. TCN = 1 / DfCN + Causal Convolution.

[0064] The structure of the TCN model is as follows: Figure 3 As shown, the TCN model uses dilated or dilated convolution to expand the field of view:

[0065]

[0066] The 768-dimensional image feature information of each frame of the surgical video is processed by dilated causal convolution to extract temporal features, and the result is a spatial feature matrix, which is also a 768-dimensional feature sequence. Due to the special structure of the TCN model, its temporal receptive field is larger than that of LSTM, and its computation speed is better than that of LSTM.

[0067] S104. Input the spatial feature matrix and the temporal feature matrix into the Transformer model, perform fusion processing of the spatial feature matrix and the temporal feature matrix, and output a 768-dimensional feature matrix.

[0068] The fusion processing methods for spatial feature matrices and temporal feature matrices include at least one of the following: linear fusion, fusion based on multi-head attention mechanisms, and fusion based on Transformer mechanisms. Taking the Transformer-based fusion method as an example, refer to... Figure 4 As shown, the temporal feature matrix is ​​input into the Transformer model, and after passing through a Transformer encoder layer, it serves as the key. The spatial feature matrix is ​​then input into the Transformer model again as the query for a second Transformer encoding. Finally, it is passed through a feedforward network to output a 768-dimensional feature matrix. Taking the linear fusion method as an example, Layer Normalization is performed after A+B.

[0069] S105. Input the 768-dimensional feature matrix into Softmax for classification and recognition to obtain the surgical video process stage corresponding to the current original surgical image frame.

[0070] The 768-dimensional feature matrix after fusion processing is used to classify and infer the current surgical stage using softmax. For each frame of the surgical video image, the 768-dimensional spatial and temporal feature matrices are processed using the softmax function to transform the model's output into the corresponding class probabilities for each stage. Specifically, the input data is mapped to real numbers between 0 and 1, and the output is normalized to ensure that the sum of the outputs is 1. The surgical stage corresponding to that frame of the surgical video image is selected based on the probability with the highest normalized probability.

[0071] Where i represents the number of the output node.

[0072] For example, a convolutional neural network and a classifier are trained using a pre-classified set of surgical videos to evaluate the recognition performance of different stages of the surgical video process and determine relevant parameters.

[0073] S106. Based on the current original surgical image frame, the corresponding surgical video process stage is matched with the stage dictionary to obtain the automatic annotation of the surgical video stage.

[0074] Based on the above classification, a dictionary of surgical video stages is established, and automatic annotation of surgical video stages is performed. Annotated timelines are generated based on the video timelines. Based on the constructed surgical stage dictionary, classification labels and specific surgical stage annotations are applied.

[0075] The system sorts each frame and its corresponding classification results temporally and smooths the video along the timeline. Since a surgical phase is unlikely to be a mere one-second pause, and considering the specific nature of surgery, extremely short phases are imperceptible, the classified data is smoothed along the timeline to remove glitches. The smoothing method involves applying mean filtering to the corresponding values ​​of each timeline phase using a filter slider of size L to eliminate glitches and improve accuracy. An abnormal phase removal module is also added to delete extremely short-duration fluctuations, significantly reducing glitches and jitter. If preceding and following phases are identical, and this phase lasts only a very short time (less than t) and its proportion with preceding and following phases is less than q%, then this phase is removed and corrected.

[0076] This application provides a surgical stage recognition method, device, and equipment based on a multi-head attention mechanism. Considering the special nature of surgery and the relatively numerous details in surgical videos, coupled with the generally long duration and strong temporal sequence of surgeries, a modified SwingTransformer network is used instead of the ResNet network for feature extraction. This allows for the extraction of more detailed features, achieving accurate identification of surgical stages. Furthermore, the accuracy is improved by adding a time-dependent TCN module. The TCN output and the previously generated spatial feature matrix are then input into the Transformer for encoding, parsing, and fusion. This approach, using techniques typically employed for text translation, achieves "translation" of the video based on video features. This enables intelligent recognition of the surgical process, automatic indexing for convenient use as medical records or teaching resources, and can also serve as a method for surgical process quality assessment. It can even provide intelligent reminders for instruments used in the next workflow.

[0077] Figure 5 This is a schematic diagram illustrating the structure of a surgical stage recognition device based on a multi-head attention mechanism, according to an example embodiment of this application. Figure 5 As shown, the surgical stage identification device 200 provided in this embodiment includes:

[0078] Processing module 201 is configured to sample and preprocess surgical videos;

[0079] Convolutional module 202 is configured to use SwinTransformer as a feature extraction network to extract patch features from each frame of surgical video and obtain 768-dimensional image feature information after convolution.

[0080] The extraction module 203 is configured to take the 768-dimensional image feature information of each frame of surgical video as a spatial feature matrix, input it into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix.

[0081] The fusion module 204 is configured to input the spatial feature matrix and the temporal feature matrix into the Transformer model, perform fusion processing of the spatial feature matrix and the temporal feature matrix, and output a 768-dimensional feature matrix.

[0082] The recognition module 205 is configured to input the 768-dimensional feature matrix into Softmax for classification and recognition, and obtain the surgical video process stage corresponding to the current original surgical image frame.

[0083] The annotation module 206 is configured to automatically annotate the surgical video stages by matching the stage dictionary with the surgical video process stage corresponding to the current original surgical image frame.

[0084] Optionally, the convolution module 202 is specifically used for:

[0085] The SwinTransformer is used as the feature extraction network. Each frame of the surgical video image (C, H, W) is input to the feature extraction module to extract patch features for each frame of the surgical video. Here, C is the number of channels, and H and W are the height and width, respectively.

[0086] Optionally, the convolution module 202 is specifically used for:

[0087] A single frame of surgical video image is divided into several patches, which are then processed by Patch Partition to become a 48-dimensional vector of 56*56.

[0088] The self-attention of each window is calculated through the Linear Embedding layer and the self-attention mechanism layer. Each window has a 7*7 patch, and the output dimension is 56*56*96.

[0089] After processing the 56*56*96 feature vector using patch merging, a 28*28*192 feature vector is output.

[0090] After processing the 28*28*192 feature vector again using patch merging, a 14*14*384 feature vector is output.

[0091] After processing the 14*14*384 feature vector again using patch merging, a 7*7*768 feature vector is output. Then, after global average pooling, a 1*768 dimensional image feature vector is output.

[0092] Optionally, the extraction module 203 is specifically used for:

[0093] The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix and input into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix. The TCN model uses dilated convolution or dilated convolution to expand the field of view.

[0094] Optionally, the fusion processing method for the spatial feature matrix and the temporal feature matrix includes at least one of the following: linear fusion, fusion based on multi-head attention mechanism, and fusion based on Transformer.

[0095] Optionally, the fusion module 204 is specifically used to include:

[0096] The temporal feature matrix is ​​input into the Transformer model, and after passing through a Transformer encoder layer, it becomes the key. The spatial feature matrix is ​​then input into the Transformer model again as the query for a second Transformer encoding. Finally, it is output as a 768-dimensional feature matrix through a feedforward network.

[0097] Optionally, the processing module 201 is specifically used for:

[0098] Downsampling was performed on the surgical video;

[0099] Preprocessing of the surgical video after downsampling includes cropping, grayscale conversion, median filtering, and smoothing.

[0100] Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of this application. For example... Figure 6 As shown, the electronic device 300 provided in this embodiment includes: a processor 301 and a memory 302; wherein:

[0101] Memory 302 is used to store computer programs, and the memory may also be flash memory.

[0102] Processor 301 is used to execute the execution instructions stored in the memory to implement the various steps in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0103] Alternatively, the memory 302 can be either standalone or integrated with the processor 301.

[0104] When the memory 302 is a device independent of the processor 301, the electronic device 300 may further include:

[0105] Bus 303 is used to connect the memory 302 and the processor 301.

[0106] This embodiment also provides a readable storage medium storing a computer program, which, when executed by at least one processor of an electronic device, enables the electronic device to perform the methods provided in the various embodiments described above.

[0107] This embodiment also provides a program product including a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the methods provided in the various embodiments described above.

[0108] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0109] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A surgical stage recognition method based on multi-head attention mechanism, characterized in that, The surgical stage identification method includes: Sampling and preprocessing of surgical videos; The SwinTransformer was used as the feature extraction network. Patch features were extracted from each frame of the surgical video, and 768-dimensional image feature information was obtained after convolution. The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix, which is then input into the feature enhancement module based on the TCN model to perform feature processing for video temporal enhancement, and a temporal feature matrix is ​​obtained. The spatial feature matrix and the temporal feature matrix are input into the Transformer model to perform fusion processing of the spatial feature matrix and the temporal feature matrix, and output a 768-dimensional feature matrix. The 768-dimensional feature matrix is ​​input into Softmax for classification and recognition to obtain the surgical video process stage corresponding to the current original surgical image frame. The surgical video process stages are automatically labeled by matching the current original surgical image frame with the corresponding stage dictionary.

2. The surgical stage identification method according to claim 1, characterized in that, The method employs SwinTransformer as the feature extraction network. By performing patch feature extraction on each frame of the surgical video and then convolving it, 768-dimensional image feature information is obtained, specifically including: The SwinTransformer is used as the feature extraction network. Each frame of the surgical video image (C, H, W) is input to the feature extraction module to extract patch features for each frame of the surgical video. Here, C is the number of channels, and H and W are the height and width, respectively.

3. The surgical stage identification method according to claim 2, characterized in that, The step of extracting patch features for each frame of the surgical video includes: A single frame of surgical video image is divided into several patches, which are then processed by Patch Partition to become a 48-dimensional vector of 56*56. The self-attention of each window is calculated through the Linear Embedding layer and the self-attention mechanism layer. Each window has a 7*7 patch, and the output dimension is 56*56*96. After processing the 56*56*96 feature vector using patch merging, a 28*28*192 feature vector is output. After processing the 28*28*192 feature vector again using patch merging, a 14*14*384 feature vector is output. After processing the 14*14*384 feature vector again using patch merging, a 7*7*768 feature vector is output. Then, after global average pooling, a 1*768 dimensional image feature vector is output.

4. The surgical stage identification method according to claim 1, characterized in that, The step involves using the 768-dimensional image feature information of each frame of the surgical video as a spatial feature matrix, inputting it into a feature enhancement module based on the TCN model for enhanced video temporal feature processing, and obtaining a temporal feature matrix. Specifically, this includes: The 768-dimensional image feature information of each frame of the surgical video is used as a spatial feature matrix and input into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix. The TCN model uses dilated convolution or dilated convolution to expand the field of view.

5. The surgical stage identification method according to claim 1, characterized in that, The fusion processing methods for spatial feature matrices and temporal feature matrices include at least one of the following: linear fusion, fusion based on multi-head attention mechanism, and fusion based on Transformer.

6. The surgical stage identification method according to claim 1, characterized in that, The process of inputting the spatial feature matrix and the temporal feature matrix into the Transformer model, fusing the spatial and temporal feature matrices, and outputting a 768-dimensional feature matrix specifically includes: The temporal feature matrix is ​​input into the Transformer model, and after passing through a Transformer encoder layer, it becomes the key. The spatial feature matrix is ​​then input into the Transformer model again as the query for a second Transformer encoding. Finally, it is output as a 768-dimensional feature matrix through a feedforward network.

7. The surgical stage identification method according to claim 1, characterized in that, Sampling and preprocessing of surgical videos includes: Downsampling was performed on the surgical video; Preprocessing of the surgical video after downsampling includes cropping, grayscale conversion, median filtering, and smoothing.

8. A surgical stage recognition device based on a multi-head attention mechanism, characterized in that, The surgical stage identification device includes: The processing module is configured to sample and preprocess surgical videos; The convolution module is configured to use SwinTransformer as the feature extraction network. By performing patch feature extraction on each frame of the surgical video, 768-dimensional image feature information is obtained after convolution. The extraction module is configured to take the 768-dimensional image feature information of each frame of surgical video as a spatial feature matrix, input it into the feature enhancement module based on the TCN model to perform feature processing to enhance the temporal sequence of the video, and obtain the temporal feature matrix. The fusion module is configured to input the spatial feature matrix and the temporal feature matrix into the Transformer model, perform fusion processing on the spatial feature matrix and the temporal feature matrix, and output a 768-dimensional feature matrix. The recognition module is configured to input the 768-dimensional feature matrix into Softmax for classification and recognition, and obtain the surgical video process stage corresponding to the current original surgical image frame. The annotation module is configured to automatically annotate the surgical video stages by matching the stage dictionary with the current original surgical image frame to the corresponding surgical video process stage.

9. An electronic device, characterized in that, include: processor; as well as, Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images