Endoscopic surgery stage identification method based on spatio-temporal information fusion

Abstract

The invention belongs to the technical field of wisdom medical treatment, and particularly relates to an endoscopic surgery stage recognition method based on spatio-temporal information fusion, which comprises the following steps: 1) collecting endoscopic surgery videos and carrying out downsampling to obtain a picture sequence data set; 2) performing preliminary feature extraction on the input image sequence data by using an improved two-way spatial feature extractor to obtain a spatial feature map containing multi-scale information; (3) modeling the extracted spatial features with the time sequence information in time sequence by using a mixed time feature extractor, and fully mining the time sequence relationship between different stages by effectively mixing two different time sequence information; 4, the time context relation of the operation video is explored through an auto-aggregation module based on a coder-decoder structure, and a final result is predicted.According to the method, the synergistic effect of time sequence information in the operation video is explored in the mode that time embedding is used for inquiring time embedding, and the accuracy of operation stage recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to an endoscopic surgery stage recognition method based on spatiotemporal information fusion. Background Art

[0002] With the development of computer-assisted surgery technology, the accuracy and safety of surgical operations have been significantly improved. As a key task in computer-assisted surgery systems, surgical stage recognition aims to identify the different stages of the surgical process in real time, thereby optimizing the surgical process, supporting decision-making, and performing skill assessment. However, surgical stage recognition faces many challenges. For example, the surgical process is highly dynamic and nonlinear, and is affected by multiple factors such as the patient's anatomical characteristics, the surgeon's technical level, and surgical events, making surgical stage recognition more difficult. In addition, the amount of data during surgery is huge and varied, and how to effectively extract useful features from it has become a difficulty in surgical stage recognition.

[0003] In recent years, the progress of deep learning has significantly promoted the development of surgical stage recognition methods, and several representative methods have emerged. These methods can be roughly divided into video-based methods, multimodal data fusion-based methods, and time series-based methods, but these methods generally have limitations. For example, although video-based methods use visual features for recognition, they are affected by the rapid changes in the surgical process and the diversity of perspectives, and the recognition effect is unstable; although multimodal data fusion can integrate multiple data sources, the technical difficulty of data synchronization and fusion is relatively large; and traditional time series methods often ignore the extraction of spatial features, resulting in insufficient accuracy when dealing with complex surgical scenes. Summary of the invention

[0004] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides an endoscopic surgery stage recognition method based on spatiotemporal information fusion, which can simultaneously capture the spatial and temporal features of the surgical process. Through multi-scale feature fusion and the introduction of a self-aggregation mechanism, the present invention significantly improves the accuracy and robustness of surgical stage recognition, and can effectively identify different surgical stages in complex surgical scenarios, thereby meeting the needs of intelligent operating room systems for high-precision surgical stage recognition and solving the problem of the limitations of traditional methods in processing variable and high-dimensional data.

[0005] (II) Technical solution In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: An endoscopic surgery stage recognition method based on spatiotemporal information fusion is as follows: 1) Data collection and preprocessing: Endoscopic surgery videos are collected. Each video records the entire process of a surgeon performing an endoscopic surgery. The videos are then cut into images by frame to obtain an image sequence dataset. The stage to which each frame belongs is marked. 2) Spatial feature extraction: Using an improved dual-path spatial feature extractor, preliminary feature extraction is performed on the input image sequence data to obtain a spatial feature map containing spatial multi-scale information; At the same time, the proposed attention mechanism is combined to strengthen feature expression and highlight key information; 3) Temporal feature extraction: A hybrid temporal feature extractor is proposed to model the extracted spatial features with temporal information in terms of temporal sequence. By effectively mixing two different temporal information, the temporal relationship between different stages can be fully explored. 4) Temporal self-aggregation: Since temporal information is very important for the task of surgical stage recognition, and in order to give full play to the ability of Transformer in processing sequence problems, the temporal information is self-aggregated to make full use of the local contextual temporal relationship, so that the model can more accurately identify different surgical stages; 5) Calculation of loss function and optimization: Since the surgical stage recognition task is modeled as a multi-category classification task, the cross entropy loss function and Adam optimizer are used for parameter optimization. This loss function can balance the learning effects of different tasks and ensure that the model can achieve the best performance in multi-task learning.

[0006] Furthermore, in step 1), before obtaining the image sequence dataset, the original video is downsampled to one frame per second, and data preprocessing and data enhancement are performed before inputting the network, including: resizing the image to a uniform size of 250×250, randomly cropping it to a size of 224×224, randomly perturbing the color, randomly rotating it, normalizing it, and other operations.

[0007] Furthermore, in step 2), the spatial feature extractor is composed of a ResNet50 model and a DenseNet121 model, combined with the proposed novel channel attention mechanism; This spatial feature extractor combines the feature reuse mechanism of residual connection and the feature re-exploration mechanism of dense connection to enrich the final spatial features and enhance the spatial feature expression ability of the model.

[0008] Furthermore, in step 3), the temporal feature extractor mixes the MS-TCN model and the LSTM model in parallel. The former focuses on temporal multi-scale feature extraction, while the latter focuses on exploring the temporal dependencies of data. By effectively combining these two models, the intrinsic temporal characteristics of surgical videos are fully modeled.

[0009] Furthermore, in step 4), the self-aggregator fully explores the local contextual relationship of temporal features through the encoding-decoding architecture, so that the model can show strong robustness during stage transitions.

[0010] (III) Beneficial effects Compared with the prior art, the present invention provides an endoscopic surgery stage recognition method based on spatiotemporal information fusion, which has the following beneficial effects: 1. The present invention proposes a novel channel attention mechanism, which enhances the quality of spatial feature representation by explicitly modeling the interdependencies between channels of convolutional feature maps.

[0011] 2. The present invention, through the design of a novel spatial feature extractor, inherits the advantages of residual connection and dense connection, so that features can be effectively reused and re-explored, thereby improving the spatial feature representation of the model and the expressiveness of the model. Compared with the traditional spatial feature extractor, the spatial feature expression ability is stronger.

[0012] 3. The present invention integrates a hybrid time model through a multi-scale temporal feature aggregation framework, and fully models the temporal characteristics of surgical videos by utilizing the advantages of MS-TCN in temporal multi-scale feature extraction and the advantages of LSTM in capturing long-term dependencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a logical flow diagram of the present invention; Figure 2 It is the overall network architecture diagram of the present invention; Figure 3 Schematic diagram of a spatial feature extractor of the present invention; Figure 4 Schematic diagram of a residual network of the present invention; Figure 5 It is a schematic diagram of the coding and decoding structure of the present invention; Figure 6 Schematic diagram of the self-aggregation of time features by the codec structure of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Example like Figure 1 and Figure 2As shown, an embodiment of the present invention proposes: a method for identifying the stage of endoscopic surgery based on spatiotemporal information fusion, comprising the following steps: 1) The frame rate of 80 endoscopic surgery videos was downsampled to obtain a frame-by-frame image sequence and construct a surgical stage recognition dataset; Then a binary file is generated, which contains information such as the image location, image sequence number, and stage to which it belongs, to guide model training; The dataset is then split into training set, validation set, and test set in a ratio of 40:8:32, which are used for model training, weight selection, and final effect testing, respectively.

[0016] For different types of surgery, a surgical stage dictionary is pre-constructed, which contains the categories of each surgical stage of the surgery. Table 1 shows the categories of each surgical stage of cholecystectomy, as shown in the following table: Table 1 Categories of surgical stages ; It should be noted that Table 1 only categorizes the stages of cholecystectomy; different types of surgery may contain different numbers and types of surgical stages.

[0017] 2) Image data preprocessing and data enhancement: Considering the particularity of surgical scenes, for example, there will be black areas around the video frames of endoscopic surgery. This area is useless information and will affect the accuracy of recognition during the surgical stage. Therefore, after downsampling the video frames, the black areas in each frame will be cropped and the image size will be adjusted to a uniform size. In order to enhance the robustness and generalization of the model and adapt to different shooting conditions and surgical scenarios, the image data is randomly transformed in terms of brightness, contrast, saturation and hue before model training.

[0018] 3) Surgical video frames usually show high complexity, including various features, such as surgical instruments, different tissues and organs, strong similarities between different stages, and the video duration may be as long as tens of minutes; Therefore, the model must be able to extract deep features from frames and effectively process long video inputs; To this end, a dual-path based spatial feature extractor is proposed, such as Figure 3 As shown, the extractor integrates a residual network focusing on feature reuse and a densely connected network focusing on new feature extraction, and adopts a parallel framework; Residual network Figure 4 As shown in (a), it consists of 5 units, and the sizes of the output feature maps are 64, 256, 512, 1024 and 2048 respectively; The first unit consists of a 3×3 max pooling layer and a 7×7 convolution kernel; The remaining four units are stacked, consisting of 3, 4, 6, and 3 bottleneck layers, respectively; The bottleneck structure used in the network is as follows Figure 4 (b) Each bottleneck is composed of three stacked convolutional residual layers: the first layer is a 1×1 convolutional layer, which is used to reduce the number of input channels and thus reduce the amount of computation; The second layer is a 3×3 convolutional layer, which is used to extract features from the data; The last layer is a 1×1 convolutional layer, which restores the number of channels to the number of inputs; Each layer is followed by a batch normalization layer and a ReLU activation layer; The residual connection between input and output is used to ensure that the feature map can inherit the features from the previous unit, so that the network can be trained deeper; Research has shown that the channel attention mechanism significantly improves the performance of convolutional neural networks by dynamically adjusting the weights of different channels in the input feature map, thereby enhancing the model's ability to effectively represent features. like Figure 4 As shown in (c), a specific attention module (represented by a dark gray rectangle) is applied to the bottleneck layer, and a new attention mechanism shuffle squeeze-and-excitation (sSE) is proposed. This attention mechanism ensures that information can be redistributed among channels by adding shuffle operations, so that the model can better capture complex feature dependencies. The structure of the attention mechanism is as follows Figure 5 As shown: Meanwhile, surgical stage recognition is a multi-class problem with class imbalance, which means that different classes (i.e., different surgical stages) do not appear with equal frequencies in the dataset; The number of samples in some stages is much larger than in others; Therefore, we used median frequency balancing to calculate category weights; During model training, the weight of each category is adjusted to deal with the problem of class imbalance; The calculation method of median frequency balance is as follows: (2) in, Indicates category The weight of Indicates category The frequency, represents the median of all category frequencies, The definition of is as follows: (3) 4) The surgical stage recognition task essentially involves capturing temporal dependencies. Since there is a temporal relationship between the stages, a parallel hybrid temporal architecture is adopted. This design can give full play to the complementary advantages of each model in capturing temporal dynamics. The architecture diagram is shown in the figure below. Figure 2 As shown in the middle part of; Long Short-Term Memory network (LSTM) is a specially designed Recurrent Neural Network (RNN) structure that aims to solve the gradient vanishing and gradient exploding problems encountered by standard RNN when dealing with long-term dependency problems. LSTM controls the flow of information by introducing memory units (cell states) and multiple gating mechanisms (input gate, forget gate, and output gate), enabling it to perform well when processing long sequence data; The core operation of LSTM can be expressed by formula (4): (4) in, , and They are input gate, forget gate and output gate. is the cell state; is the hidden state, parameter , and are weights and biases, respectively. and Represent the Sigmoid activation function and the hyperbolic tangent activation function respectively. This mechanism enables LSTM to capture both short-term and long-term dependencies, which is crucial for processing surgical stage sequences of different lengths.

[0019] Multi Stages-Temporal Convolutional Networks (MS-TCN) is an architecture based on Temporal Convolutional Networks (TCN), specifically designed for online action segmentation and recognition tasks. The model uses dilated convolutions to capture long-term dependencies while maintaining causality to ensure that the prediction of each time step only uses information from the current and past frames. The output of a single dilated convolution layer is as follows: (5) in, It is Layer dilation convolutional layer in time The output, is the size of the convolution kernel, is the weight, is the expansion factor, which is obtained by changing ,MS-TCN is able to capture both local transitions and long-term stage trends, and compared with recursive architectures, MS-TCN benefits from parallel computing, thereby reducing inference time.

[0020] The parallel integration of LSTM and MS-TCN takes advantage of their complementary strengths. LSTM effectively captures long-term sequence dependencies, while MS-TCN excels in multi-scale temporal modeling through convolution operations. The final output of the temporal feature extractor is obtained by summing and averaging the outputs of LSTM and MS-TCN.

[0021] 5) Use Figure 6 The encoding and decoding structure shown in the figure performs self-aggregation on the temporal features. The encoder consists of a multi-head self-attention layer, a residual normalization layer, and a feedforward layer. In order to give full play to the role of temporal information in the surgical stage recognition task, time embedding is used to represent the temporal relationship in the surgical video, and it is used as the input of the codec for self-aggregation operation. This self-aggregation mechanism enables the model to focus more on feature extraction in the time dimension.

[0022] As a key component, the encoder is responsible for processing and transforming the input sequence into a comprehensive, context-aware representation. By leveraging multi-head self-attention and feedforward networks, the encoder is able to capture complex dependencies in the input sequence. The context-enhanced output is then passed to the decoder.

[0023] The encoder receives a temporal embedding vector , as shown in the following formula (6): (6) in, represents the sequence length, Represents the embedding dimension.

[0024] The encoder uses a multi-head self-attention mechanism to capture the dependencies in the input sequence. , the linear projection is expressed as shown in the following formula (7): (7) in are all learnable matrices, Indicates the dimensions of each head.

[0025] For each head , use the SoftMax activation function to calculate the attention weight, as shown in the following formula (8): (8) After obtaining the attention weights, the output of the self-attention mechanism can be calculated according to the following formula (9): (9) The outputs of all attention heads are concatenated along the channel dimension to combine the information captured by each head. The concatenated result is then restored to the original embedding dimension through a linear transformation. So that the subsequent layers can effectively process the combined information, as shown in the following formula (10): (10) in, .

[0026] After the multi-head attention, a residual and normalization layer is used to enhance the stability and performance of the model by combining the input and output features and normalizing the results, as shown in the following formula (11): (11) Finally, a feed-forward network (FFN) is used to enhance the model’s ability to independently learn complex transformations at each position, thereby improving the overall expressive power, as shown in the following formula (12): (12) The specific implementation of the feedforward network is shown in formula (13): (13) in, , .

[0027] Final output of the encoder The decoder is provided with a representation of the input sequence with rich context.

[0028] The decoder consists of an encoder-decoder attention layer, a residual normalization layer, and a feed-forward network to embed the current input into With the encoder from the sequence The context information captured in is integrated, and the decoder query is , keywords and values ​​are the output of the encoder. Through such a setting, the intrinsic connection of timing information can be better explored.

[0029] Finally, the CE loss function is used to train the model, and the function is shown in the following formula (14): (14) in, Indicates The predicted class of the frame, Indicates The true class of the frame.

[0030] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An endoscopic surgery stage recognition method based on spatiotemporal information fusion, characterized in that: The following steps are involved: 1) Collect endoscopic surgery videos, downsample the video frame rate and mark the stage to which each frame belongs; 2) Crop the video frames and resize the images to a uniform size, and enhance the image data; 3) Using an improved two-way spatial feature extractor, we perform preliminary feature extraction on the input image sequence data to obtain a spatial feature map containing multi-scale information. At the same time, we combine the proposed attention mechanism to enhance feature expression and highlight key information. 4) Using the hybrid temporal feature extractor, the extracted spatial features with temporal information are modeled in terms of time series. By effectively mixing two different types of temporal information, the temporal relationship between different stages is fully explored. 5) Self-aggregation of temporal information to fully utilize local temporal relationships, so that the model can more accurately identify different surgical stages; 6) Use the cross entropy loss function and Adam optimizer for parameter optimization. This loss function can balance the learning effects of different tasks and ensure that the model can achieve the best performance in multi-task learning.

2. The method for identifying the stage of endoscopic surgery based on spatiotemporal information fusion according to claim 1, characterized in that: In the step 1), before obtaining the image sequence data set, the original surgical video is downsampled to a frame rate of one frame per second, and the stage to which each frame belongs is marked.

3. The method for identifying the stage of endoscopic surgery based on spatiotemporal information fusion according to claim 1, characterized in that: In the step 2), taking into account the particularity of the surgical scene, such as the black area around the video frame of endoscopic surgery, which is useless information and will affect the accuracy of recognition during the surgical stage, the black area in each frame will be cropped and the image size will be adjusted to a uniform size after downsampling the video frame; in order to enhance the robustness and generalization of the model and adapt to different shooting conditions and surgical scenes, the image data is randomly transformed in terms of brightness, contrast, saturation and hue before model training.

4. The method for identifying endoscopic surgery stages based on spatiotemporal information fusion according to claim 1, characterized in that: In the step 3), the spatial feature extractor is composed of ResNet50 and DenseNet121, which are used to realize feature reuse and new feature extraction, and a parallel framework is adopted. The results of the two are combined by channel splicing; the proposed channel attention mechanism sSE is applied to the bottleneck layer of ResNet50 to capture complex feature dependencies.

5. The method for identifying endoscopic surgery stages based on spatiotemporal information fusion according to claim 1, characterized in that: In step 4), the temporal feature extractor is composed of MS-TCN and LSTM in parallel, which are used to extract temporal multi-scale features and capture short-term and long-term dependencies of surgical videos respectively. Since the output sizes of the two are the same, the results are combined by element-wise averaging.

6. The method for identifying the stage of endoscopic surgery based on spatiotemporal information fusion according to claim 1, characterized in that: In the step 5), the temporal features are self-aggregated using a coding and decoding structure, and temporal embedding is used to represent the temporal relationship in the surgical video, and the self-aggregation operation is performed as the input of the codec. The encoder is composed of a multi-head self-attention layer, a residual normalization layer, and a feedforward layer; the multi-head self-attention layer is used to model context information; The features with contextual information are then fed into a feedforward network to learn complex nonlinear mapping relationships; the encoder consists of an encoder-decoder attention layer, a residual normalization layer, and a feedforward layer; the output of the encoder is input as a key-value pair into the encoder-decoder attention layer of the decoder for combining with the initial time embedding to generate the final output; The encoder-decoder attention layer introduces contextual information into the decoder by accepting the output of the encoder, and finally predicts the final result through the feedforward network.

7. The method for identifying the stage of endoscopic surgery based on spatiotemporal information fusion according to claim 1, characterized in that: In step 6), the cross entropy loss function for surgical stage identification is shown in the following formula (1): （1）； in, Indicates the total number of frames in the video. Indicates The predicted class of the frame, Indicates The actual class of the frame.

Images