A surgical stage recognition method based on time-frequency feature fusion
By constructing the TFRCDLFormer network and combining multi-level frequency domain and multi-scale time domain feature modules, the problems of difficulty in local feature recognition and high computational resource requirements in surgical videos are solved, and efficient surgical stage recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2024-11-14
- Publication Date
- 2026-05-29
AI Technical Summary
Existing deep learning-based surgical stage recognition methods suffer from difficulties in identifying local features and high computational resource requirements when processing surgical videos.
A time-frequency feature fusion-based approach is adopted. By constructing the TFRCDLFormer network, multi-level frequency domain feature modules and multi-scale time domain feature modules are used, combined with ResNet network, RCDL and Transformer model to extract and recognize features from surgical videos.
It improves the accuracy and efficiency of surgical stage identification, reduces the demand for computing resources, and enhances the ability to identify local features.
Smart Images
Figure CN119559554B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and in particular relates to a surgical stage recognition method based on time-frequency feature fusion. Background Technology
[0002] In the operating room (OR), surgical procedure analysis is an indispensable task. Surgical stage identification provides crucial input to physicians as an early warning method for deviations and anomalies. Another role is the automatic extraction of surgical protocols, which is significant for archiving, education, and postoperative patient monitoring. However, the task of surgical stage identification is extremely challenging due to the small direct differences between different surgical activities, artifacts in surgical records, camera movement, and the influence of factors such as blood and smoke. Machine learning-based computer-aided (CAI) systems have been developed for surgical procedure analysis. These methods utilize not only OR signals but also intraoperative videos captured during laparoscopic surgery. In recent years, deep learning has been used to extract temporal and spatial information from trimmed surgical videos to automatically identify surgical stages, such as TCN (Temporal Convolutional Networks), LSTM, and Transformer network models. Due to the rapid development of deep learning and the ease of acquiring surgical videos, these methods have become mainstream for surgical stage identification tasks. They first utilize ResNet networks to extract spatial features from images to reduce computation, and then input these features into a sequence model to obtain temporal features.
[0003] However, despite these methods significantly advancing surgical phase recognition, they still have some limitations. RCDL (Residual Causal Dilated Temporal Convolution Layer) methods utilize multiple dilated convolution layers to obtain a larger receptive field, improving the ability to model long sequences. LSTM-based methods leverage the stronger long sequence modeling capability of LSTM models compared to RNNs, storing past features and acquiring longer sequence information. Influenced by the huge success of Transformers in NLP and CV, Transformer-based methods have been widely applied to surgical phase recognition tasks, using multi-head attention to find highly relevant features globally. However, the corresponding drawbacks are that local features are difficult to identify, and longer sequences require significant computational resources.
[0004] To address the above issues, a surgical stage identification method based on time-frequency feature fusion is proposed. Summary of the Invention
[0005] To address the above technical problems, this invention provides a surgical stage identification method based on time-frequency feature fusion.
[0006] The technical solution adopted by this invention to solve its technical problem is:
[0007] A surgical stage identification method based on time-frequency feature fusion, the method comprising the following steps:
[0008] S100: Collect laparoscopic surgical videos, cut them according to a preset frequency to obtain multiple consecutive photos, input the cut photos into a trained ResNet network, extract useful spatial features, and obtain a dataset;
[0009] S200: Construct the TFRCDLFormer network, which adopts a dual-branch structure, including a multi-level frequency domain feature module, a multi-scale time domain feature module, and a multi-level frequency domain feature enhancement module. Branch 1 is the multi-level frequency domain feature module, and branch 2 is the multi-scale time domain feature module.
[0010] S300: The multi-level frequency domain feature module is used to receive images in the dataset, perform multi-layer dilated convolutional RCDL processing, and then perform discrete wavelet transform to obtain multi-level frequency information containing high and low frequencies. After performing preset corresponding processing on the high and low frequency information respectively, the inverse discrete wavelet transform is performed to obtain multi-level frequency domain features.
[0011] S400: The multi-scale temporal feature module is used to receive images in the dataset, perform RCDL processing, and then perform average pooling and upsampling processing to obtain feature information at different scales. The feature information at each scale is processed by RCDL and Transformer to obtain multi-scale temporal features.
[0012] S500: The multi-level frequency domain feature enhancement module is used to fuse multi-level frequency domain features and multi-scale time domain features through a fully connected layer, and then perform frequency domain feature enhancement according to the method in S300. After passing through the fully connected layer, the output is the surgical stage recognition and prediction result.
[0013] S600: The TFRCDLFormer network is trained based on a preset loss function. When the preset training termination is reached, the trained TFRCDLFormer network is obtained, and real-time surgical stage recognition is completed.
[0014] Preferably, S300 includes:
[0015] S310: Use multi-layer RCDL to obtain the frequency information of images in the dataset;
[0016] S320: Performs three discrete wavelet transforms on the frequency information, and obtains low-frequency and high-frequency features based on high-pass and low-pass filters;
[0017] S330: Low-frequency features are processed using RCDL, and then processed using the Transformer's attention mechanism; high-frequency features are processed using multi-layer RCDL.
[0018] S340: Perform inverse wavelet transform on the processed multi-level frequency features.
[0019] Preferably, S310 specifically comprises:
[0020] Z l =ReLU(W 1,l *F l-1 +b 1,l )
[0021] F l =F l-1 +W 2,l *Z l +b 2,l
[0022] Among them, F l Represents the frequency features of the l-th layer, * indicates convolution calculation, W 1,l It is a 1D dilated convolution kernel, W 2,l These are the weights of a 1x1 convolution, b 1,l ,b 2,l It is a paranoid vector;
[0023] The discrete wavelet transform in S320 is specifically as follows:
[0024]
[0025] Where g[n] are the low-pass filter coefficients, h[n] are the high-pass filter coefficients, a(n) are the approximate coefficients after decomposition, and d(n) are the detail coefficients after decomposition. The input frequency information is represented by n, where n represents the length of the discrete input signal, and k is the horizontal coordinate position.
[0026] Preferably, the low-frequency features in S330 are processed using RCDL, and then processed using the attention mechanism of Transformer, specifically as follows:
[0027] x = RCDLs(x)
[0028] x = Transformer(x) + x
[0029] Among them, RCDLs and Transformer represent multi-layer RCDL modules and Transformer modules, respectively;
[0030] The inverse wavelet transform in S340 is specifically as follows:
[0031]
[0032] Where g'[n] and h'[n] are the coefficients of the inverse transform filter. This represents the frequency domain characteristics of the output.
[0033] Preferably, S400 includes:
[0034] S410: Utilizes multi-layer RCDL to obtain temporal information of input features, and downsamples the sequence through average pooling to obtain feature information at four different scales;
[0035] S420: Feature upsampling is performed using the RCDL+ upsampling method;
[0036] S430: Features at each scale after upsampling are processed using the RCDL+Transformer method. That is, the features are first processed by multiple layers of RCDL, and then the attention mechanism of Transformer is used to process them. The features are first added with position encoding, and then the query vector Q matrix, key vector K matrix, and numerical vector V matrix are generated using the features of this layer, and attention is calculated.
[0037] S440: It adopts the attention mechanism of Transformer to perform cross-scale from top to bottom, and passes the feature information of different scales to the bottom layer step by step. That is, first upsampling of high-level features, position encoding of input features, then using high-level features to generate K and V, low-level features to generate Q, and finally performing attention calculation to obtain multi-scale temporal features.
[0038] Preferably, S420 specifically comprises:
[0039]
[0040] in, DRCL and Upsample represent the i-th layer features of the second batch, i∈[0,1,2], and DRCL and Upsample represent RCDL and upsampling processing, respectively.
[0041] The specific formula for calculating the attention mechanism in S440 is as follows:
[0042]
[0043] Where, d k It is the number of columns in Q and K, i.e., the vector dimension.
[0044] Preferably, S500 includes:
[0045] S510: Output from branch one and branch two The inputs are fed into a multi-level frequency domain enhancement module, which concatenates, fully connects, and regularizes the two inputs, and increases the feature dimension to the sum of the feature dimensions of branch one and branch two, resulting in the fused features.
[0046] S520: Features A three-stage discrete wavelet transform method is used to obtain multi-level frequency information containing high and low frequencies. Low-frequency features are processed using RCDL and then processed using the Transformer attention mechanism. High-frequency features are processed using multi-layer RCDL. The processed high-frequency and low-frequency features are then subjected to inverse discrete wavelet transform. Finally, a prediction result of size Tx7 is output through a fully connected layer. Each frame corresponds to a 1x7 prediction result, and the one with the highest probability is the final prediction result, where T is the number of video images.
[0047] Preferably, S510 specifically comprises:
[0048]
[0049] In this context, cat, FC, and Norm represent concatenation, fully connected layers, and regularization operations, respectively. This represents the result of the fusion. Preferably, the preset loss function includes a frame loss function, a segment loss function, and a smoothing loss function, specifically:
[0050] L all =L frame +L segment +L smooth
[0051]
[0052] in, This represents the truth value corresponding to the 0th scale; These represent the predicted values of the multi-scale time-domain feature module, the multi-level frequency-domain feature module, and the multi-level frequency-domain feature enhancement module, respectively. β represents the true value and predicted value at the i-th scale; β is a hyperparameter; C is the category of surgical stage.
[0053] The surgical stage recognition method based on time-frequency feature fusion described above adopts a dual-branch structure in its entire algorithm framework. Branch one is used to obtain frequency domain features, and branch two is used to obtain time domain features. The multi-level frequency domain enhancement module fuses the features of branch one and branch two, allowing the two features to complement each other and improve the recognition rate. Attached Figure Description
[0054] Figure 1 This is a flowchart of a surgical stage identification method based on time-frequency feature fusion in one embodiment of the present invention;
[0055] Figure 2 This is a schematic diagram illustrating the principle of a surgical stage identification method based on time-frequency feature fusion in one embodiment of the present invention;
[0056] Figure 3 This is a structural diagram of a multi-scale temporal domain feature module in one embodiment of the present invention;
[0057] Figure 4 This is a schematic diagram of RCDL upsampling in one embodiment of the present invention. Detailed Implementation
[0058] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0059] In one embodiment, such as Figure 1 and Figure 2 As shown, a surgical stage identification method based on time-frequency feature fusion includes the following steps:
[0060] S100: Collect laparoscopic surgical videos, cut them according to a preset frequency to obtain multiple consecutive photos, input the cut photos into a trained ResNet network, extract useful spatial features, and obtain a dataset;
[0061] Furthermore, the surgical video content is a complete laparoscopic surgical procedure with a resolution of 1920×1080. The surgical video is segmented at a frequency of 25fps to obtain multiple consecutive photos, which replace the original video. The segmented photos are input into a trained ResNet network. The feature extracted from each frame is 1x2048. Assuming the number of video images is T, 1x2048xT spatial features are obtained, which can greatly reduce computational resources.
[0062] S200: Construct the TFRCDLFormer (Time-frequency fusion RCDL+Transformer) network. This network adopts a dual-branch structure, including a multilevel frequency domain feature module (MLFDFM), a multi-scale time domain feature module (MSTDFM), and a multilevel frequency domain feature enhancement module. Branch 1 is the multilevel frequency domain feature module, and branch 2 is the multi-scale time domain feature module.
[0063] S300: The multi-level frequency domain feature module is used to receive images in the dataset, perform multi-layer dilated convolutional RCDL processing, and then perform discrete wavelet transform to obtain multi-level frequency information containing high and low frequencies. After performing preset corresponding processing on the high and low frequency information respectively, the inverse discrete wavelet transform is performed to obtain multi-level frequency domain features.
[0064] S400: The multi-scale temporal feature module is used to receive images in the dataset, perform RCDL processing, and then perform average pooling and upsampling processing to obtain feature information at different scales. The feature information at each scale is processed by RCDL and Transformer to obtain multi-scale temporal features.
[0065] S500: The multi-level frequency domain feature enhancement module is used to fuse multi-level frequency domain features and multi-scale time domain features through a fully connected layer, and then perform frequency domain feature enhancement according to the method in S300. After passing through the fully connected layer, the output is the surgical stage recognition and prediction result.
[0066] S600: The TFRCDLFormer network is trained based on a preset loss function. When the preset training termination is reached, the trained TFRCDLFormer network is obtained, and real-time surgical stage recognition is completed.
[0067] In one embodiment, such as Figure 2 As shown, S300 includes:
[0068] S310: Use multi-layer RCDL to obtain the frequency information of images in the dataset;
[0069] S320: Performs three discrete wavelet transforms on the frequency information, and obtains low-frequency and high-frequency features based on high-pass and low-pass filters;
[0070] S330: Low-frequency features are processed using RCDL, and then processed using the Transformer's attention mechanism; high-frequency features are processed using multi-layer RCDL.
[0071] S340: Perform inverse wavelet transform on the processed multi-level frequency features.
[0072] Specifically, discrete wavelet transform is used to obtain frequency features at different levels, which are then processed separately to form four levels of frequency features. Among them, RCDL+Transformer is used to capture low-frequency features more effectively.
[0073] In one embodiment, S310 specifically includes:
[0074] Z l =ReLU(W 1,l *Fl-1 +b 1,l )
[0075] F l =F l-1 +W 2,l *Z l +b 2,l
[0076] Among them, F l Represents the frequency features of the l-th layer, * indicates convolution calculation, W 1,l It is a 1D dilated convolution kernel, W 2,l These are the weights of a 1x1 convolution, b 1,l ,b 2,l It is a paranoid vector;
[0077] The discrete wavelet transform in S320 is specifically as follows:
[0078]
[0079] Where g[n] are the low-pass filter coefficients, h[n] are the high-pass filter coefficients, a(n) are the approximate coefficients after decomposition, and d(n) are the detail coefficients after decomposition. The input frequency information is represented by n, where n represents the length of the discrete input signal, and k is the horizontal coordinate position.
[0080] In one embodiment, the low-frequency features in S330 are processed using RCDL, and then processed using the attention mechanism of Transformer, specifically as follows:
[0081] x = RCDLs(x)
[0082] x = Transformer(x) + x
[0083] Among them, RCDLs and Transformer represent multi-layer RCDL modules and Transformer modules, respectively;
[0084] The inverse wavelet transform in S340 is specifically as follows:
[0085]
[0086] Where g'[n] and h'[n] are the coefficients of the inverse transform filter. This represents the frequency domain characteristics of the output.
[0087] In one embodiment, such as Figure 3 As shown and Figure 4 As shown, S400 includes:
[0088] S410: Utilizes multi-layer RCDL to obtain temporal information of input features, expands its receptive field, and downsamples the sequence through average pooling to obtain feature information at four different scales;
[0089] S420: Employs RCDL+ upsampling for feature upsampling; better captures features between frames.
[0090] S430: To enhance temporal features, each scale of features after upsampling is processed using the RCDL+Transformer method. That is, the features are first processed by multiple layers of RCDL, and then the attention mechanism of Transformer is used to process them. The features are first added with position encoding, and then the query vector Q matrix, key vector K matrix, and numerical vector V matrix are generated using the features of this layer, and attention is calculated.
[0091] S440: It adopts the attention mechanism of Transformer to perform cross-scale from top to bottom, and passes the feature information of different scales to the bottom layer step by step. That is, first upsampling of high-level features, position encoding of input features, then using high-level features to generate K and V, low-level features to generate Q, and finally performing attention calculation to obtain multi-scale temporal features.
[0092] In one embodiment, S420 specifically refers to:
[0093]
[0094] in, DRCL and Upsample represent the i-th layer features of the second batch, i∈[0,1,2], and DRCL and Upsample represent RCDL and upsampling processing, respectively.
[0095] The specific formula for calculating the attention mechanism in S440 is as follows:
[0096]
[0097] Where, d k It is the number of columns in Q and K, i.e., the vector dimension.
[0098] In one embodiment, S500 includes:
[0099] S510: Output from branch one and branch two The inputs are fed into a multi-level frequency domain enhancement module, which concatenates, fully connects, and regularizes the two inputs, and increases the feature dimension to the sum of the feature dimensions of branch one and branch two, resulting in the fused features.
[0100] S520: Features A three-stage discrete wavelet transform method is used to obtain multi-level frequency information containing high and low frequencies. Low-frequency features are processed using RCDL and then processed using the Transformer attention mechanism. High-frequency features are processed using multi-layer RCDL. The processed high-frequency and low-frequency features are then subjected to inverse discrete wavelet transform. Finally, a prediction result of size Tx7 is output through a fully connected layer. Each frame corresponds to a 1x7 prediction result, and the one with the highest probability is the final prediction result, where T is the number of video images.
[0101] In one embodiment, S510 specifically includes:
[0102]
[0103] In this context, cat, FC, and Norm represent concatenation, fully connected layers, and regularization operations, respectively. It represents the result of integration.
[0104] In one embodiment, the preset loss function includes a frame loss function, a segment loss function, and a smoothing loss function, specifically:
[0105] L all =L frame +L segment +L smooth
[0106]
[0107] in, This represents the truth value corresponding to the 0th scale; These represent the predicted values of the multi-scale time-domain feature module, the multi-level frequency-domain feature module, and the multi-level frequency-domain feature enhancement module, respectively. β represents the true value and predicted value at the i-th scale; β is a hyperparameter; C is the category of surgical stage.
[0108] The above provides a detailed description of a surgical stage identification method based on time-frequency feature fusion provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of the invention. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.
Claims
1. A surgical stage identification method based on time-frequency feature fusion, characterized in that, The method includes the following steps: S100: Collect laparoscopic surgical videos, cut them according to a preset frequency to obtain multiple consecutive photos, input the cut photos into a trained ResNet network, extract useful spatial features, and obtain a dataset; S200: Construct the TFRCDLFormer network, which adopts a dual-branch structure, including a multi-level frequency domain feature module, a multi-scale time domain feature module, and a multi-level frequency domain feature enhancement module. Branch 1 is the multi-level frequency domain feature module, and branch 2 is the multi-scale time domain feature module. S300: The multi-level frequency domain feature module receives images from the dataset, performs multi-layer dilated convolutional linearization (RCDL) processing, and then performs discrete wavelet transform to obtain multi-level frequency information containing high and low frequencies. After pre-defined corresponding processing of the high and low frequency information, inverse discrete wavelet transform is performed to obtain multi-level frequency domain features. Here, RCDL is a residual causal dilated temporal convolutional layer. S300 includes: S310: Obtain the frequency information of images in the dataset using multi-layer RCDL; S310 specifically involves: in, Representing the Layer frequency features, * indicates convolution calculation. It is a 1D dilated convolution kernel. These are the weights of a 1x1 convolution. It is a paranoid vector; S320: Performs three discrete wavelet transforms on the frequency information, and obtains low-frequency and high-frequency features based on high-pass and low-pass filters; S330: Low-frequency features are processed using RCDL, followed by Transformer attention mechanism; high-frequency features are processed using multi-layer RCDL. In S330, low-frequency features are processed using RCDL, followed by Transformer attention mechanism, specifically: in, and These represent the multi-layer RCDL module and the Transformer module, respectively. S340: Perform inverse wavelet transform on the processed multi-level frequency features; S400: The multi-scale temporal feature module receives images from the dataset, performs RCDL processing, then average pooling and upsampling to obtain feature information at different scales. For each scale, the feature information is processed using a combination of RCDL and Transformer methods to obtain multi-scale temporal features. S400 includes: S410: Utilizes multi-layer RCDL to obtain temporal information of input features, and downsamples the sequence through average pooling to obtain feature information at four different scales; S420: Feature upsampling is performed using the RCDL+ upsampling method; S430: Features at each scale after upsampling are processed using the RCDL+Transformer method. That is, the features are first processed by multiple layers of RCDL, and then processed by the Transformer attention mechanism. The features are first added with position encoding, and then the query vector Q matrix, key vector K matrix, and numerical vector V matrix are generated using the features of this layer, and attention is calculated. S440: The Transformer attention mechanism is used to perform cross-scale processing from top to bottom, passing feature information from different scales to the bottom layer step by step. That is, first, the high-level features are upsampled, the input features are positionally encoded, then K and V are generated using the high-level features, Q is generated using the low-level features, and finally attention is calculated to obtain multi-scale temporal features. S500: The multi-level frequency domain feature enhancement module is used to fuse multi-level frequency domain features and multi-scale time domain features through a fully connected layer, and then perform frequency domain feature enhancement according to the method in S300. After passing through the fully connected layer, the output is the surgical stage recognition and prediction result. S600: The TFRCDLFormer network is trained based on a preset loss function. When the preset training termination is reached, the trained TFRCDLFormer network is obtained, and real-time surgical stage recognition is completed.
2. The method according to claim 1, characterized in that, The discrete wavelet transform in S320 is specifically as follows: in These are the coefficients of the low-pass filter. These are the coefficients of the high-pass filter. These are the approximate coefficients after decomposition. These are the detail coefficients after decomposition. For the input frequency information, Indicates the length of the discrete input signal. The horizontal axis position.
3. The method according to claim 2, characterized in that, The inverse wavelet transform in S340 is specifically as follows: in, and These are the coefficients of the inverse transform filter. This represents the frequency domain characteristics of the output.
4. The method according to claim 3, characterized in that, The S420 specifically refers to: in, Represents the i-th layer features of the second batch. , These represent RCDL and upsampling processing, respectively. The specific formula for calculating the attention mechanism in S440 is as follows: in, It is the number of columns in Q and K, i.e., the vector dimension.
5. The method according to claim 4, characterized in that, The S500 includes: S510: Output from branch one and branch two The inputs are fed into a multi-level frequency domain enhancement module, which concatenates, fully connects, and regularizes the two inputs, and increases the feature dimension to the sum of the feature dimensions of branch one and branch two, resulting in the fused features. ; S520: Features A three-stage discrete wavelet transform method is used to obtain multi-level frequency information containing high and low frequencies. Low-frequency features are processed using RCDL and then processed using the Transformer attention mechanism. High-frequency features are processed using multi-layer RCDL. The processed high-frequency and low-frequency features are then subjected to inverse discrete wavelet transform. Finally, a prediction result of size Tx7 is output through a fully connected layer. Each frame corresponds to a 1x7 prediction result, and the one with the highest probability is the final prediction result, where T is the number of video images.
6. The method according to claim 5, characterized in that, The S510 specifically refers to: in, The methods include representative splicing, fully connected layers, and regularization operations. It represents the result of integration.
7. The method according to claim 6, characterized in that, The preset loss functions include frame loss function, segment loss function, and smoothing loss function.