Operation stage identification method based on time-frequency feature fusion

Abstract

The invention discloses an operation stage identification method based on time-frequency feature fusion, which is realized based on a constructed TFRCDLFormer network, a multi-level frequency domain feature module performs multi-layer cavity convolution RCDL processing on an input picture, then discrete wavelet transform is performed to obtain multi-level frequency information containing high frequency and low frequency, and the multi-level frequency information is subjected to time-frequency feature fusion. Processing the frequency information and then carrying out discrete wavelet inverse transformation; the multi-scale time domain feature module carries out RCDL processing on an input picture, then carries out average pooling and up-sampling processing to obtain feature information of different scales, and carries out comprehensive processing on the feature information of each scale by adopting RCDL and Transform methods; and the multi-level frequency domain feature enhancement module carries out feature fusion on the multi-level frequency domain features and the multi-scale time domain features, carries out frequency domain feature enhancement on the fused features, and outputs and obtains an operation stage identification prediction result after passing through a full connection layer. And the recognition speed and precision are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a surgical stage recognition method based on time-frequency feature fusion. Background Art

[0002] In the operating room (OR), surgical process analysis is an indispensable task. Surgical stage recognition can provide important input to physicians in the form of early warnings in the event of deviations and anomalies. Another role is to automatically extract surgical protocols, which is important for archiving, education, and postoperative patient monitoring. However, the task of surgical stage recognition is very difficult due to the small differences in different surgical activities, artifacts in surgical records, camera activities, and the influence of factors such as blood and smoke. . Computer-aided intervention (CAI) systems based on machine learning have been developed for surgical process analysis. These methods not only use OR signals but also utilize intraoperative videos captured during laparoscopic surgery. In recent years, deep learning has been used to extract temporal and spatial information from trimmed surgical videos to automatically recognize surgical stage information, such as TCN (Temporal Convolutional Networks), LSTM, Transformer and other network models. Due to the rapid development of deep learning and the easy acquisition of surgical videos, these methods have become the mainstream of surgical stage recognition tasks. They first use the Resnet network to extract the spatial features of the image to reduce the amount of computation, and then obtain the temporal features by inputting them into the sequence model.

[0003] However, although these methods have made great progress in surgical stage recognition, they still have some defects. The RCDL (Residual Causal Dilated temporal convolution Layer) method uses multi-layer dilated convolution to obtain a larger receptive field and improve the long sequence modeling ability of convolution. The LSTM-based method uses the LSTM model to have a stronger long sequence modeling ability than RNN, stores previous features, and obtains longer sequence information. Influenced by the great success of Transformer in the fields of NLP and CV, Transformer-based methods have been widely used in surgical stage recognition tasks. Through multi-head attention, feature information with high correlation is found globally. However, the corresponding disadvantage is that local features are not easy to identify, and longer sequences require greater computing resources.

[0004] In order to solve the above problems, a surgical stage recognition method based on time-frequency feature fusion was proposed. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a surgical stage identification method based on time-frequency feature fusion.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A method for identifying surgical stages based on time-frequency feature fusion, the method comprising the following steps:

[0008] S100: Collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain a data set.

[0009] S200: Construct a TFRCDLFormer network, which adopts a dual-branch structure, including a multi-level frequency domain feature module, a multi-scale time domain feature module and a multi-level frequency domain feature enhancement module, wherein branch one is a multi-level frequency domain feature module and branch two is a multi-scale time domain feature module;

[0010] S300: The multi-level frequency domain feature module is used to receive the images in the data set, perform multi-layer dilated convolution RCDL processing, and then perform discrete wavelet transform to obtain multi-level frequency information including high frequency and low frequency, and perform preset corresponding processing on the high frequency and low frequency information respectively, and then perform discrete wavelet inverse transform to obtain multi-level frequency domain features;

[0011] S400: The multi-scale time domain feature module is used to receive the images in the data set, perform RCDL processing, and then perform average pooling and upsampling processing to obtain feature information of different scales. The feature information of each scale is comprehensively processed using the RCDL and Transformer methods to obtain multi-scale time domain features;

[0012] S500: The multi-level frequency domain feature enhancement module is used to fuse the multi-level frequency domain features and the multi-scale time domain features through full connection, and then enhance the frequency domain features according to the method in S300, and output the surgical stage recognition prediction results after passing through the full connection layer;

[0013] S600: The TFRCDLFormer network is trained based on a preset loss function. When the preset training termination is reached, a trained TFRCDLFormer network is obtained to complete the real-time surgical stage recognition.

[0014] Preferably, S300 includes:

[0015] S310: Obtain frequency information of images in the data set using multi-layer RCDL;

[0016] S320: performing three discrete wavelet transforms on the frequency information, and obtaining low-frequency features and high-frequency features based on a high-pass filter and a low-pass filter;

[0017] S330: Low-frequency features are processed using RCDL and then the Transformer attention mechanism, and high-frequency features are processed using multi-layer RCDL;

[0018] S340: Performing an inverse wavelet transform on the processed multi-level frequency features.

[0019] Preferably, S310 specifically includes:

[0020] Z l =ReLU(W 1,l *F l-1 +b 1,l )

[0021] F l =F l-1 +W 2,l *Z l +b 2,l

[0022] Among them, F l represents the frequency feature of the lth layer, * represents the convolution calculation, W 1,l is a 1D dilated convolution kernel, W 2,l is the weight of the 1x1 convolution, b 1,l ,b 2,l is the bias vector;

[0023] The discrete wavelet transform in S320 is specifically:

[0024]

[0025] Where g[n] is the low-pass filter coefficient, h[n] is the high-pass filter coefficient, a(n) is the approximate coefficient after decomposition, and d(n) is the detail coefficient after decomposition. is the input frequency information, n represents the length of the discrete input signal, and k is the horizontal coordinate position.

[0026] Preferably, the low-frequency features in S330 are processed using RCDL, and then processed using the attention mechanism of Transformer, specifically:

[0027] x=RCDLs(x)

[0028] x=Transformer(x)+x

[0029] Among them, RCDLs and Transformer represent multi-layer RCDL modules and Transformer modules, respectively;

[0030] The specific inverse wavelet transform in S340 is:

[0031]

[0032] where g'[n] and h'[n] are the inverse transform filter coefficients, is the frequency domain feature of the output.

[0033] Preferably, S400 includes:

[0034] S410: using multi-layer RCDL to obtain time domain information of input features, downsampling the sequence by average pooling to obtain feature information of four different scales;

[0035] S420: feature upsampling is performed using the RCDL+upsampling method;

[0036] S430: The features of each scale after upsampling are processed by the RCDL+Transformer method, that is, the features are first processed by multi-layer RCDL, and then processed by the Transformer attention mechanism. The features are firstly added with position encoding, and then the features of this layer are used to generate the query vector Q matrix, the key vector K matrix, and the numerical vector V matrix for attention calculation;

[0037] S440: Transformer's attention mechanism is used to perform cross-scale analysis from top to bottom, and feature information of different scales is passed to the bottom layer step by step. That is, high-level features are first upsampled, input features are positionally encoded, and then high-level features are used to generate K and V, and low-level features are used to generate Q. Finally, attention calculation is performed to obtain multi-scale time domain features.

[0038] Preferably, S420 specifically includes:

[0039]

[0040] in, represents the i-th layer feature of the second batch, i∈[0,1,2], DRCL, Upsample represent RCDL and upsampling processing respectively;

[0041] The calculation formula of the attention mechanism in S440 is as follows:

[0042]

[0043] Among them, d k is the number of columns of Q and K, i.e. the vector dimension.

[0044] Preferably, S500 includes:

[0045] S510: The outputs of branch 1 and branch 2 are connected. The input is sent to the multi-level frequency domain enhancement module, which concatenates, fully connects and regularizes the two inputs, and increases the feature dimension to the sum of the feature dimensions of branch one and branch two to obtain the fused feature.

[0046] S520: Features The discrete wavelet transform method is performed three times to obtain multi-level frequency information including high frequency and low frequency. The low-frequency features are processed by RCDL and then processed by the Transformer's attention mechanism. The high-frequency features are processed by multi-layer RCDL. The processed high-frequency features and low-frequency features are subjected to discrete wavelet inverse transform. Finally, the prediction results of size Tx7 are output through the fully connected layer. Each frame corresponds to a 1x7 prediction result. The one with the highest probability is the final prediction result, where T is the number of video pictures.

[0047] Preferably, S510 specifically includes:

[0048]

[0049] Among them, cat, FC, and Norm represent concatenation, fully connected layers, and regularization operations. Represents the result of fusion. Preferably, the preset loss function includes a frame loss function, a segment loss function and a smoothing loss function, specifically:

[0050] L all =L frame +L segment +L smooth

[0051]

[0052] in, Represents the corresponding true value on the 0th scale; They represent the predicted values ​​of the multi-scale time domain feature module, the multi-level frequency domain feature module, and the multi-level frequency domain feature enhancement module respectively; represents the corresponding true value and predicted value at the i-th scale; β is a hyperparameter; C is the category of the surgical stage.

[0053] The above-mentioned surgical stage recognition method based on time-frequency feature fusion adopts a dual-branch structure in the entire algorithm framework. Branch one is used to obtain frequency domain features, and branch two obtains time domain features. The multi-level frequency domain enhancement module integrates the features of branch one and branch two, allowing the two features to complement each other and improve the recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of a method for identifying surgical stages based on time-frequency feature fusion in one embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of the principle of a surgical stage identification method based on time-frequency feature fusion in one embodiment of the present invention;

[0056] Figure 3 This is a structural diagram of a multi-scale time-domain feature module in one embodiment of the present invention;

[0057] Figure 4 FIG. 4 is a schematic diagram of RCDL upsampling in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0059] In one embodiment, Figure 1 and Figure 2 As shown, a surgical stage recognition method based on time-frequency feature fusion comprises the following steps:

[0060] S100: Collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain a data set.

[0061] Furthermore, the surgical video content is a complete laparoscopic surgery process with a video resolution of 1920×1080. The surgical video is cut at a frequency of 25fps to obtain multiple continuous photos to replace the original video. The cut photos are input into the trained Resnet network. The feature extracted from each frame is 1x2048. Assuming that the number of video pictures is T, the spatial feature of 1x2048xT is obtained, which can greatly reduce the computing resources.

[0062] S200: Construct a TFRCDLFormer (Time-frequency fusion RCDL+Transformer) network, which adopts a dual-branch structure, including a multi-level frequency domain feature module (Multilevel frequency domain feature module, MLFDFM), a multi-scale time domain feature module (Multi-scale time domain feature module, MSTDFM) and a multi-level frequency domain feature enhancement module, where branch one is a multi-level frequency domain feature module and branch two is a multi-scale time domain feature module;

[0063] S300: The multi-level frequency domain feature module is used to receive the images in the data set, perform multi-layer dilated convolution RCDL processing, and then perform discrete wavelet transform to obtain multi-level frequency information including high frequency and low frequency, and perform preset corresponding processing on the high frequency and low frequency information respectively, and then perform discrete wavelet inverse transform to obtain multi-level frequency domain features;

[0064] S400: The multi-scale time domain feature module is used to receive the images in the data set, perform RCDL processing, and then perform average pooling and upsampling processing to obtain feature information of different scales. The feature information of each scale is comprehensively processed using the RCDL and Transformer methods to obtain multi-scale time domain features;

[0065] S500: The multi-level frequency domain feature enhancement module is used to fuse the multi-level frequency domain features and the multi-scale time domain features through full connection, and then enhance the frequency domain features according to the method in S300, and output the surgical stage recognition prediction results after passing through the full connection layer;

[0066] S600: The TFRCDLFormer network is trained based on a preset loss function. When the preset training termination is reached, a trained TFRCDLFormer network is obtained to complete the real-time surgical stage recognition.

[0067] In one embodiment, Figure 2 As shown, S300 includes:

[0068] S310: Obtain frequency information of images in the data set using multi-layer RCDL;

[0069] S320: performing three discrete wavelet transforms on the frequency information, and obtaining low-frequency features and high-frequency features based on a high-pass filter and a low-pass filter;

[0070] S330: Low-frequency features are processed using RCDL and then the Transformer attention mechanism, and high-frequency features are processed using multi-layer RCDL;

[0071] S340: Performing an inverse wavelet transform on the processed multi-level frequency features.

[0072] Specifically, discrete wavelet transform is used to obtain frequency features of different levels, and then they are processed separately and divided into four levels of frequency features. Among them, the low-frequency features can be captured more accurately by using RCDL+Transformer.

[0073] In one embodiment, S310 specifically includes:

[0074] Z l =ReLU(W 1,l *Fl-1 +b 1,l )

[0075] F l =F l-1 +W 2,l *Z l +b 2,l

[0076] Among them, F l represents the frequency feature of the lth layer, * represents the convolution calculation, W 1,l is a 1D dilated convolution kernel, W 2,l is the weight of the 1x1 convolution, b 1,l ,b 2,l is the bias vector;

[0077] The discrete wavelet transform in S320 is specifically:

[0078]

[0079] Where g[n] is the low-pass filter coefficient, h[n] is the high-pass filter coefficient, a(n) is the approximate coefficient after decomposition, and d(n) is the detail coefficient after decomposition. is the input frequency information, n represents the length of the discrete input signal, and k is the horizontal coordinate position.

[0080] In one embodiment, the low-frequency features in S330 are processed using RCDL and then processed using the attention mechanism of Transformer, specifically:

[0081] x=RCDLs(x)

[0082] x=Transformer(x)+x

[0083] Among them, RCDLs and Transformer represent multi-layer RCDL modules and Transformer modules, respectively;

[0084] The specific inverse wavelet transform in S340 is:

[0085]

[0086] where g'[n] and h'[n] are the inverse transform filter coefficients, is the frequency domain feature of the output.

[0087] In one embodiment, Figure 3 Shown and Figure 4 As shown, S400 includes:

[0088] S410: Use multi-layer RCDL to obtain the time domain information of the input features, expand its receptive field, downsample the sequence by average pooling, and obtain feature information of four different scales;

[0089] S420: Use RCDL+ upsampling method to perform feature upsampling to better obtain features between frames;

[0090] S430: In order to enhance the temporal features, the features of each scale after upsampling are processed by the RCDL+Transformer method, that is, the features are first processed by multi-layer RCDL, and then processed by the Transformer attention mechanism. The features are firstly added with position encoding, and then the features of this layer are used to generate the query vector Q matrix, the key vector K matrix, and the numerical vector V matrix for attention calculation;

[0091] S440: Transformer's attention mechanism is used to perform cross-scale analysis from top to bottom, and feature information of different scales is passed to the bottom layer step by step. That is, high-level features are first upsampled, input features are positionally encoded, and then high-level features are used to generate K and V, and low-level features are used to generate Q. Finally, attention calculation is performed to obtain multi-scale time domain features.

[0092] In one embodiment, S420 specifically includes:

[0093]

[0094] in, represents the i-th layer feature of the second batch, i∈[0,1,2], DRCL, Upsample represent RCDL and upsampling processing respectively;

[0095] The calculation formula of the attention mechanism in S440 is as follows:

[0096]

[0097] Among them, d k is the number of columns of Q and K, i.e. the vector dimension.

[0098] In one embodiment, S500 includes:

[0099] S510: The outputs of branch 1 and branch 2 are connected. The input is sent to the multi-level frequency domain enhancement module, which concatenates, fully connects and regularizes the two inputs, and increases the feature dimension to the sum of the feature dimensions of branch one and branch two to obtain the fused feature.

[0100] S520: Features The discrete wavelet transform method is performed three times to obtain multi-level frequency information including high frequency and low frequency. The low-frequency features are processed by RCDL and then processed by the Transformer's attention mechanism. The high-frequency features are processed by multi-layer RCDL. The processed high-frequency features and low-frequency features are subjected to discrete wavelet inverse transform. Finally, the prediction results of size Tx7 are output through the fully connected layer. Each frame corresponds to a 1x7 prediction result. The one with the highest probability is the final prediction result, where T is the number of video pictures.

[0101] In one embodiment, S510 specifically includes:

[0102]

[0103] Among them, cat, FC, and Norm represent concatenation, fully connected layers, and regularization operations. Represents the result of fusion.

[0104] In one embodiment, the preset loss function includes a frame loss function, a segment loss function and a smoothing loss function, specifically:

[0105] L all =L frame +L segment +L smooth

[0106]

[0107] in, Represents the corresponding true value on the 0th scale; They represent the predicted values ​​of the multi-scale time domain feature module, the multi-level frequency domain feature module, and the multi-level frequency domain feature enhancement module respectively; represents the corresponding true value and predicted value at the i-th scale; β is a hyperparameter; C is the category of the surgical stage

[0108] The above is a detailed introduction to a surgical stage identification method based on time-frequency feature fusion provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A surgical stage recognition method based on time-frequency feature fusion, characterized in that: The method comprises the following steps: S100: Collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain a data set. S200: Construct a TFRCDLFormer network, which adopts a dual-branch structure, including a multi-level frequency domain feature module, a multi-scale time domain feature module and a multi-level frequency domain feature enhancement module, wherein branch one is a multi-level frequency domain feature module and branch two is a multi-scale time domain feature module; S300: The multi-level frequency domain feature module is used to receive the images in the data set, perform multi-layer dilated convolution RCDL processing, and then perform discrete wavelet transform to obtain multi-level frequency information including high frequency and low frequency, and perform preset corresponding processing on the high frequency and low frequency information respectively, and then perform discrete wavelet inverse transform to obtain multi-level frequency domain features; S400: The multi-scale time domain feature module is used to receive the images in the data set, perform RCDL processing, and then perform average pooling and upsampling processing to obtain feature information of different scales. The feature information of each scale is comprehensively processed using the RCDL and Transformer methods to obtain multi-scale time domain features; S500: The multi-level frequency domain feature enhancement module is used to fuse the multi-level frequency domain features and the multi-scale time domain features through full connection, and then enhance the frequency domain features according to the method in S300, and output the surgical stage recognition prediction results after passing through the full connection layer; S600: The TFRCDLFormer network is trained based on a preset loss function. When the preset training termination is reached, a trained TFRCDLFormer network is obtained to complete the real-time surgical stage recognition.

2. The method according to claim 1, characterized in that S300 includes: S310: Obtain frequency information of images in the data set using multi-layer RCDL; S320: performing three discrete wavelet transforms on the frequency information, and obtaining low-frequency features and high-frequency features based on a high-pass filter and a low-pass filter; S330: Low-frequency features are processed using RCDL and then the Transformer attention mechanism, and high-frequency features are processed using multi-layer RCDL; S340: Performing an inverse wavelet transform on the processed multi-level frequency features.

3. The method according to claim 2, characterized in that S310 is specifically: WITH l =ReLU(W 1,l *F l-1 +b 1,l ) F l =F l-1 +W 2,l *Z l +b 2,l Among them, F l represents the frequency feature of the lth layer, * represents the convolution calculation, W 1,l is a 1D dilated convolution kernel, W 2,l is the weight of the 1x1 convolution, b 1,l ,b 2,l is the bias vector; The discrete wavelet transform in S320 is specifically: Where g[n] is the low-pass filter coefficient, h[n] is the high-pass filter coefficient, a(n) is the approximate coefficient after decomposition, and d(n) is the detail coefficient after decomposition. is the input frequency information, n represents the length of the discrete input signal, and k is the horizontal coordinate position.

4. The method according to claim 3, characterized in that The low-frequency features in S330 are processed using RCDL, and then processed using the Transformer attention mechanism, specifically: x=RCDLs(x) x=Transformer(x)+x Among them, RCDLs and Transformer represent multi-layer RCDL modules and Transformer modules, respectively; The specific inverse wavelet transform in S340 is: where g'[n] and h'[n] are the inverse transform filter coefficients, is the frequency domain feature of the output.

5. The method according to claim 4, characterized in that S400 includes: S410: using multi-layer RCDL to obtain time domain information of input features, downsampling the sequence by average pooling to obtain feature information of four different scales; S420: feature upsampling is performed using the RCDL+upsampling method; S430: The features of each scale after upsampling are processed by the RCDL+Transformer method, that is, the features are first processed by multi-layer RCDL, and then processed by the Transformer attention mechanism. The features are firstly added with position encoding, and then the features of this layer are used to generate the query vector Q matrix, the key vector K matrix, and the numerical vector V matrix for attention calculation; S440: Transformer's attention mechanism is used to perform cross-scale analysis from top to bottom, and feature information of different scales is passed to the bottom layer step by step. That is, high-level features are first upsampled, input features are positionally encoded, and then high-level features are used to generate K and V, and low-level features are used to generate Q. Finally, attention calculation is performed to obtain multi-scale time domain features.

6. The method according to claim 5, characterized in that S420 is specifically: in, represents the i-th layer feature of the second batch, i∈[0,1,2], DRCL, Upsample represent RCDL and upsampling processing respectively; The calculation formula of the attention mechanism in S440 is as follows: Among them, d k is the number of columns of Q and K, i.e. the vector dimension.

7. The method according to claim 6, characterized in that S500 includes: S510: The outputs of branch 1 and branch 2 are connected. The input is sent to the multi-level frequency domain enhancement module, which concatenates, fully connects and regularizes the two inputs, and increases the feature dimension to the sum of the feature dimensions of branch one and branch two to obtain the fused feature. S520: Features The discrete wavelet transform method is performed three times to obtain multi-level frequency information including high frequency and low frequency. The low-frequency features are processed by RCDL and then processed by the Transformer's attention mechanism. The high-frequency features are processed by multi-layer RCDL. The processed high-frequency features and low-frequency features are subjected to discrete wavelet inverse transform. Finally, the prediction results of size Tx7 are output through the fully connected layer. Each frame corresponds to a 1x7 prediction result. The one with the highest probability is the final prediction result, where T is the number of video pictures.

8. The method according to claim 7, characterized in that S510 is specifically: Among them, cat, FC, and Norm represent concatenation, fully connected layers, and regularization operations. Represents the result of fusion.

9. The method according to claim 8, characterized in that The preset loss functions include frame loss function, segment loss function and smoothing loss function, specifically: L all =L frame +L segment +L smooth in, Represents the corresponding true value on the 0th scale; They represent the predicted values ​​of the multi-scale time domain feature module, the multi-level frequency domain feature module, and the multi-level frequency domain feature enhancement module respectively; represents the corresponding true value and predicted value at the i-th scale; β is a hyperparameter; C is the category of the surgical stage.

Images