Operation stage identification method, system and equipment based on time-frequency hybrid network

Abstract

The invention discloses an operation stage identification method, system and device based on a time-frequency hybrid network, the time-frequency hybrid network is provided, the time-frequency hybrid network comprises a prediction stage and a refining stage, the prediction stage and the refining stage each comprise a single-stage time sequence convolution module, a wavelet time sequence convolution module and a Fourier attention module, and the wavelet time sequence convolution module is used for carrying out time sequence convolution on the single-stage time sequence convolution module. According to the method, a large receptive field can be obtained at a small cost, the method has higher frequency domain feature extraction capability than a common single-stage time sequence convolution module, the single-stage time sequence convolution module and a wavelet time sequence convolution module are combined, and local feature details and filtering can be better obtained; the Fourier attention module is provided, the Fourier attention module is formed by connecting a Fourier filtering module and an attention module in series, the Fourier filtering module can filter out unimportant features, the subsequent attention module can better model the global range, and the recognition efficiency in the operation stage is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method, system and device for identifying surgical stages based on a time-frequency hybrid network. Background Art

[0002] The emergence of computer-assisted surgery (CAS) provides a new solution to the bottleneck of minimally invasive surgery. CAS uses computer technology, including machine vision, image processing and analysis, communication transmission and surgical medicine, to obtain richer information and assist doctors to better complete the operation. Surgical stage recognition and surgical tool detection are key technologies in computer-assisted surgery. Unlike offline recognition, surgical stage recognition requires real-time recognition of current activities without utilizing future frames. Specifically, surgical stage recognition can alert doctors early in the event of deviations or abnormalities in the operation. In addition, it can also be used to automatically extract surgical protocols, which is of great significance for archiving, education and postoperative patient detection. However, due to the changes in physician skills, changes in patient conditions, the quality of video recordings, non-uniform lighting, blood and smoke in the surgical field, and other factors, the task of accurate surgical stage recognition has become challenging.

[0003] Influenced by the rapid development of deep learning, many researchers have applied deep learning methods to surgical stage recognition tasks. Some methods based on CNN, TCN (Temporal Convolutional Network), LSTM and Transformer have been applied to surgical stage recognition tasks. The CNN method mainly extracts spatial features without considering the correlation between frames. The TCN method can obtain a larger receptive field at a lower cost by stacking multiple layers of residual causal dilated convolutions. LSTM has a better performance in obtaining long sequence information. The Transformer, which is widely used in the fields of NLP and CV, can obtain global feature information well because of its multi-head attention mechanism, but it also has the problem of high computational complexity. How to obtain better surgical stage recognition rate at a lower computational cost is a problem worth studying.

[0004] In order to solve the above problems, a surgical stage identification method, system and device based on time-frequency hybrid network are proposed. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a surgical stage identification method, system and device based on a time-frequency hybrid network.

[0006] The technical solution adopted by the present invention to solve the technical problem is: A method for identifying surgical stages based on a time-frequency hybrid network, the method comprising the following steps: S100: Collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain a data set. S200: construct a time-frequency hybrid network, including a prediction stage and a refinement stage, both of which include a single-stage time series convolution module, a wavelet time series convolution module, and a Fourier attention module; wherein the composition and structure of the modules in the prediction stage and the refinement stage are consistent, but the parameters are not shared; S300: Input the data set into the single-stage temporal convolutional network of the prediction stage, use multi-layer residual causal dilated convolution to expand the receptive field, and obtain the temporal feature information of each layer in the image; S400: Input the data set into the wavelet time series convolution module of the prediction stage, obtain multi-level frequency information through discrete wavelet transform, perform a layer of residual causal convolution on the frequency information of the last layer, and then perform discrete wavelet inverse transform, replace the high-frequency features of the previous layer with the discrete wavelet inverse transform results of the next layer, discard the high-frequency features of other layers, and obtain the frequency feature information with some high-frequency features filtered out; S500: fusing the temporal feature information and the frequency feature information in the image and inputting the information into the Fourier attention module for Fourier filtering and attention operation to obtain the output of the prediction stage; S600: using the output of the prediction stage as the input of the refinement stage, repeating the steps from S300 to S500, mapping the obtained features to the dimension of the category through the fully connected layer, and outputting the surgical stage recognition prediction result; S700: The time-frequency hybrid network is trained based on a preset loss function. When the preset training termination is reached, a trained time-frequency hybrid network is obtained, and real-time surgical laparoscopic surgery video is acquired. After the features are extracted by the trained Resnet network, they are input into the trained time-frequency hybrid network to complete the real-time surgical stage recognition.

[0007] Preferably, S100 includes: S110: Collect surgical laparoscopic surgery videos, cut the surgery videos at a frequency of 1 fps, obtain multiple continuous photos, and replace the original videos; S120: training the Resnet network, inputting the cut photos into the Resnet network, using the Resnet network to extract spatial features of the pictures, using the annotated labels for back propagation, obtaining better Resnet model parameters, and obtaining a trained Resnet network.

[0008] S130: Input all the cut photos into the trained Resnet network to extract useful spatial features and obtain a data set.

[0009] Preferably, S300 includes: The dataset is input into the single-stage temporal convolutional network in the prediction phase, and the features of the previous layer are input into the residual causal void convolution of the current layer. The activation function is then passed through a residual causal dilated convolution and the features of the previous layer are added to obtain the temporal feature information of the current layer, specifically: ; in, Representative The temporal characteristic information of the layer, represents the convolution calculation, is a 1D residual causal dilated convolution kernel, is a 1x1 residual causal atrous convolution kernel, is the bias vector, Representative The temporal characteristic information of the layer, is an intermediate variable.

[0010] Preferably, S400 specifically includes: S410: Input the data set into the wavelet time series convolution module in the prediction phase, and perform multi-level discrete wavelet transform on the input features. Each transformation inputs the low-frequency features of the previous layer. , we get multi-level frequency features, specifically: ; in, stands for discrete wavelet transform, and Respectively represent The low-frequency and high-frequency characteristics of the layer; S420: Perform a layer of residual causal convolution operation on the high frequency and low frequency of the last layer of discrete wavelet transform, respectively, as follows: ; in, represents the number of discrete wavelet layers, represents residual causal convolution, and They represent the high-frequency and low-frequency features after residual causal convolution respectively; S430: In order to remove the interference of some high-frequency features, except for retaining the high-frequency features of the Kth layer, the high-frequency features of the remaining layers are discarded, and the high-frequency features of the i-1 layer are replaced by the inverse discrete wavelet transform results of the i layer, as follows: ; in, Stands for Inverse Discrete Wavelet Transform.

[0011] Preferably, the Fourier attention module includes a Fourier filter module and an attention module, the Fourier filter module includes a first normalization layer, a Fourier transform layer, an inverse Fourier transform layer, a second normalization layer and a fully connected layer, the attention module includes a first normalization layer, a multi-head attention mechanism, a second normalization layer and a multilayer perceptron, and S500 includes: S510: splicing the time series feature information and the frequency feature information in the image and then passing them through a fully connected layer to obtain fused features; S520: Input the fused features into the Fourier attention module. First, the fused features are normalized by the first normalization layer and then input into the Fourier transform layer. The Fourier transform is performed in combination with the learnable weights. The features after the Fourier transform are input into the inverse Fourier transform layer for inverse Fourier transform. The features after the inverse Fourier transform are input into the second normalization layer for normalization and then pass through the fully connected layer. The output of the fully connected layer is added with the fused features to obtain the output of the Fourier filter module. S530: The output of the Fourier filter module is input to the attention module. First, the output of the Fourier filter module is normalized by the first normalization layer, and then input into the multi-head attention mechanism for attention operation, and then input into the second normalization layer for normalization, and then feature mapping is performed through the multi-layer perceptron, and the input of the attention module is added to obtain the output result of the attention module as the output of the prediction stage.

[0012] Preferably, S520 specifically includes: ; ; ; ; S530 is specifically: ; in, represents the input of the Fourier module, represent Fourier transform and inverse Fourier transform respectively, represents normalization, are learnable weights, is a fully connected layer, is a multi-head attention module, is a multi-layer perceptron, , , , and They represent Fourier transform features, learnable weight features, inverse Fourier transform features, Fourier filter features, and attention features respectively.

[0013] Preferably, the preset loss function includes a frame loss function and a smoothing loss function, specifically: ; ; ; in, represents the frame loss function, represents the smooth loss function, Represents the true value corresponding to class c in the tth frame; Represent the predicted values ​​of the prediction stage and the refinement stage respectively; C is the category of the surgical stage, Represents the number of frames.

[0014] Preferably, in S700, real-time laparoscopic surgery video is obtained and input into a trained time-frequency hybrid network to complete real-time surgery stage recognition, specifically: ; ; ; ; ; The refining stage is: ; ; ; ; ; in, Represent single-stage temporal convolution and wavelet temporal convolution, respectively. Represents splicing, represents the fully connected layer, represents the Fourier attention module; They represent the single-stage time series characteristics, wavelet time series characteristics, time series characteristics and output in the prediction stage respectively; They represent the single-stage time series characteristics, wavelet time series characteristics, time series characteristics and output of the refining stage respectively.

[0015] A surgical stage recognition system based on a time-frequency hybrid network, comprising a data set acquisition module, a time-frequency hybrid network construction module, a time series feature information acquisition module, a frequency feature information acquisition module, a prediction stage result prediction module, a refinement stage result prediction module and a surgical stage recognition module; The data set acquisition module is used to collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain the data set. A time-frequency hybrid network building module is used to build a time-frequency hybrid network, including a prediction stage and a refinement stage. Both the prediction stage and the refinement stage include a single-stage time series convolution module, a wavelet time series convolution module, and a Fourier attention module. The modules in the prediction stage and the refinement stage have the same composition and structure, but do not share parameters. The temporal feature information acquisition module is used to input the data set into the single-stage temporal convolutional network in the prediction stage, and use multi-layer residual causal dilated convolution to expand the receptive field to obtain the temporal feature information of each layer in the image; The frequency feature information acquisition module is used to input the data set into the wavelet time series convolution module in the prediction stage, obtain multi-level frequency information through discrete wavelet transform, perform a layer of residual causal convolution on the frequency information of the last layer, and then perform discrete wavelet inverse transform, replace the high-frequency features of the previous layer with the discrete wavelet inverse transform results of the next layer, discard the high-frequency features of other layers, and obtain the frequency feature information with some high-frequency features filtered out; The prediction stage result prediction module is used to fuse the temporal feature information and frequency feature information in the image and input them into the Fourier attention module of the prediction stage for Fourier filtering and attention operation to obtain the output of the prediction stage; The result prediction module of the refining stage is used to use the output of the prediction stage as the input of the refining stage, repeatedly obtain the time series feature information and the frequency feature information, fuse the time series feature information and the frequency feature information in the image, and input them into the Fourier attention module of the refining stage for Fourier filtering and attention operation. The obtained features are mapped to the dimension of the category through the fully connected layer, and the recognition prediction result of the surgical stage is output; The surgical stage recognition module is used to train the time-frequency hybrid network based on a preset loss function. When the preset training termination is reached, the trained time-frequency hybrid network is obtained, and the real-time surgical laparoscopic surgery video is obtained and input into the trained time-frequency hybrid network to complete the real-time surgical stage recognition.

[0016] A device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a surgical stage identification method based on a time-frequency hybrid network when executing the computer program.

[0017] The above-mentioned surgical stage recognition method, system and equipment based on time-frequency hybrid network proposes a wavelet time series convolution module, which can obtain a larger receptive field at a lower cost. It has a stronger frequency domain feature extraction capability than the commonly used single-stage time series convolution module. The present invention combines the single-stage time series convolution module with the wavelet time series convolution module to better obtain local feature details and filtering; proposes a Fourier attention module FFA, which is composed of a Fourier filter module and an attention module in series. The Fourier filter module can filter out unimportant features, allowing the subsequent attention module to better model the global range and improve the efficiency of surgical stage recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for identifying surgical stages based on a time-frequency hybrid network in one embodiment of the present invention; Figure 2 It is a schematic diagram of the principle of a time-frequency hybrid network in one embodiment of the present invention; Figure 3 A flowchart of a multi-level discrete wavelet transform process according to an embodiment of the present invention; Figure 4 is a schematic diagram of a Fourier filter module in one embodiment of the present invention; Figure 5 Schematic diagram of an attention module in one embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0020] In one embodiment, Figure 1 and Figure 2 As shown, a surgical stage identification method based on a time-frequency hybrid network comprises the following steps: S100: Collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain a data set. S200: construct a time-frequency hybrid network, including a prediction stage and a refinement stage, both of which include a single-stage time series convolution module, a wavelet time series convolution module, and a Fourier attention module; wherein the composition and structure of the modules in the prediction stage and the refinement stage are consistent, but the parameters are not shared; S300: Input the data set into the single-stage temporal convolutional network of the prediction stage, use multi-layer residual causal dilated convolution to expand the receptive field, and obtain the temporal feature information of each layer in the image; S400: Input the data set into the wavelet time series convolution module of the prediction stage, obtain multi-level frequency information through discrete wavelet transform, perform a layer of residual causal convolution on the frequency information of the last layer, and then perform discrete wavelet inverse transform, replace the high-frequency features of the previous layer with the discrete wavelet inverse transform results of the next layer, discard the high-frequency features of other layers, and obtain the frequency feature information with some high-frequency features filtered out; S500: fusing the temporal feature information and the frequency feature information in the image and inputting them into the Fourier attention module for Fourier filtering and attention operation to obtain the output of the prediction stage; S600: using the output of the prediction stage as the input of the refinement stage, repeating the steps from S300 to S500, mapping the obtained features to the dimension of the category through the fully connected layer, and outputting the surgical stage recognition prediction result; S700: Train the time-frequency hybrid network based on a preset loss function. When the preset training termination is reached, a trained time-frequency hybrid network is obtained, and real-time surgical laparoscopic surgery video is acquired and input into the trained time-frequency hybrid network to complete real-time surgical stage recognition.

[0021] In one embodiment, S100 includes: S110: Collect surgical laparoscopic surgery videos, cut the surgery videos at a frequency of 1 fps, obtain multiple continuous photos, and replace the original videos; S120: training the Resnet network, inputting the cut photos into the Resnet network, using the Resnet network to extract spatial features of the pictures, using the annotated labels for back propagation, obtaining better Resnet model parameters, and obtaining a trained Resnet network.

[0022] S130: Input all the cut photos into the trained Resnet network to extract useful spatial features and obtain a data set.

[0023] In one embodiment, S300 includes: The dataset is input into the single-stage temporal convolutional network in the prediction phase, and the features of the previous layer are input into the residual causal void convolution of the current layer. The activation function is then passed through a residual causal dilated convolution and the features of the previous layer are added to obtain the temporal feature information of the current layer, specifically: ; in, Representative The temporal characteristic information of the layer, represents the convolution calculation, is a 1D residual causal dilated convolution kernel, is a 1x1 residual causal atrous convolution kernel, is the bias vector, Representative The temporal characteristic information of the layer, is an intermediate variable.

[0024] Furthermore, the receptive field grows exponentially with the number of layers, specifically: ; Represents the receptive field. Therefore, a large receptive field can be achieved by stacking multiple layers.

[0025] Furthermore, the wavelet time series convolution module, unlike the single-stage time series convolution module which obtains the receptive field through the method of dilated convolution, obtains the receptive field through discrete wavelet transform.

[0026] In one embodiment, Figure 3 As shown, S400 is specifically: S410: Input the data set into the wavelet time series convolution module in the prediction phase, and perform multi-level discrete wavelet transform on the input features. Each transformation inputs the low-frequency features of the previous layer. , we get multi-level frequency features, specifically: ; in, stands for discrete wavelet transform, and Respectively represent The low-frequency and high-frequency characteristics of the layer; S420: Perform a layer of residual causal convolution operation on the high frequency and low frequency of the last layer of discrete wavelet transform, respectively, as follows: ; in, represents the number of discrete wavelet layers, represents residual causal convolution, and They represent the high-frequency and low-frequency features after residual causal convolution respectively; S430: In order to remove the interference of some high-frequency features, except for retaining the high-frequency features of the Kth layer, the high-frequency features of the remaining layers are discarded, and the high-frequency features of the i-1 layer are replaced by the inverse discrete wavelet transform results of the i layer, as follows: ; in, Stands for Inverse Discrete Wavelet Transform.

[0027] In one embodiment, Figure 4 and Figure 5As shown, the Fourier attention module includes a Fourier filter module and an attention module. The Fourier filter module includes a first normalization layer, a Fourier transform layer, an inverse Fourier transform layer, a second normalization layer and a fully connected layer. The attention module includes a first normalization layer, a multi-head attention mechanism, a second normalization layer and a multi-layer perceptron. S500 includes: S510: splicing the time series feature information and the frequency feature information in the image and then passing them through a fully connected layer to obtain fused features; S520: Input the fused features into the Fourier attention module. First, the fused features are normalized by the first normalization layer and then input into the Fourier transform layer. The Fourier transform is performed in combination with the learnable weights. The features after the Fourier transform are input into the inverse Fourier transform layer for inverse Fourier transform. The features after the inverse Fourier transform are input into the second normalization layer for normalization and then pass through the fully connected layer. The output of the fully connected layer is added with the fused features to obtain the output of the Fourier filter module. S530: The output of the Fourier filter module is input to the attention module. First, the output of the Fourier filter module is normalized by the first normalization layer, and then input into the multi-head attention mechanism for attention operation, and then input into the second normalization layer for normalization, and then feature mapping is performed through the multi-layer perceptron, and the input of the attention module is added to obtain the output result of the attention module as the output of the prediction stage.

[0028] Specifically, the Fourier attention module is used to convert the features into the frequency domain, and then multiply them with a learnable weight to filter out unimportant features, and then the attention module is used for global modeling.

[0029] In one embodiment, S520 specifically includes: ; ; ; ; S530 is specifically: ; in, represents the input of the Fourier module, represent Fourier transform and inverse Fourier transform respectively, represents normalization, are learnable weights, is a fully connected layer, is a multi-head attention module, is a multi-layer perceptron, , , , and They represent Fourier transform features, learnable weight features, inverse Fourier transform features, Fourier filter features, and attention features respectively.

[0030] In one embodiment, the preset loss function includes a frame loss function and a smoothing loss function, specifically: ; ; ; in, represents the frame loss function, represents the smooth loss function, Represents the true value corresponding to class c in the tth frame; Represent the predicted values ​​of the prediction stage and the refinement stage respectively; C is the category of the surgical stage, Represents the number of frames.

[0031] In one embodiment, in S700, a real-time laparoscopic surgery video is obtained, and features are extracted through a trained Resnet network and then input into a trained time-frequency hybrid network to complete real-time surgery stage recognition, specifically: ; ; ; ; ; The refining stage is: ; ; ; ; ; in, Represent single-stage temporal convolution and wavelet temporal convolution, respectively. Represents splicing, represents the fully connected layer, represents the Fourier attention module; They represent the single-stage time series characteristics, wavelet time series characteristics, time series characteristics and output in the prediction stage respectively; They represent the single-stage time series characteristics, wavelet time series characteristics, time series characteristics and output of the refining stage respectively.

[0032] A surgical stage recognition system based on a time-frequency hybrid network, comprising a data set acquisition module, a time-frequency hybrid network construction module, a time series feature information acquisition module, a frequency feature information acquisition module, a prediction stage result prediction module, a refinement stage result prediction module and a surgical stage recognition module; The data set acquisition module is used to collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain the data set. A time-frequency hybrid network building module is used to build a time-frequency hybrid network, including a prediction stage and a refinement stage. Both the prediction stage and the refinement stage include a single-stage time series convolution module, a wavelet time series convolution module, and a Fourier attention module. The modules in the prediction stage and the refinement stage have the same composition and structure, but do not share parameters. The temporal feature information acquisition module is used to input the data set into the single-stage temporal convolutional network in the prediction stage, and use multi-layer residual causal dilated convolution to expand the receptive field to obtain the temporal feature information of each layer in the image; The frequency feature information acquisition module is used to input the data set into the wavelet time series convolution module in the prediction stage, obtain multi-level frequency information through discrete wavelet transform, perform a layer of residual causal convolution on the frequency information of the last layer, and then perform discrete wavelet inverse transform, replace the high-frequency features of the previous layer with the discrete wavelet inverse transform results of the next layer, discard the high-frequency features of other layers, and obtain the frequency feature information with some high-frequency features filtered out; The prediction stage result prediction module is used to fuse the temporal feature information and frequency feature information in the image and input them into the Fourier attention module of the prediction stage for Fourier filtering and attention operation to obtain the output of the prediction stage; The result prediction module of the refining stage is used to use the output of the prediction stage as the input of the refining stage, repeatedly obtain the time series feature information and the frequency feature information, fuse the time series feature information and the frequency feature information in the image, and input them into the Fourier attention module of the refining stage for Fourier filtering and attention operation. The obtained features are mapped to the dimension of the category through the fully connected layer, and the recognition prediction result of the surgical stage is output; The surgical stage recognition module is used to train the time-frequency hybrid network based on a preset loss function. When the preset training termination is reached, the trained time-frequency hybrid network is obtained, and the real-time surgical laparoscopic surgery video is obtained. After the features are extracted by the trained Resnet network, they are input into the trained time-frequency hybrid network to complete the real-time surgical stage recognition.

[0033] For the specific definition of a surgical stage identification device based on a time-frequency hybrid network, please refer to the definition of a surgical stage identification method based on a time-frequency hybrid network above, which will not be repeated here. Each module in the above-mentioned surgical stage identification device based on a time-frequency hybrid network can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0034] A device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a surgical stage identification method based on a time-frequency hybrid network when executing the computer program.

[0035] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0036] The above-mentioned surgical stage recognition method, system and equipment based on time-frequency hybrid network proposes a wavelet time series convolution module, which can obtain a larger receptive field at a lower cost. It has a stronger frequency domain feature extraction capability than the commonly used single-stage time series convolution module. The present invention combines the single-stage time series convolution module with the wavelet time series convolution module to better obtain local feature details and filtering effects; proposes a Fourier attention module FFA, which is composed of a Fourier filter module and an attention module in series. The Fourier filter module can filter out unimportant features, allowing the subsequent attention module to better model the global range and improve the efficiency of surgical stage recognition.

[0037] The above is a detailed introduction to a surgical stage identification method, system and device based on a time-frequency hybrid network provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A surgical stage recognition method based on a time-frequency hybrid network, characterized in that: The method comprises the following steps: S100: Collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain a data set. S200: construct a time-frequency hybrid network, including a prediction stage and a refinement stage, both of which include a single-stage time series convolution module, a wavelet time series convolution module, and a Fourier attention module; wherein the composition and structure of the modules in the prediction stage and the refinement stage are consistent, but the parameters are not shared; S300: Input the data set into the single-stage temporal convolutional network of the prediction stage, use multi-layer residual causal dilated convolution to expand the receptive field, and obtain the temporal feature information of each layer in the image; S400: Input the data set into the wavelet time series convolution module of the prediction stage, obtain multi-level frequency information through discrete wavelet transform, perform a layer of residual causal convolution on the frequency information of the last layer, and then perform discrete wavelet inverse transform, replace the high-frequency features of the previous layer with the discrete wavelet inverse transform results of the next layer, discard the high-frequency features of other layers, and obtain the frequency feature information with some high-frequency features filtered out; S500: fusing the temporal feature information and the frequency feature information in the image and inputting the information into the Fourier attention module for Fourier filtering and attention operation to obtain the output of the prediction stage; S600: using the output of the prediction stage as the input of the refinement stage, repeating the steps from S300 to S500, mapping the obtained features to the dimension of the category through the fully connected layer, and outputting the surgical stage recognition prediction result; S700: The time-frequency hybrid network is trained based on a preset loss function. When the preset training termination is reached, a trained time-frequency hybrid network is obtained, and real-time surgical laparoscopic surgery video is acquired. After the features are extracted by the trained Resnet network, they are input into the trained time-frequency hybrid network to complete the real-time surgical stage recognition.

2. The method according to claim 1, characterized in that S100 includes: S110: Collect surgical laparoscopic surgery videos, cut the surgery videos at a frequency of 1 fps, obtain multiple continuous photos, and replace the original videos; S120: training the Resnet network, inputting the cut photos into the Resnet network, using the Resnet network to extract the spatial features of the photos, using the annotated labels to perform back propagation, obtaining better Resnet model parameters, and obtaining a trained Resnet network; S130: Input all the cut photos into the trained Resnet network to extract useful spatial features and obtain a data set.

3. The method according to claim 2, characterized in that S300 includes: The dataset is input into the single-stage temporal convolutional network in the prediction phase, and the features of the previous layer are input into the residual causal void convolution of the current layer. The activation function is then passed through a residual causal dilated convolution and the features of the previous layer are added to obtain the temporal feature information of the current layer, specifically: ； in, Representative The temporal characteristic information of the layer, represents the convolution calculation, is a 1D residual causal dilated convolution kernel, is a 1x1 residual causal dilated convolution kernel, is the bias vector, Representative The temporal characteristic information of the layer, is an intermediate variable.

4. The method according to claim 3, characterized in that S400 is specifically: S410: Input the data set into the wavelet time series convolution module in the prediction phase, and perform multi-level discrete wavelet transform on the input features. Each transformation inputs the low-frequency features of the previous layer. , we get multi-level frequency features, specifically: ； in, stands for discrete wavelet transform, and Respectively represent The low-frequency and high-frequency characteristics of the layer; S420: Perform a layer of residual causal convolution operation on the high frequency and low frequency of the last layer of discrete wavelet transform, respectively, as follows: ； in, represents the number of discrete wavelet layers, represents residual causal convolution, and They represent the high-frequency and low-frequency features after residual causal convolution respectively; S430: In order to remove the interference of some high-frequency features, except for retaining the high-frequency features of the Kth layer, the high-frequency features of the remaining layers are discarded, and the high-frequency features of the i-1 layer are replaced by the inverse discrete wavelet transform results of the i layer, as follows: ； in, Stands for Inverse Discrete Wavelet Transform.

5. The method according to claim 4, characterized in that The Fourier attention module includes the Fourier filter module and the attention module. The Fourier filter module includes the first normalization layer, the Fourier transform layer, the inverse Fourier transform layer, the second normalization layer and the fully connected layer. The attention module includes the first normalization layer, the multi-head attention mechanism, the second normalization layer and the multi-layer perceptron. S500 includes: S510: splicing the time series feature information and the frequency feature information in the image and then passing them through a fully connected layer to obtain fused features; S520: Input the fused features into the Fourier attention module. First, the fused features are normalized by the first normalization layer and then input into the Fourier transform layer. The Fourier transform is performed in combination with the learnable weights. The features after the Fourier transform are input into the inverse Fourier transform layer for inverse Fourier transform. The features after the inverse Fourier transform are input into the second normalization layer for normalization and then pass through the fully connected layer. The output of the fully connected layer is added with the fused features to obtain the output of the Fourier filter module. S530: The output of the Fourier filter module is input to the attention module. First, the output of the Fourier filter module is normalized by the first normalization layer, and then input into the multi-head attention mechanism for attention operation, and then input into the second normalization layer for normalization, and then feature mapping is performed through the multi-layer perceptron, and the input of the attention module is added to obtain the output result of the attention module as the output of the prediction stage.

6. The method according to claim 5, characterized in that S520 is specifically: ； ； ； ； S530 is specifically: ； in, represents the input of the Fourier module, represent Fourier transform and inverse Fourier transform respectively, represents normalization, are learnable weights, is a fully connected layer, is a multi-head attention module, is a multi-layer perceptron, , , , and They represent Fourier transform features, learnable weight features, inverse Fourier transform features, Fourier filter features, and attention features respectively.

7. The method according to claim 6, characterized in that The preset loss functions include frame loss function and smoothing loss function, specifically: ； ； ； in, represents the frame loss function, represents the smooth loss function, Represents the corresponding true value of class c in the tth frame; Represent the predicted values ​​of the prediction stage and the refinement stage respectively; C is the category of the surgical stage, Represents the number of frames.

8. The method according to claim 7, characterized in that S700 obtains real-time laparoscopic surgery video and inputs it into the trained time-frequency hybrid network to complete real-time surgery stage recognition, specifically: ； ； ； ； ； The refining stage is: ； ； ； ； ； in, Represent single-stage temporal convolution and wavelet temporal convolution, respectively. Represents splicing, represents the fully connected layer, represents the Fourier attention module; They represent the single-stage time series characteristics, wavelet time series characteristics, time series characteristics and output in the prediction stage respectively; They represent the single-stage time series characteristics, wavelet time series characteristics, time series characteristics and output of the refining stage respectively.

9. A surgical stage recognition system based on a time-frequency hybrid network, characterized in that: It includes a data set acquisition module, a time-frequency hybrid network construction module, a time series feature information acquisition module, a frequency feature information acquisition module, a prediction stage result prediction module, a refinement stage result prediction module and a surgery stage recognition module; The data set acquisition module is used to collect surgical laparoscopic surgery videos and cut them according to the preset frequency to obtain multiple continuous photos. The cut photos are input into the trained Resnet network to extract useful spatial features and obtain the data set. A time-frequency hybrid network building module is used to build a time-frequency hybrid network, including a prediction stage and a refinement stage. Both the prediction stage and the refinement stage include a single-stage time series convolution module, a wavelet time series convolution module, and a Fourier attention module. The modules in the prediction stage and the refinement stage have the same composition and structure, but do not share parameters. The temporal feature information acquisition module is used to input the data set into the single-stage temporal convolutional network in the prediction stage, and use multi-layer residual causal dilated convolution to expand the receptive field to obtain the temporal feature information of each layer in the image; The frequency feature information acquisition module is used to input the data set into the wavelet time series convolution module in the prediction stage, obtain multi-level frequency information through discrete wavelet transform, perform a layer of residual causal convolution on the frequency information of the last layer, and then perform discrete wavelet inverse transform, replace the high-frequency features of the previous layer with the discrete wavelet inverse transform results of the next layer, discard the high-frequency features of other layers, and obtain the frequency feature information with some high-frequency features filtered out; The prediction stage result prediction module is used to fuse the temporal feature information and frequency feature information in the image and input them into the Fourier attention module of the prediction stage for Fourier filtering and attention operation to obtain the output of the prediction stage; The result prediction module of the refining stage is used to use the output of the prediction stage as the input of the refining stage, repeatedly obtain the time series feature information and the frequency feature information, fuse the time series feature information and the frequency feature information in the image, and input them into the Fourier attention module of the refining stage for Fourier filtering and attention operation. The obtained features are mapped to the dimension of the category through the fully connected layer, and the recognition prediction result of the surgical stage is output; The surgical stage recognition module is used to train the time-frequency hybrid network based on a preset loss function. When the preset training termination is reached, the trained time-frequency hybrid network is obtained, and the real-time surgical laparoscopic surgery video is obtained. After the features are extracted by the trained Resnet network, they are input into the trained time-frequency hybrid network to complete the real-time surgical stage recognition.

10. A device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images