Endoscopic surgery action recognition method and system and storage medium

Abstract

The invention discloses an endoscopic surgery action recognition method and system and a storage medium, and is applied to the field of medical image recognition. Extracting feature maps of different levels of the endoscopic surgery video image, fusing low-level visual features, high-level semantic features and surgery action component features, and adopting mutual information distillation based on component feature space to reserve different component features and remove redundant information; non-target information distillation is carried out to eliminate head class consistency deviation, and head-deviation-free features are obtained; time dimension rich features are utilized, the influence of similar frames on the current frame is reduced, effective time features aligned with the surgical action component features are obtained, and time-space fusion features are obtained; positive and negative samples are constructed by adopting a comparative learning method, and tail features are enhanced by adopting a momentum-based sample updating strategy; and combining the head deviation-free feature, the space-time fusion feature and the tail enhancement feature to obtain a surgical action recognition result. According to the invention, the accuracy of surgical action recognition is ensured.

Description

Technical Field

[0001] The present invention relates to the field of medical image recognition, and in particular to a method, system and storage medium for recognizing endoscopic surgical actions. Background Art

[0002] Since the success of the first laparoscopic cholecystectomy, minimally invasive surgery has become a common surgical option in the field of surgery due to its advantages such as small surgical incision, less intraoperative bleeding, fast postoperative recovery and less pain for patients. With the continuous development of robotic technology, surgical equipment represented by the da Vinci surgical robot has inherited the advantages of minimally invasive surgery and even surpassed the former. The surgical robot is integrated with an ultra-high-definition 3D endoscope, which greatly facilitates the doctor's clear positioning of tissues and accurate grasp of the surgical process during surgery. The tip of the robot's arm has a higher degree of freedom than human hands, which greatly increases the flexibility of operation and improves the accuracy of surgical movements. In order to handle and respond to various situations during surgery in a timely manner, the surgical robot needs to accurately identify various key elements in the operation, such as surgical instruments, surgical movements, surgical targets, etc. Surgical instruments are the medium for doctors and surgical robots to interact with patients. Common surgical instruments include grasping forceps, scissors, hooks, bipolar forceps, etc.; surgical actions are the basic units of the entire surgical process, generally expressed by verbs, such as pulling, grasping, and dissecting; the surgical goal depends on the type of surgery performed. Taking laparoscopic cholecystectomy as an example, common surgical goals include gallbladder, bile duct, bile artery, etc. The surgical instruments, surgical actions, and surgical goals constitute the surgical action triple. The surgical action triple is the basic unit of the endoscopic surgical process, which can accurately reflect the execution of the surgery, reduce errors during the surgery, and generate detailed annotated surgical videos after the surgery to help new doctors learn the surgical process faster. Provide rich clinical information for surgeons and surgical robots. Therefore, accurate recognition of surgical actions is very important for the development of surgical robots and the construction of modern operating rooms.

[0003] With the continuous development of deep learning, a large number of works have adopted deep learning methods to identify surgical actions. Many works have designed various models from the perspectives of image feature recognition, component multi-task learning, component multi-label relationship, spatiotemporal feature fusion, etc. Using these models to perform surgical action recognition tasks can effectively help doctors and surgical robots perceive surgical elements such as instruments, tissues, and actions. However, due to the limitations of the endoscope's shooting angle and the abdominal environment, endoscopic surgery movements are very delicate, and different surgical instruments and surgical actions are very similar, making recognition very difficult. Previous methods can effectively identify those surgical action triplets that appear in large numbers, but it is often difficult to achieve good recognition results for those actions that appear less frequently. Most methods transfer basic visual task models applicable to natural images. These models have low computational complexity and fast inference speed, but they ignore the complex situations under the endoscope's perspective, and do not fully utilize the features in the time dimension to improve recognition accuracy. Methods based on time series models such as recurrent convolutional networks can be used to process time series tasks and use time features, but the long-term dependence and short-term dependence accuracy of time features cannot be accurately controlled, and adjacent frame actions often interfere with each other. In addition, the training time is long and the deployment cost is high, which is not conducive to clinical practice and promotion. In the context of robotic surgery or robot-assisted surgery, how to accurately identify rarely occurring actions and ensure the recognition accuracy and speed of surgical actions is very important, and this is a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0004] In view of this, the present invention provides a method, system and storage medium for identifying endoscopic surgical actions to solve the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] An action recognition method for endoscopic surgery comprises the following steps:

[0007] Step 1: Extract feature maps of different levels of endoscopic surgery video images and divide the feature maps into low-level visual features and high-level semantic features;

[0008] Step 2: Obtain surgical action component features, fuse low-level visual features, high-level semantic features with component features, and use mutual information distillation based on component feature space to retain different component features and remove redundant information;

[0009] Step 3: Use the non-target outputs with similar semantic information to distill the features of different components to eliminate the consistency bias of the head class and obtain features without head bias;

[0010] Step 4: Use the time dimension to enrich features, evaluate the significance scores of feature maps of different time frames, and sample time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, obtain effective time features aligned with surgical action component features, and obtain spatiotemporal fusion features;

[0011] Step 5: Use contrastive learning to construct positive and negative samples and momentum-based sample update strategy to enhance the tail features and obtain tail enhancement features;

[0012] Step 6: Based on the weight selection, the head-free bias feature, spatiotemporal fusion feature and tail enhancement feature are combined to obtain the final surgical action recognition result.

[0013] Optionally, extract feature maps of different levels of the endoscopic surgery video image, and for each level of features:

[0014]

[0015] in, The number of feature layers is S and the feature space is θ.

[0016] Optionally, the surgical action component features include instrument features F I 、Organizational characteristics F T and action feature F V , the low-level visual feature is recorded as F 1 , F 2 , the high-level semantic feature is recorded as F 3 , F s , the low-level visual features, high-level semantic features and component features are integrated, specifically: the instrument feature F I 、Organizational characteristics F T With the lower level features F 1 , F 2 Feature fusion, action feature F V With F 3 、F s Feature fusion.

[0017] Optionally, in step 4, based on the triple feature F IVT , obtain the query vector embedding Q about the time feature, the key value vector embedding K, the value vector embedding V, Q = F IVT U q ,K=F IVT U k ,V=F IVT U v , Q, K, V obtain new embedding representation through the standard self-attention layer Significance score S = [s 1 ,s 2 ,…sN ] T =tanh (KU s ), where U q ,U k ,U v ,U s They are all learnable matrices, and the softmax layer is used to calculate the sampling probability Match key-value embeddings based on cosine similarity Merge F through an average pooling layer IVT , The obtained time feature F IVT,t With F IVT Perform residual connection to obtain spatiotemporal fusion feature F IVT,st .

[0018] Optionally, in step 5, obtain the triple feature F IVT , for the tail k, extract F IVT Tail C K channels, and F S Concatenate in dimension and obtain tail features through 1×1 convolution:

[0019]

[0020] Optionally, in step six, the head-free bias feature is passed through an adaptive weight selection module to obtain a weighted meta-feature, the weighted meta-feature is element-wise added to the spatiotemporal fusion feature, and multiplied with the tuple tail enhancement feature, and finally output through a fully connected layer to obtain the final surgical action recognition result.

[0021] An action recognition system for endoscopic surgery, comprising:

[0022] ResNet50 backbone network: used to extract feature maps at different levels of endoscopic surgery video images and divide the feature maps into low-level visual features and high-level semantic features;

[0023] Deep separation feature fusion module: used to obtain surgical action component features, fuse low-level visual features, high-level semantic features with component features, and use mutual information distillation based on component feature space to retain different component features and remove redundant information;

[0024] Dynamic feature migration module: used to distill the features of different components using non-target outputs with similar semantic information to eliminate the head class consistency deviation and obtain features without head deviation;

[0025] Adaptive time feature merging module: used to enrich features using the time dimension, evaluate the significance scores of feature maps of different time frames, and sample time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, obtain effective time features aligned with surgical action component features, and obtain spatiotemporal fusion features;

[0026] Triplet tail contrastive learning module: used to promote mutual learning between tail classes, using contrastive learning to construct positive and negative samples and momentum-based strategy updates to enhance tail features and obtain tail enhancement features;

[0027] Adaptive weight classification module: used to combine the elements of head-free bias features, spatiotemporal fusion features and tail enhancement features based on weight selection to obtain the final surgical action recognition results.

[0028] Optionally, the downsampling module is also included, and the surgical action component features include the instrument features F I 、Organizational characteristics F T and action feature F V , the low-level visual feature is recorded as F 1 , F 2 , the high-level semantic feature is recorded as F 3 , F s The downsampling module fuses low-level visual features, high-level semantic features and component features, specifically: instrument feature F I 、Organizational characteristics F T With the lower level features F 1 , F 2 Feature fusion, action feature F V With F 3 、F s Feature fusion.

[0029] A computer storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for motion recognition of endoscopic surgery.

[0030] Through the above technical solutions, it can be seen that compared with the prior art, the present invention provides a method, system and storage medium for recognizing endoscopic surgical actions. Low-level visual features and high-level semantic features are extracted through the ResNet50 backbone network, and the triple component features are fused through the deep separation feature fusion network to obtain a richer feature expression, and the fused features are subjected to mutual information distillation to retain effective related features and remove redundant features. The dynamic feature migration module uses non-target outputs with similar semantic information to alleviate the head class consistency tendency of the features and enhance the accuracy of the tail class feature expression. The adaptive time feature merging module makes full use of the time dimension to enrich the features, evaluates the significance scores of the features of different time frames, and samples the time features according to the scores to maintain time consistency, reduce the influence of the close frames on the current frame, and obtain effective features aligned with the surgical action triplet. The tail class comparison learning module enables the tail classes to learn from each other, obtain enhanced tail class features, and finally obtain the final surgical action recognition result through the classifier. In summary, the present invention can effectively utilize spatiotemporal features, eliminate the mutual influence of adjacent frame actions, obtain accurate action features, and at the same time enhance the tail class recognition effect, thereby ensuring the accuracy of surgical action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0032] Figure 1 A schematic diagram of the network structure of the surgical action recognition method of the present invention;

[0033] Figure 2 It is the deep separation feature fusion module of the present invention;

[0034] Figure 3 The dynamic feature migration module of the present invention;

[0035] Figure 4 The adaptive time feature merging module of the present invention;

[0036] Figure 5 It is the tail class comparison learning module of the present invention;

[0037] Figure 6 It is the adaptive weight classifier of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1

[0040] This embodiment discloses a method for motion recognition of endoscopic surgery, comprising the following steps:

[0041] Step 1: Extract feature maps of different levels of endoscopic surgery video images and divide the feature maps into low-level visual features and high-level semantic features;

[0042] Step 2: Obtain surgical action component features, fuse low-level visual features, high-level semantic features with component features, and use mutual information distillation based on component feature space to retain different component features and remove redundant information;

[0043] Step 3: Use the non-target outputs with similar semantic information to distill the features of different components to eliminate the consistency bias of the head class and obtain features without head bias;

[0044] Step 4: Use the time dimension to enrich features, evaluate the significance scores of feature maps of different time frames, and sample time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, obtain effective time features aligned with surgical action component features, and obtain spatiotemporal fusion features;

[0045] Step 5: Use the positive and negative sample construction method of contrastive learning and the sample update strategy based on momentum to enhance the tail feature and obtain the tail enhanced feature;

[0046] Step 6: Based on the weight selection, the head-free bias feature, spatiotemporal fusion feature and tail enhancement feature are combined to obtain the final surgical action recognition result.

[0047] In step 1, the ResNet50 network consists of multiple residual linked feature blocks, which can extract image features at different levels. The surgical video input is x∈R N×C×H×W , the number of feature layers of ResNet50 is S, the feature space is denoted as θ, and for each layer of features

[0048]

[0049] in,

[0050] Optionally, in step 2, the depth separation feature fusion module separates the feature maps by depth (1 <s1 <s 2 <s 3 <S) is divided into four parts, and the extracted feature scores are denoted as F 1 , F 2 , F 3 , F S , where F 1 , F 2 contains rich low-level features such as texture, boundary, color, etc., and F 3 , F 4 has rich high-level semantic features.

[0051] F s The feature obtains the component features of the surgical action triple through a 1×1 convolutional network. In the triple <instrument, action, tissue>, the instrument feature F I , the tissue feature F T and the lower-level feature F 1 , F 2 are feature-fused, and the action feature F V and F 3 , F s are feature-fused.

[0052] F j = Conv 1×1,j (F S ), j = I, V, T, IVT.

[0053] F 1 , F 2 , F s The feature is aligned with the triple component feature through the downsampling module, and the intermediate layer feature of the visual backbone is fully utilized through the element-wise multiplication (Hadamard product) to enhance the expression of the triple component feature, and the mutual information distillation based on the feature space is introduced to retain different component features of and remove redundant information.

[0054] Optionally, in step three, the dynamic feature transfer module uses those non-target outputs with similar semantic information. The dynamic feature transfer module decouples the logical output into the target logical output z I,y , z V,y , z T,y and the non-target logical output Z I = [z I,1 , z I,2 , …, z I,c-1 , Z V = [z V,1 , z V,2 , …, z V,c-1 , Z T = [z T,1,z T,2 ,…,z T,c-1 ], c≠y, non-target logic output average Non-target logic output maximum value Due to the feature-based mutual distillation learning mechanism action organize It is not inclined to recognize the same non-target surgical action triplet at the same time, so the non-target maximum value can be used to alleviate the consistency bias of the head class.

[0055] The probability of a component becoming a teacher model is The probability of a component becoming a student model Constructing non-target losses based on probability Optimized components Feature learning,enhances the accurate expression of tail class features.

[0056] Optionally, in step 4, the adaptive time feature merging module makes full use of the time dimension to enrich the features, evaluates the significance scores of features in different time frames, and samples the time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, and obtain effective features aligned with the surgical action triplet, F IVT Input the adaptive time feature merging module, first obtain the query vector about the time feature and embed it into Q, the key value vector and embed it into K, and the value vector and embed it into V, Q = F IVT U q ,K=F IVT U k ,V=F IVT U v , Q, K, V obtain new embedding representation through the standard self-attention layer Significance score S = [s 1 ,s 2 ,…s N ] T =tanh (KU s ), where U q ,U k ,U v ,U s They are all learnable matrices, and the softmax layer is used to calculate the sampling probability Match key-value embeddings based on cosine similarity Merge F through an average pooling layer IVT , The obtained time feature F IVT,t With F IVT Perform residual connection to obtain spatiotemporal fusion feature F IVT,st

[0057] Optionally, in step 5, the triple tail class comparison learning module converts the backbone feature F S After 1×1 convolution, the triple feature F is obtained IVT , for the tail class k, extract F IVT Tail C K channels, and F S Concatenate in dimension and obtain the tail class features through 1×1 convolution, namely:

[0058]

[0059] The tail class contrast learning module enables the tail classes to learn from each other and adopts a positive and negative sample construction method based on a memory bank, where F IVT,st Embed e as a query vector, As the key vector embedding e', the key vector embedding e' and the historical key vector embedding are placed in the storage block, Bank = {e 1 ,e 2 ,…,e t}, in the storage block, the current tail class is used as a positive sample, and the historical tail class is used as a negative sample, E pos ={e i ∈Bank,y i =y label The update of negative samples adopts a momentum-based strategy, and the training loss is The new tail class features obtained Used to enhance the final classification.

[0060] Optionally, in step 6, the adaptive weight classification module inputs the spatiotemporal fusion feature F of the surgical action triplet IVT,st , triple tail enhancement feature Headless bias features of triple components after mutual knowledge distillation The final recognition results are as follows:

[0061] After the adaptive weight selection module, the weighted meta-features are obtained and spatiotemporal fusion feature F IVT,st Element-wise addition and tuple tail enhancement Multiply them together and finally output them through the fully connected layer to obtain the final surgical action recognition result.

[0062] This embodiment also discloses an endoscopic surgery action recognition system, including:

[0063] ResNet50 backbone network: used to extract feature maps at different levels of endoscopic surgery video images and divide the feature maps into low-level visual features and high-level semantic features;

[0064] Deep separation feature fusion module: used to obtain surgical action component features, fuse low-level visual features, high-level semantic features with component features, and use mutual information distillation based on component feature space to retain different component features and remove redundant information;

[0065] Dynamic feature migration module: used to distill the features of different components using non-target outputs with similar semantic information to eliminate the head class consistency deviation and obtain features without head deviation;

[0066] Adaptive time feature merging module: used to enrich features using the time dimension, evaluate the significance scores of feature maps of different time frames, and sample time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, obtain effective time features aligned with surgical action component features, and obtain spatiotemporal fusion features;

[0067] Triplet tail contrastive learning module: used to promote mutual learning between tail classes, using a positive and negative sample construction method based on contrastive learning and a momentum-based sample update strategy to enhance tail features and obtain tail enhanced features;

[0068] Adaptive weight classification module: used to combine the elements of head-free bias features, spatiotemporal fusion features and tail enhancement features based on weight selection to obtain the final surgical action recognition results.

[0069] Finally, this embodiment further discloses a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for motion recognition of endoscopic surgery are implemented.

[0070] Example 2

[0071] Embodiment 2 of the present invention discloses a method for motion recognition of endoscopic surgery, such as Figure 1 As shown, including:

[0072] Step (1): Use the ResNet50 backbone network to extract the visual features of the surgical video image. The ResNet50 network consists of multiple residual linked feature blocks and can extract image features at different levels. The surgical video input is x∈R N×C×H×W , the number of feature layers of ResNet50 is S, the feature space is denoted as θ, and for each layer of features

[0073] F i =θ i °θ i-1 °…θ1 (x), 1 < i < S

[0074] where θ i °θ i-1 (x) = θ i (θ i-1 (x)),

[0075] Step (2): Use the depth - wise feature fusion module (Depth - wise Feature Fusion Module, DFF) as shown in Figure 2 to fuse the triple component features and backbone features, and introduce mutual information distillation based on the feature space, specifically:

[0076] Divide the feature map into four parts according to the depth (1 < s 1 < s 2 < s 3 < S), and the extracted features are denoted as F 1 , F 2 , F 3 , F S , where F 1 , F 2 contain rich low - level features such as texture, boundary, color, etc., and F 3 , F 4 have rich high - level semantic features.

[0077] F s features obtain surgical action triple component features through a 1×1 convolutional network. In the triple <instrument, action, tissue>, the instrument feature F I , the tissue feature F T are fused with the lower - level features F 1 , F 2 features, and the action feature F V is fused with F 3 , F s features,

[0078] F j = Conv 1×1,j (F S ), j = I, V, T, IVT,

[0079] F 1 , F 2 , F sThe features are aligned with the triplet component features through the downsampling module, and the features of the middle layer of the visual backbone are fully utilized by element-wise multiplication (Hadamard product) to enhance the expression of triplet component features. The downsampling module is composed of depth-wise convolution and pointwise convolution.

[0080] DownSampling(X)=Conv p (Conv d (X)

[0081]

[0082]

[0083]

[0084] Introduce mutual information distillation based on feature space to retain Different component features and remove redundant information, mutual information distillation is based on mutual information loss Specifically

[0085]

[0086] p j ,p k is the normalized probability. according to Optimize the target surgical action triplet Y IVT Label recognition effect.

[0087] Step (3): Introduce Figure 3 The dynamic feature transfer module (Dynamic Feature Transfer, DFT) shown in the figure, in the tail class recognition, the effect of information distillation depends largely on the non-target outputs with similar semantic information. The deep separation feature fusion module will make the model feature This will cause a head class consistency deviation, and the dynamic feature migration module will The logic output is decoupled to the target logic output z I,y ,z V,y ,z T,y and non-target logic output Z I =[z I,1 ,z I,2 ,…,z I,c-1 ],Z V =

[0088] [z V,1 ,z V,2 ,…,zV,c-1 ],Z T =[z T,1 ,z T,2 ,…,z T,c-1 ], c≠y, non-target logic output average Non-target logic output maximum value Due to the feature-based mutual distillation learning mechanism action organize It is not inclined to recognize the same non-target surgical action triplet at the same time, so the non-target maximum value can be used to alleviate the head consistency bias. The probability of a component becoming a teacher model is The probability of a component becoming a student model Constructing non-target losses based on probability Optimized components Feature learning.

[0089] Step (4): Use Figure 4 The adaptive temporal feature merging module shown in the figure makes full use of the temporal dimension to enrich features, evaluates the significance scores of features in different time frames, and samples temporal features according to the scores to maintain temporal consistency, reduce the impact of similar frames on the current frame, and obtain effective features aligned with the surgical action triplet. IVT Input the adaptive time feature merging module, first obtain the query vector embedding Q about the time feature, the key value vector embedding K, and the value vector embedding Q = F IVT U q ,K=F IVT U k ,V=F IVT U v , Q, K, V obtain new embedding representation through the standard self-attention layer Significance score S = [s 1 ,s 2 ,…s N ] T =tanh (KU s ), where U q ,U k ,U v ,U s They are all learnable matrices, and the softmax layer is used to calculate the sampling probability Match key-value embeddings based on cosine similarity Merge F through an average pooling layer IVT , The obtained time feature F IVT,t With F IVTPerform residual connection to obtain spatiotemporal fusion feature F IVT,st

[0090]

[0091] S = tanh(KU s

[0092]

[0093] Step (5): Use Figure 5 The triple tail class contrast learning module shown in the figure converts the backbone feature F S After 1×1 convolution, the triple feature F is obtained IVT , for the tail class k, extract F IVT Tail C K channels, and F S Concatenate in dimension and obtain the tail class features through 1×1 convolution, namely:

[0094]

[0095] The tail class contrast learning module enables the tail classes to learn from each other and adopts a positive and negative sample construction method based on a memory bank, where F IVT,st Embed e as a query vector, As the key vector embedding e', the key vector embedding e' and the historical key vector embedding are placed in the storage block, Bank = {e 1 ,e 2 ,…,e t}, in the storage block, the current tail class is used as a positive sample, and the historical tail class is used as a negative sample, E pos ={e i ∈Bank,y i =y label The update of negative samples adopts a momentum-based strategy, and the training loss is The new tail class features obtained Used to enhance the final classification.

[0096] Step (6): Use Figure 6 The adaptive weight classification module shown in the figure takes as input the spatiotemporal fusion feature F of the surgical action triplet IVT,st , triple tail enhancement feature Headless bias features of triple components after mutual knowledge distillation The final recognition results are as follows:

[0097] After the adaptive weight selection module, the weighted meta-features are obtained and spatiotemporal fusion feature F IVT,st Element-wise addition and tuple tail enhancement Multiply them together and finally output them through the fully connected layer to obtain the final surgical action recognition result.

[0098]

[0099]

[0100] Y IVT =sigmoid(fc(F o ))

[0101] In order to evaluate the recognition ability of a surgical action recognition method proposed in the present invention, experiments were conducted on the Cholect45 dataset as follows:

[0102] The CholecT45 dataset consists of 45 laparoscopic surgery videos, and images are extracted from the videos at one frame per second. There are a total of 100.9k frames of images and 161k surgical action triplet labels. There are 100 types of surgical action triples, of which the types of components are: 6 surgical instruments, 10 surgical actions, and 15 target tissues. In order to reasonably evaluate the recognition ability of the surgical action recognition method proposed in this invention, a 5-fold mutual validation strategy is used to verify the effectiveness of the model. The dataset is divided into 31 videos for training, 5 videos for validation, and 9 videos for testing. For fairness in comparison, we have adopted the mAP, a commonly used average precision indicator for surgical action recognition. The mAP indicator includes AP I ,AP V ,AP T ,AP W ,AP IT ,AP IVT

[0103] Table 1 Comparison of experimental results of different methods on the CholecT45 laparoscopic surgery dataset

[0104]

[0105] The results of the evaluation using the evaluation index are shown in Table 1. It can be seen that the surgical action recognition method proposed in the present invention outperforms other algorithms in the triple recognition and triple component recognition on the Cholect45 dataset. The accuracy results of each sub-mAP are improved by 1%-2%. This proves the effectiveness of the method in the action recognition task of endoscopic surgery.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0107] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for motion recognition in endoscopic surgery, characterized in that: The following steps are involved: Step 1: Extract feature maps of different levels of endoscopic surgery video images and divide the feature maps into low-level visual features and high-level semantic features; Step 2: Obtain surgical action component features, fuse low-level visual features, high-level semantic features with surgical action component features, and use mutual information distillation based on component feature space to retain different component features and remove redundant information; Step 3: Use the non-target outputs with similar semantic information to distill the features of different components to eliminate the consistency bias of the head class and obtain features without head bias; Step 4: Use the time dimension to enrich features, evaluate the significance scores of feature maps of different time frames, and sample time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, obtain effective time features aligned with surgical action component features, and obtain spatiotemporal fusion features; Step 5: Use contrastive learning to construct positive and negative samples and momentum-based sample update strategy to enhance the tail features and obtain tail enhancement features; Step 6: Combine the head-free deviation features, spatiotemporal fusion features, and tail enhancement features based on weight selection to obtain the final surgical action recognition result.

2. The method for motion recognition of endoscopic surgery according to claim 1, characterized in that: Extract feature maps of different levels of endoscopic surgery video images, and for each layer of features: F i =θ i °θ i-1 °…θ1(x),1 <i<S; Among them, θ i °θ i-1 (x) = θ i (θ i-1 (x)), the number of feature layers is S, and the feature space is θ.

3. The method for motion recognition of endoscopic surgery according to claim 1, characterized in that: The surgical action component features include instrument features F I 、Organizational characteristics F T and action feature F V , low-level visual features are denoted as F1, F2, and high-level semantic features are denoted as F3, F s , the low-level visual features, high-level semantic features and component features are integrated, specifically: the device feature F I 、Organizational characteristics F T Fusion with lower-level features F1, F2, action feature F V With F3, F s Feature fusion.

4. The method for motion recognition of endoscopic surgery according to claim 1, characterized in that: In step 4, based on the triple feature F IVT , obtain the query vector embedding Q about the time feature, the key value vector embedding K, the value vector embedding V, Q = F IVT U q ,K=F IVT U k ,V=F IVT U v , Q, K, V obtain new embedding representation through the standard self-attention layer Significance score S = [s1, s2, ...s N ] T =tanh (KU s ), where U q ,U k ,U v ,U s All are learned matrices, and the softmax layer is used to calculate the sampling probability Match key-value embeddings based on cosine similarity Merge F through an average pooling layer IVT , The obtained time feature F IVT,t With F IVT Perform residual connection to obtain spatiotemporal fusion feature F IVT,st .

5. The method for motion recognition of endoscopic surgery according to claim 1, characterized in that: In step 5, the triple feature F is obtained IVT , for the tail k, extract F IVT Tail C K channels, and F S Concatenate in dimension and obtain tail features through 1×1 convolution:

6. The method for motion recognition of endoscopic surgery according to claim 1, characterized in that: In step six, the head-free bias feature is passed through the adaptive weight selection module to obtain the weighted meta-feature. The weighted meta-feature is added element-wise with the spatiotemporal fusion feature and multiplied with the tuple tail enhancement feature. Finally, it is output through the fully connected layer to obtain the final surgical action recognition result.

7. An action recognition system for endoscopic surgery, characterized in that: include: ResNet50 backbone network: used to extract feature maps at different levels of endoscopic surgery video images and divide the feature maps into low-level visual features and high-level semantic features; Deep separation feature fusion module: used to obtain surgical action component features, fuse low-level visual features, high-level semantic features with component features, and use mutual information distillation based on component feature space to retain different component features and remove redundant information; Dynamic feature migration module: used to distill the features of different components using non-target outputs with similar semantic information to eliminate the head class consistency deviation and obtain features without head deviation; Adaptive time feature merging module: used to enrich features using the time dimension, evaluate the significance scores of feature maps of different time frames, and sample time features according to the scores to maintain time consistency, reduce the impact of similar frames on the current frame, obtain effective time features aligned with surgical action component features, and obtain spatiotemporal fusion features; Triplet tail contrastive learning module: used to promote mutual learning between tail classes, using contrastive learning to construct positive and negative samples and momentum-based strategy updates to enhance tail features and obtain tail enhancement features; Adaptive weight classification module: used to combine elements based on weight-selected head-free bias features, spatiotemporal fusion features, and tail enhancement features to obtain the final surgical action recognition results.

8. The motion recognition system for endoscopic surgery according to claim 7, characterized in that: The downsampling module is also included, and the surgical action component features include the instrument features F I 、Organizational characteristics F T and action feature F V , low-level visual features are denoted as F1, F2, and high-level semantic features are denoted as F3, F s The downsampling module fuses low-level visual features, high-level semantic features and component features, specifically: instrument feature F I 、Organizational characteristics F T Fusion with lower-level features F1, F2, action feature F V With F3, F s Feature fusion.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the motion recognition method for endoscopic surgery as described in any one of claims 1 to 6 are implemented.

Images