Text semantic enhancement-based abdominal surgery video triple recognition method

Abstract

The invention relates to an abdominal surgery video triple recognition method based on text semantic enhancement. The method comprises the steps of designing a text prompt and extracting features of the text prompt to obtain text feature representation; an image encoder is adopted to extract image coding features of the abdominal cavity operation video frame image, and triple features are output from the image coding features through a bottleneck layer; based on a designed ternary decoupling attention module, image coding features are decoupled into instrument features, action features and target tissue features; fusing the triple features, the instrument features, the action features and the target tissue features to obtain image feature representation; fusing the text feature representation and the image feature representation to obtain an image feature after text enhancement; training a time convolutional network based on the text-enhanced image features to obtain a trained time convolutional network; obtaining a second image feature representation of the to-be-identified abdominal cavity operation video frame image; and inputting the second image feature representation into a trained time convolutional network to obtain a triple classification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of surgical video recognition, in particular to a laparoscopic surgery video triplets recognition method based on text semantic enhancement. BACKGROUND

[0002] The laparoscopic surgery video triplets contain surgery instruments, surgery actions and surgery target tissue information, which are of great significance for intraoperative guidance and postoperative analysis. At present, it is difficult to eliminate the inherent ambiguity in vision based on image information alone, and it is also difficult to model the semantic association between image frames and labels, thus resulting in low triplets recognition accuracy.

[0003] Using text features to strengthen image features can effectively reduce the ambiguity of image information, but there are still two difficulties in specific implementation: 1. How to design text prompts to fully exploit the semantic information of multi-modal models, because different text templates will have a great impact on model results; 2. How to design an effective multi-modal feature extraction and fusion alignment method to fully integrate dual modal information to help improve video triplet recognition accuracy. SUMMARY

[0004] Therefore, it is necessary to provide a laparoscopic surgery video triplets recognition method based on text semantic enhancement, comprising: S1: design a text prompt, the text prompt includes a standardized template prompt and a learnable prompt, and use a text encoder to extract a standard prompt feature and a learnable prompt feature, and add the standard prompt feature and the learnable prompt feature to obtain a text feature representation; S2: obtain a laparoscopic surgery video frame image, use an image encoder to extract an image encoding feature of the laparoscopic surgery video frame image, and output a triplet feature through a bottleneck layer; decouple the image encoding feature into an instrument feature, an action feature and a target tissue feature based on a designed triple decoupling attention module; and fuse the triplet feature, the instrument feature, the action feature and the target tissue feature using a multi-head hybrid attention mechanism to obtain an image feature representation; S3: fuse the text feature representation and the image feature representation using a multi-head cross-attention mechanism to obtain a text-enhanced image feature; and train a temporal convolution network based on the text-enhanced image feature to obtain a trained temporal convolution network; S4: obtain a corresponding second image feature representation and a second instrument feature by performing step S2 on the laparoscopic surgery video frame image to be recognized; input the second image feature representation into the trained temporal convolution network to obtain a triplet classification result; and perform foreground extraction on the second instrument feature to obtain a surgery instrument positioning result, and visualize the surgery instrument positioning result to provide information guidance for surgery and verify the triplet classification result.

[0005] Preferably, the standardized template prompt is designed as "a {LABEL} laparoscopic image", and the triple tag of each laparoscopic surgery video frame image in the laparoscopic surgery video frame image data set is filled in {LABEL}; all standardized templates filled with triple tags are respectively input into a text encoder to obtain a plurality of standardized prompt features; The learnable prompt is a learnable parameter tensor containing 16 randomly initialized variables. The learnable prompt is input into a text encoder to obtain a learnable prompt feature. The text feature representation is obtained by adding all the standardized prompt features and the learnable prompt feature.

[0006] Preferably, the text encoder is a text encoder in a pre-trained CLIP model, and the image encoder is an image encoder in the pre-trained CLIP model.

[0007] Preferably, the bottleneck layer includes a first convolutional layer including 256 channels and a convolutional kernel size of 3x3, a second convolutional layer including triple channels and a convolutional kernel size of 1x1; the image encoding feature is sequentially input into the first convolutional layer and the second convolutional layer to output a triple feature.

[0008] Preferably, the triple decoupling attention module includes a third convolutional layer including 64 channels and a convolutional kernel size of 3x3, a fourth convolutional layer including instrument feature channels and a convolutional kernel size of 1x1, a fifth convolutional layer including action feature channels and a convolutional kernel size of 1x1, and a sixth convolutional layer including target tissue feature channels and a convolutional kernel size of 1x1; the third convolutional layer is connected to the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer, respectively. After the image encoding feature passes through the third convolutional layer, it is input into the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer, respectively, to decouple the instrument feature, the action feature, and the target tissue feature.

[0009] Preferably, the process of obtaining the image feature representation includes: In the partial attention head, the triple feature is mapped into a query vector, a key vector, and a value vector, respectively, and self-attention fusion is performed based on the query vector, the key vector, and the value vector to obtain a self-attention feature; In the remaining partial attention head, the instrument feature, the action feature, and the target tissue feature are combined into a second triple feature, and the second triple feature is mapped into a second key vector and a second value vector, respectively; cross-attention fusion is performed based on the query vector, the second key vector, and the second value vector to obtain a cross-attention feature; The self-attention features and the cross-attention features in all the attention heads are spliced to obtain a spliced feature; the spliced feature is multiplied by a first linear projection parameter to obtain an image feature representation.

[0010] Preferably, the process of obtaining the text-enhanced image feature comprises: mapping the image feature representation into a second query vector, mapping the text feature representation into a third key vector and a third value vector; obtaining the text-enhanced image feature based on the second query vector, the third key vector and the third value vector, and through a multi-head cross attention mechanism.

[0011] Preferably, the training process of the time convolution network comprises: dividing the laparoscopic surgery video frame image dataset into a training set and a test set; subjecting all laparoscopic surgery video frame images in the training set to steps S2-S3 to obtain corresponding text-enhanced image features, respectively; inputting the text-enhanced image features into the time convolution network to output corresponding predicted labels; calculating a cross-entropy loss based on the predicted labels and the triple labels of the laparoscopic surgery video frame images; setting parameters of an SGD optimizer, setting an initial first learning rate to 0.001, using linear decay, setting the first learning rate to 0.0005 after the 200th epoch, and setting a maximum training period to 1000 epochs; performing gradient descent optimization on the time convolution network based on the cross-entropy loss and the SGD optimizer with the set parameters, and updating network parameters of the time convolution network; in the training process, when the cross-entropy loss corresponding to all laparoscopic surgery video frame images in the validation set does not decrease for 10 consecutive epochs, or when the training reaches the maximum number of iterations, stopping the training to obtain the trained time convolution network.

[0012] Preferably, the training process of the CLIP model comprises: freezing parameters of the text encoder and setting parameters of the image encoder to be trainable; performing data enhancement operations on the laparoscopic surgery video frame images in the laparoscopic surgery video frame image dataset to obtain an enhanced dataset, the text feature representation and the triple label corresponding to each laparoscopic surgery video frame image in the enhanced dataset being unchanged; subjecting each laparoscopic surgery video frame image in the enhanced dataset to step S2 to obtain a corresponding second image feature representation; calculating a contrast loss based on each second image feature representation and the corresponding text feature representation, and the formula for calculating the contrast loss being: ; ; ; wherein, represents the contrast loss. Represents expectations on the augmented dataset; represents KL divergence; represents cosine similarity; represents a similarity score between the second image feature representation and the text feature representation; represents a similarity score between the text feature representation and the second image feature representation; Represents the distribution of triplet labels from the second image feature representation to the text feature representation; Represents the distribution of triplet labels from text feature representation to the second image feature representation; represents a second image feature representation; Indicates the first A second image feature representation corresponding to a frame image of a laparoscopic surgery video; Indicates the first A second image feature representation corresponding to a frame image of a laparoscopic surgery video; Represents text feature representation; Indicates the first Text feature representation corresponding to the abdominal surgery video frame image; Indicates the first Text feature representation corresponding to the abdominal surgery video frame image; represents the temperature parameter; D represents the enhanced dataset; N represents the number of abdominal surgery video frame images in the enhanced dataset; Set the parameters of the Adam optimizer. The initial second learning rate is set to 1e-5, and the cosine decay strategy is used. During training, the second learning rate decays to 2e-4, the minimum learning rate is set to 1e-6, and the maximum training cycle is set to 200 epochs. Based on the contrastive loss and the Adam optimizer with set parameters, the image encoder is optimized by gradient descent to update the parameters of the image encoder. When the training reaches the maximum number of iterations, the training is stopped to obtain the pre-trained CLIP model.

[0013] Preferably, obtaining the surgical instrument positioning result includes: performing foreground extraction on the second instrument feature using a threshold of 0.5, and using the obtained minimum bounding rectangle as the surgical instrument positioning result.

[0014] Beneficial effects: the method obtains text feature representation by designing text prompts and extracting features thereof; adopts an image encoder to extract image coding features of the laparoscopic surgery video frame image, and outputs triplet features through a bottleneck layer; decouples the image coding features into instrument features, action features and target tissue features based on the designed triple-decoupling attention module; fuses the triplet features, instrument features, action features and target tissue features to obtain image feature representation; fuses the text feature representation and the image feature representation to obtain text-enhanced image features; trains a temporal convolution network based on the text-enhanced image features to obtain a trained temporal convolution network; obtains second image feature representation and second instrument features of the laparoscopic surgery video frame image to be recognized; inputs the second image feature representation into the trained temporal convolution network to obtain triplet classification results; and performs foreground extraction on the second instrument features to obtain surgical instrument positioning results. The method realizes the fusion of image-text features through text feature representation guidance, significantly improves the semantic information of the laparoscopic surgery video frame, and can effectively improve the triplet recognition effect of the surgery video; at the same time, the method solves the problems of unclear feature semantics and insufficient feature robustness of single image modal, and provides a more effective and reliable solution for laparoscopic surgery video triplet recognition. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 The flowchart of the laparoscopic surgery video triplet recognition method based on text semantic enhancement in the embodiments of the present application. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below in conjunction with the drawings. In the following description, a lot of specific details are set forth in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0018] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as implying or suggesting relative importance or an indicated number of the technical features being referred to. Thus, features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0019] As shown in Figure 1 The embodiment provides a laparoscopic surgery video triple recognition method based on text semantic enhancement, which comprises the following steps: S1: design a text prompt, the text prompt comprises a standardized template prompt and a learnable prompt, and a text encoder is used to extract standardized prompt features and learnable prompt features respectively, and the text features are obtained by adding the standardized prompt features and the learnable prompt features.

[0020] Specifically, the standardized template prompt is designed as "a {LABEL} laparoscopic image", and the triple label of each laparoscopic surgery video frame image in the laparoscopic surgery video frame image data set is filled in {LABEL}; all the triple labels filled in the standardized template are respectively input into the text encoder to obtain a plurality of standardized prompt features; The learnable prompt is a learnable parameter tensor comprising 16 randomly initialized variables, and the learnable prompt is input into the text encoder to obtain learnable prompt features; All the standardized prompt features and the learnable prompt features are added to obtain the text features.

[0021] In the embodiment, the standardized template prompt and the learnable prompt are jointly designed, the features are extracted respectively and added to obtain the text features, and the semantic richness and robustness of the text features are improved.

[0022] The standardized template prompt integrates prior knowledge, can effectively summarize the semantic features of the image category, improve the general description of the image, and enable the model to effectively capture the global information in the video frame; the learnable prompt can effectively alleviate the difference between the pre-training data domain and the laparoscopic surgery video domain, and the model can adaptively learn the prompt features in a data-driven manner, thereby improving the generalization ability of the text features. By combining the two prompts, the semantic information in the video frame can be effectively captured, and the domain difference of pre-training can be effectively reduced.

[0023] In the embodiment, the laparoscopic surgery video frame image dataset adopts the standard Cholect50 dataset, which contains 161005 frames of 50 surgery videos. The images in the dataset are divided into 100 triple categories according to <surgical instrument, action, target tissue>. In order to train and test, the embodiment adopts 5-fold cross-validation to evaluate the model, and adjusts the size of all images in the dataset to 384x384 pixels.

[0024] S2: Obtain the laparoscopic surgery video frame image, extract the image coding features of the laparoscopic surgery video frame image by using the image encoder, and output the triple feature through the bottleneck layer; decouple the image coding features into instrument features, action features, and target tissue features based on the designed triple decoupling attention module; fuse the triple features, instrument features, action features, and target tissue features by using the multi-head mixed attention mechanism to obtain the image feature representation.

[0025] In the embodiment, the text encoder is a text encoder in a pre-trained CLIP model, and the image encoder is an image encoder in the pre-trained CLIP model.

[0026] In the embodiment, the training process of the CLIP model includes: Freeze the parameters of the text encoder, and set the parameters of the image encoder to be trainable; Perform data enhancement operation on the laparoscopic surgery video frame images in the laparoscopic surgery video frame image dataset to obtain an enhanced dataset, and the text feature representation and the triple label corresponding to each laparoscopic surgery video frame image after enhancement remain unchanged; Each laparoscopic surgery video frame image in the enhanced dataset is subjected to step S2 to obtain the corresponding second image feature representation; calculate the contrast loss based on each second image feature representation and the corresponding text feature representation, and the contrast loss calculation formula is: ; ; ; Wherein, represents the contrast loss; represents the expectation on the enhanced dataset; represents the KL divergence; represents the cosine similarity; represents the similarity score of the second image feature representation to the text feature representation; represents the similarity score of the text feature representation to the second image feature representation; represents the distribution of the triple label in the second image feature representation to the text feature representation; represents a distribution of the triple tag in the text feature representation to the second image feature representation; represents the second image feature representation; represents the second image feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the second image feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the second image feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the second image feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the text feature representation; represents the text feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the text feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the text feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the text feature representation corresponding to the abdominal surgery video frame image in the enhanced data set; represents the temperature parameter; D represents the enhanced data set; N represents the number of abdominal surgery video frame images in the enhanced data set; The parameters of the Adam optimizer are set, the initial second learning rate is set to 1e-5, and the cosine decay strategy is used, the second learning rate is decayed to 2e-4 during training, the minimum learning rate is set to 1e-6, and the maximum training period is set to 200 epochs. Based on the contrast loss and the Adam optimizer with the set parameters, the image encoder is optimized by gradient descent, and the parameters of the image encoder are updated; when the training reaches the maximum number of iterations, the training is stopped, and the pre-trained CLIP model is obtained.

[0027] In this embodiment, the data enhancement operation includes random horizontal flip, random vertical flip, random cropping, and random rotation.

[0028] Further, the bottleneck layer includes: a first convolutional layer including 256 channels and a convolutional kernel size of 3x3, a second convolutional layer including a triple channel and a convolutional kernel size of 1x1; the image coding feature sequentially passes through the first convolutional layer, the second convolutional layer, and outputs a triple feature.

[0029] Still further, the triple decoupling attention module includes: a third convolutional layer including 64 channels and a convolutional kernel size of 3x3, a fourth convolutional layer including an instrument feature channel and a convolutional kernel size of 1x1, a fifth convolutional layer including an action feature channel and a convolutional kernel size of 1x1, and a sixth convolutional layer including a target tissue feature channel and a convolutional kernel size of 1x1; the third convolutional layer is connected with the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer, respectively; After the image coding feature passes through the third convolutional layer, it is input to the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer, respectively, to decouple the instrument feature, the action feature, and the target tissue feature.

[0030] In the embodiment, the global maximum pooling is further included for the triplet feature, the instrument feature, the action feature, and the target tissue feature to obtain a feature vector with a length of a corresponding channel size, so as to compress the spatial resolution of each feature; and a second cross-entropy loss is calculated based on the classification probability output by the auxiliary classifier and the classification probability of the decoupled feature (including the pooled triplet feature, the instrument feature, the action feature, and the target tissue feature), and the calculation formula is: ; wherein, represents the second cross-entropy loss; n represents the number of laparoscopic surgery video frame images in the training set; represents the prediction label of the decoupled feature corresponding to the m-th laparoscopic surgery video frame image in the training set ; represents the probability distribution of the prediction label of the decoupled feature corresponding to the m-th laparoscopic surgery video frame image in the training set ; represents the probability distribution of the label output by the auxiliary classifier corresponding to the m-th laparoscopic surgery video frame image in the training set ; and the model is supervised and learned based on the second cross-entropy loss.

[0031] Specifically, the process of obtaining the image feature representation includes: In some attention heads, the triplet feature is mapped into a query vector, a key vector, and a value vector, respectively, and self-attention fusion is performed based on the query vector, the key vector, and the value vector to obtain a self-attention feature; In the remaining attention heads, the instrument feature, the action feature, and the target tissue feature are combined into a second triplet feature, and the second triplet feature is mapped into a second key vector and a second value vector, respectively; cross-attention fusion is performed based on the query vector, the second key vector, and the second value vector to obtain a cross-attention feature; The self-attention features and the cross-attention features in all attention heads are spliced to obtain a spliced feature; and the spliced feature is multiplied by the first linear projection parameter to obtain the image feature representation.

[0032] S3: a multi-head cross-attention mechanism is adopted to fuse the text feature representation and the image feature representation to obtain a text-enhanced image feature; and a time convolution network is trained based on the text-enhanced image feature to obtain a trained time convolution network.

[0033] Specifically, the process of obtaining the text-enhanced image feature includes: The image feature representation is mapped into a second query vector, and the text feature representation is mapped into a third key vector and a third value vector; The second query vector, the third key vector, and the third value vector are used to obtain the text-enhanced image feature through the multi-head cross-attention mechanism.

[0034] Further, the training process of the time convolution network comprises: dividing the laparoscopic surgery video frame image dataset into a training set and a test set; all laparoscopic surgery video frame images in the training set are subjected to steps S2-S3 to obtain corresponding text-enhanced image features respectively; inputting the text-enhanced image features into the time convolution network to output corresponding predicted labels; based on the predicted labels and the triple labels of the laparoscopic surgery video frame images, the cross-entropy loss is calculated, and the cross-entropy loss calculation formula is: ; wherein, represents the cross-entropy loss; n represents the number of laparoscopic surgery video frame images in the training set; represents the probability distribution of the predicted label of the n-th laparoscopic surgery video frame image in the training set; represents the probability distribution of the triple label of the n-th laparoscopic surgery video frame image in the training set; ; setting the parameters of the SGD optimizer, setting the initial first learning rate to 0.001, using linear decay, and setting the first learning rate to 0.0005 after 200 epochs, and setting the maximum training period to 1000 epochs; based on the cross-entropy loss and the SGD optimizer with the set parameters, the time convolution network is subjected to gradient descent optimization to update the network parameters of the time convolution network; in the training process, when the cross-entropy loss corresponding to all laparoscopic surgery video frame images in the validation set does not decrease for 10 consecutive epochs, or the training reaches the maximum number of iterations, the training is stopped, and the trained time convolution network is obtained.

[0035] S4: subjecting the laparoscopic surgery video frame image to be recognized to step S2 to obtain corresponding second image feature representation and second instrument feature; inputting the second image feature representation into the trained time convolution network to obtain a triple classification result; performing foreground extraction on the second instrument feature to obtain a surgical instrument positioning result; the visualized surgical instrument positioning result provides information guidance for surgery and is used to verify the triple classification result.

[0036] In this embodiment, the threshold value of 0.5 is used for foreground extraction on the second instrument feature, and the obtained minimum bounding rectangle is taken as the surgical instrument positioning result.

[0037] The laparoscopic surgery video triple recognition method based on text semantic enhancement provided in this embodiment has the following beneficial effects:​ The method effectively improves the problem of insufficient semantic information in the structured analysis of surgical videos through a text semantic guided cross-modal learning mechanism, and has important application value in the fields of surgical navigation and intraoperative decision support. At the same time, after training the CLIP model, the time convolution network is used to model the time sequence information of the video frame, further improving the semantic information of the feature; after training, the image of the abdominal surgery video frame to be identified successively passes through the image encoder and the time convolution network, and the triple classification result of the abdominal surgery video can be obtained, improving the applicability and reliability of the model.

[0038] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0039] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for triple recognition in abdominal surgery videos based on text semantic enhancement, characterized in that: include: S1: Design text prompts, including standardized template prompts and learnable prompts, and use a text encoder to extract standard prompt features and learnable prompt features respectively, and add the standard prompt features and learnable prompt features to obtain text feature representation; S2: Obtain laparoscopic surgery video frame images, use an image encoder to extract image coding features from the laparoscopic surgery video frame images, and output triple features through the bottleneck layer. Based on the designed ternary decoupling attention module, the image coding features are decoupled into instrument features, action features, and target tissue features. A multi-head hybrid attention mechanism is used to fuse the triple features, instrument features, action features, and target tissue features to obtain image feature representation. S3: Use a multi-head cross-attention mechanism to fuse text feature representation and image feature representation to obtain text-enhanced image features; train a temporal convolutional network based on the text-enhanced image features to obtain a trained temporal convolutional network; S4: The abdominal surgery video frame image to be identified is subjected to step S2 to obtain the corresponding second image feature representation and second instrument feature; the second image feature representation is input into the trained temporal convolutional network to obtain a triplet classification result; the foreground of the second instrument feature is extracted to obtain the surgical instrument positioning result. The visualized surgical instrument positioning result provides information guidance for the surgery and is used to verify the triplet classification result.

2. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 1 is characterized in that: The standardized template prompt is designed as "a {LABEL} laparoscopic image", and the triplet label of each laparoscopic video frame image in the laparoscopic video frame image dataset is filled into {LABEL}; all the standardized templates filled with triplet labels are respectively passed through the text encoder to obtain a number of standard prompt features; The learnable hint is a learnable parameter tensor containing 16 randomly initialized variables. The learnable hint is passed through a text encoder to obtain a learnable hint feature; All standard cue features are added to the learnable cue features to obtain the text feature representation.

3. The method for triple recognition of laparoscopic surgery videos based on text semantic enhancement according to claim 2 is characterized in that: The text encoder is a text encoder in a pre-trained CLIP model, and the image encoder is an image encoder in a pre-trained CLIP model.

4. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 1 is characterized in that: The bottleneck layer includes: a first convolution layer containing 256 channels and a convolution kernel size of 3×3, and a second convolution layer containing a triple channel and a convolution kernel size of 1×1; the image coding features pass through the first convolution layer and the second convolution layer in sequence, and output the triple features.

5. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 1 is characterized in that: The ternary decoupled attention module includes: a third convolutional layer containing 64 channels and a convolution kernel size of 3×3, a fourth convolutional layer containing a device feature channel and a convolution kernel size of 1×1, a fifth convolutional layer containing an action feature channel and a convolution kernel size of 1×1, and a sixth convolutional layer containing a target tissue feature channel and a convolution kernel size of 1×1; the third convolutional layer is connected to the fourth, fifth, and sixth convolutional layers respectively; After passing through the third convolutional layer, the image coding features are input into the fourth, fifth, and sixth convolutional layers respectively to decouple the device features, action features, and target tissue features.

6. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 1 is characterized in that: The process of obtaining image feature representation includes: In some attention heads, the triplet features are mapped into query vector, key vector, and value vector respectively, and self-attention fusion is performed based on the query vector, key vector, and value vector to obtain self-attention features; In the remaining attention heads, the device features, action features, and target tissue features are combined into a second triplet feature, and the second triplet feature is mapped to a second key vector and a second value vector respectively; cross-attention fusion is performed based on the query vector, the second key vector, and the second value vector to obtain a cross-attention feature; The self-attention features and cross-attention features in all attention heads are concatenated to obtain the concatenated features; the concatenated features are multiplied by the first linear projection parameters to obtain the image feature representation.

7. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 1 is characterized in that: The process of obtaining text-enhanced image features includes: Mapping the image feature representation into a second query vector, and mapping the text feature representation into a third key vector and a third value vector; Based on the second query vector, the third key vector and the third value vector, and through a multi-head cross attention mechanism, the text-enhanced image features are obtained.

8. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 2 is characterized in that: The training process of the temporal convolutional network includes: The abdominal surgery video frame image dataset is divided into a training set and a test set; All the abdominal surgery video frames in the training set are subjected to steps S2-S3 to obtain the corresponding text-enhanced image features; Input the text-enhanced image features into the temporal convolutional network and output the corresponding predicted labels; The cross entropy loss is calculated based on the predicted label and the triplet label of the abdominal surgery video frame image; Set the parameters of the SGD optimizer. The initial first learning rate is set to 0.001 and linear decay is used. After the 200th epoch, the first learning rate decays to 0.0005. The maximum training cycle is set to 1000 epochs. Based on the cross entropy loss and the SGD optimizer with set parameters, the temporal convolutional network is optimized by gradient descent to update the network parameters of the temporal convolutional network. During the training process, when the cross entropy loss corresponding to all abdominal surgery video frame images in the validation set does not decrease for 10 consecutive epochs, or the training reaches the maximum number of iterations, the training is stopped and the trained temporal convolutional network is obtained.

9. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 3 is characterized in that: The training process of the CLIP model includes: Freeze the parameters of the text encoder and set the parameters of the image encoder to trainable; Data augmentation is performed on the abdominal surgery video frame images in the abdominal surgery video frame image dataset to obtain an enhanced dataset. After the augmentation, the text feature representation and triplet label corresponding to each abdominal surgery video frame image remain unchanged. Each abdominal surgery video frame image in the enhanced data set is subjected to step S2 to obtain the corresponding second image feature representation; the contrast loss is calculated based on each second image feature representation and the corresponding text feature representation. The contrast loss calculation formula is: ； ； ； in, represents contrast loss; Represents expectations on the augmented dataset; represents KL divergence; represents cosine similarity; represents a similarity score between the second image feature representation and the text feature representation; represents a similarity score between the text feature representation and the second image feature representation; Represents the distribution of triplet labels from the second image feature representation to the text feature representation; Represents the distribution of triplet labels from text feature representation to the second image feature representation; represents a second image feature representation; Indicates the first A second image feature representation corresponding to a frame image of a laparoscopic surgery video; Indicates the first A second image feature representation corresponding to a frame image of a laparoscopic surgery video; Represents text feature representation; Indicates the first Text feature representation corresponding to the abdominal surgery video frame image; Indicates the first Text feature representation corresponding to the abdominal surgery video frame image; represents the temperature parameter; D represents the enhanced dataset; N represents the number of abdominal surgery video frame images in the enhanced dataset; Set the parameters of the Adam optimizer. The initial second learning rate is set to 1e-5, and the cosine decay strategy is used. During training, the second learning rate decays to 2e-4, the minimum learning rate is set to 1e-6, and the maximum training cycle is set to 200 epochs. Based on the contrast loss and the Adam optimizer with set parameters, the image encoder is optimized by gradient descent to update the parameters of the image encoder. When the training reaches the maximum number of iterations, the training is stopped to obtain the pre-trained CLIP model.

10. The method for triple recognition of abdominal surgery videos based on text semantic enhancement according to claim 1, characterized in that: Obtaining the surgical instrument positioning result includes: performing foreground extraction on the second instrument feature using a threshold of 0.5, and using the obtained minimum bounding rectangle as the surgical instrument positioning result.

Images