A laparoscopic surgery video triple recognition method based on text semantic enhancement

Abstract

This application relates to a method for triplet recognition in laparoscopic surgery videos based on text semantic enhancement, comprising: designing text prompts and extracting their features to obtain text feature representations; using an image encoder to extract image coding features of laparoscopic surgery video frame images, and outputting triplet features through a bottleneck layer; decoupling the image coding features into instrument features, action features, and target tissue features based on a designed triplet decoupling attention module; fusing the triplet features, instrument features, action features, and target tissue features to obtain image feature representations; fusing the text feature representations and image feature representations to obtain text-enhanced image features; training a temporal convolutional network based on the text-enhanced image features to obtain a trained temporal convolutional network; obtaining a second image feature representation of the laparoscopic surgery video frame images to be recognized; and inputting the second image feature representation into the trained temporal convolutional network to obtain triplet classification results.

Description

Technical Field

[0001] This application relates to the field of surgical video recognition technology, and in particular to a method for recognizing triples in laparoscopic surgical videos based on text semantic enhancement. Background Technology

[0002] The triplet in laparoscopic surgery video contains information on surgical instruments, surgical actions, and the target tissue, which is of great significance for intraoperative guidance and postoperative analysis. Currently, it is difficult to eliminate the inherent ambiguity in vision based solely on image information. At the same time, it is difficult to model the semantic association between image frames and labels, thus resulting in low triplet recognition accuracy.

[0003] Using text features to enhance image features can effectively reduce the ambiguity of image information, but there are still two challenges in its implementation: 1. How to design text prompts to fully explore the semantic information of multimodal models, because different text templates will have a significant impact on the model results; 2. How to design effective multimodal feature extraction and fusion alignment methods, and how to fully integrate bimodal information to help improve the accuracy of video triplet recognition. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for triplet recognition in laparoscopic surgery videos based on text semantic enhancement, including:

[0005] S1: Design text prompts, which include standardized template prompts and learnable prompts. Use a text encoder to extract the standard prompt features and learnable prompt features respectively. Add the standard prompt features and learnable prompt features to obtain the text feature representation.

[0006] S2: Acquire video frame images of abdominal surgery, extract image coding features of the video frame images of abdominal surgery using an image encoder, and output triplet features through a bottleneck layer; based on the designed triplet decoupled attention module, decouple the image coding features into instrument features, motion features, and target tissue features; use a multi-head hybrid attention mechanism to fuse triplet features, instrument features, motion features, and target tissue features to obtain image feature representation;

[0007] S3: Employ a multi-head cross-attention mechanism to fuse text feature representations and image feature representations to obtain text-enhanced image features; train a temporal convolutional network based on the text-enhanced image features to obtain a well-trained temporal convolutional network;

[0008] S4: The video frame image of the abdominal surgery to be identified is processed through step S2 to obtain the corresponding second image feature representation and second instrument feature; the second image feature representation is input into the trained temporal convolutional network to obtain the triplet classification result; the foreground is extracted from the second instrument feature to obtain the surgical instrument localization result. The visualized surgical instrument localization result provides information guidance for the surgery and is used to verify the triplet classification result.

[0009] Preferably, the standardized template prompt is designed as "a {LABEL} laparoscopic image", and the triplet labels of each laparoscopic surgery video frame image in the laparoscopic surgery video frame image dataset are filled into {LABEL}; all the standardized templates after filling in the triplet labels are processed by a text encoder to obtain several standard prompt features;

[0010] The learnable hint is a learnable parameter tensor containing 16 randomly initialized variables. The learnable hint is processed by a text encoder to obtain learnable hint features.

[0011] The text feature representation is obtained by adding all standard cue features to the learnable cue features.

[0012] Preferably, the text encoder is a text encoder in a pre-trained CLIP model, and the image encoder is an image encoder in a pre-trained CLIP model.

[0013] Preferably, the bottleneck layer includes: a first convolutional layer with 256 channels and a kernel size of 3×3, and a second convolutional layer with triplet channels and a kernel size of 1×1; the image encoding features pass through the first convolutional layer and the second convolutional layer in sequence to output triplet features.

[0014] Preferably, the ternary decoupled attention module includes: a third convolutional layer with 64 channels and a kernel size of 3×3, a fourth convolutional layer with a kernel size of 1×1 containing instrument feature channels, a fifth convolutional layer with a kernel size of 1×1 containing motion feature channels, and a sixth convolutional layer with a kernel size of 1×1 containing target tissue feature channels; the third convolutional layer is connected to the fourth, fifth, and sixth convolutional layers respectively;

[0015] After passing through the third convolutional layer, the image coding features are input into the fourth, fifth, and sixth convolutional layers respectively, decoupling the instrument features, motion features, and target tissue features.

[0016] Preferably, the process of obtaining image feature representation includes:

[0017] In a partial attention head, the triplet features are mapped to query vector, key vector, and value vector respectively. Self-attention fusion is then performed based on the query vector, key vector, and value vector to obtain self-attention features.

[0018] In the remaining attention head, the instrument features, motion features, and target tissue features are combined into a second triplet feature, and the second triplet feature is mapped to a second key vector and a second value vector, respectively. Cross-attention fusion is performed based on the query vector, the second key vector, and the second value vector to obtain cross-attention features.

[0019] The self-attention features and cross-attention features from all attention heads are concatenated to obtain the concatenated features; the concatenated features are multiplied by the first linear projection parameter to obtain the image feature representation.

[0020] Preferably, the process of obtaining text-enhanced image features includes:

[0021] The image feature representation is mapped to the second query vector, and the text feature representation is mapped to the third key vector and the third value vector;

[0022] Based on the second query vector, the third key vector, and the third value vector, and through a multi-head cross-attention mechanism, the text-enhanced image features are obtained.

[0023] Preferably, the training process for a temporal convolutional network includes:

[0024] The dataset of laparoscopic surgery video frame images was divided into a training set and a test set.

[0025] All abdominal surgery video frame images in the training set are processed through steps S2-S3 to obtain the corresponding text-enhanced image features;

[0026] The text-enhanced image features are input into a temporal convolutional network, which outputs the corresponding predicted labels.

[0027] Cross-entropy loss is calculated based on the predicted labels and triplet labels of laparoscopic surgery video frame images.

[0028] Set the parameters of the SGD optimizer, with the initial first learning rate set to 0.001 and linear decay used. After the 200th epoch, the first learning rate decays to 0.0005, and the maximum training epoch is set to 1000 epochs.

[0029] Based on cross-entropy loss and a well-configured SGD optimizer, gradient descent optimization is performed on the temporal convolutional network to update the network parameters.

[0030] During training, training is stopped when the cross-entropy loss corresponding to all abdominal surgery video frame images in the validation set does not decrease for 10 consecutive epochs, or when the training reaches the maximum number of iterations, and the trained temporal convolutional network is obtained.

[0031] Preferably, the training process of the CLIP model includes:

[0032] Freeze the parameters of the text encoder and set the parameters of the image encoder for training;

[0033] Data augmentation was performed on the abdominal surgery video frame images in the abdominal surgery video frame image dataset to obtain an augmented dataset. After augmentation, the text feature representation and triplet labels of each abdominal surgery video frame image remained unchanged.

[0034] Each abdominal surgery video frame image in the augmented dataset is processed through step S2 to obtain its corresponding second image feature representation. The contrast loss is then calculated based on each second image feature representation and its corresponding text feature representation. The contrast loss calculation formula is as follows:

[0035] ;

[0036] ;

[0037] ;

[0038] in, Indicates comparative loss; This represents the expectation on the augmented dataset; Indicates KL divergence; Indicates cosine similarity; The similarity score between the second image feature representation and the text feature representation is represented. The similarity score represents the difference between the text feature representation and the second image feature representation. This represents the distribution of triplet labels from the second image feature representation to the text feature representation; This represents the distribution of triplet labels from text feature representation to second image feature representation; This represents the second image feature representation; Indicates the first in the augmented dataset The second image feature representation corresponding to the video frame image of Zhang's abdominal surgery; Indicates the first in the augmented dataset The second image feature representation corresponding to the video frame image of Zhang's abdominal surgery; Representing text features; Indicates the first in the augmented dataset Text feature representation of video frame images from an abdominal surgery procedure; Indicates the first in the augmented dataset Text feature representation of video frame images from an abdominal surgery procedure; The parameter represents temperature; D represents the augmented dataset; N represents the number of laparoscopic surgery video frame images in the augmented dataset.

[0039] Set the parameters of the Adam optimizer, with the initial second learning rate set to 1e-5 and a cosine decay strategy used. During training, the second learning rate decays to 2e-4, the minimum learning rate is set to 1e-6, and the maximum training period is set to 200 epochs.

[0040] Based on contrastive loss and the Adam optimizer with pre-set parameters, gradient descent optimization is performed on the image encoder to update the parameters of the image encoder; when the training reaches the maximum number of iterations, training stops and a pre-trained CLIP model is obtained.

[0041] Preferably, obtaining the surgical instrument localization result includes: extracting the foreground using a threshold of 0.5 for the second instrument feature, and using the obtained minimum bounding rectangle as the surgical instrument localization result.

[0042] Beneficial effects: This method obtains text feature representation by designing text prompts and extracting their features; it uses an image encoder to extract image coding features from laparoscopic surgery video frame images, and outputs triplet features through a bottleneck layer; based on the designed triple decoupled attention module, the image coding features are decoupled into instrument features, action features, and target tissue features; the triplet features, instrument features, action features, and target tissue features are fused to obtain image feature representation; the text feature representation and image feature representation are fused to obtain text-enhanced image features; a temporal convolutional network is trained based on the text-enhanced image features to obtain a trained temporal convolutional network; the second image feature representation and the second instrument feature of the laparoscopic surgery video frame image to be identified are obtained; the second image feature representation is input into the trained temporal convolutional network to obtain the triplet classification result; foreground extraction is performed on the second instrument feature to obtain the surgical instrument localization result. This method, guided by text feature representation, achieves the fusion of image and text features, significantly improving the semantic information of laparoscopic surgery video frames and effectively enhancing the recognition performance of surgical video triples. At the same time, this method solves the problems of unclear semantics and insufficient feature robustness of single-image modal features, providing a more effective and reliable solution for laparoscopic surgery video triple recognition. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the method for identifying triplets in laparoscopic surgery videos based on text semantic enhancement in the embodiments of this application. Detailed Implementation

[0045] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0046] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0047] like Figure 1 As shown, this embodiment provides a method for triplet recognition in laparoscopic surgery videos based on text semantic enhancement, including:

[0048] S1: Design text prompts, which include standardized template prompts and learnable prompts. Use a text encoder to extract the standard prompt features and learnable prompt features respectively, and add the standard prompt features and learnable prompt features to obtain the text feature representation.

[0049] Specifically, the standardized template prompt is designed as "a {LABEL} laparoscopic image", and the triplet labels of each laparoscopic surgery video frame image in the laparoscopic surgery video frame image dataset are filled into {LABEL}; all the standardized templates after filling in the triplet labels are processed by a text encoder to obtain several standard prompt features;

[0050] The learnable hint is a learnable parameter tensor containing 16 randomly initialized variables. The learnable hint is processed by a text encoder to obtain learnable hint features.

[0051] The text feature representation is obtained by adding all standard cue features to the learnable cue features.

[0052] In this embodiment, standardized template prompts and learnable prompts were jointly designed, and features were extracted from them and added together to obtain text feature representations, thereby improving the semantic richness and robustness of text features.

[0053] Standardized template prompts, incorporating prior knowledge, can effectively summarize the semantic features of image categories, improving the generalization of image descriptions and enabling the model to effectively capture global information within video frames. Learnable prompts effectively mitigate the differences between the pre-training data domain and the laparoscopic surgery video domain. Through a data-driven approach, the model adaptively learns prompt features, improving the generalization ability of text features. By combining these two types of prompts, not only can semantic information within video frames be effectively captured, but the domain differences from pre-training can also be significantly reduced.

[0054] In this embodiment, the abdominal surgery video frame image dataset uses the standard Cholect50 dataset, which contains 161,005 frames from 50 surgical videos. The images in the dataset are categorized into 100 triplet classes based on <surgical instrument, action, target tissue>. For training and testing, this embodiment uses 5-fold cross-validation to evaluate the model, and all images in the dataset are resized to 384×384 pixels.

[0055] S2: Acquire video frame images of abdominal surgery, extract image coding features of the video frame images of abdominal surgery using an image encoder, and output triple features of the image coding features through a bottleneck layer; based on the designed triple decoupled attention module, decouple the image coding features into instrument features, motion features, and target tissue features; use a multi-head hybrid attention mechanism to fuse triple features, instrument features, motion features, and target tissue features to obtain image feature representation.

[0056] In this embodiment, the text encoder is the text encoder in the pre-trained CLIP model, and the image encoder is the image encoder in the pre-trained CLIP model.

[0057] In this embodiment, the training process of the CLIP model includes:

[0058] Freeze the parameters of the text encoder and set the parameters of the image encoder for training;

[0059] Data augmentation was performed on the abdominal surgery video frame images in the abdominal surgery video frame image dataset to obtain an augmented dataset. After augmentation, the text feature representation and triplet labels of each abdominal surgery video frame image remained unchanged.

[0060] Each abdominal surgery video frame image in the augmented dataset is processed through step S2 to obtain its corresponding second image feature representation. The contrast loss is then calculated based on each second image feature representation and its corresponding text feature representation. The contrast loss calculation formula is as follows:

[0061] ;

[0062] ;

[0063] ;

[0064] in, Indicates comparative loss; This represents the expectation on the augmented dataset; Indicates KL divergence; Indicates cosine similarity; The similarity score between the second image feature representation and the text feature representation is represented. The similarity score represents the difference between the text feature representation and the second image feature representation. This represents the distribution of triplet labels from the second image feature representation to the text feature representation; This represents the distribution of triplet labels from text feature representation to second image feature representation; This represents the second image feature representation; Indicates the first in the augmented dataset The second image feature representation corresponding to the video frame image of Zhang's abdominal surgery; Indicates the first in the augmented dataset The second image feature representation corresponding to the video frame image of Zhang's abdominal surgery; Representing text features; Indicates the first in the augmented dataset Text feature representation of video frame images from an abdominal surgery procedure; Indicates the first in the augmented dataset Text feature representation of video frame images from an abdominal surgery procedure; The parameter represents temperature; D represents the augmented dataset; N represents the number of laparoscopic surgery video frame images in the augmented dataset.

[0065] Set the parameters of the Adam optimizer, with the initial second learning rate set to 1e-5 and a cosine decay strategy used. During training, the second learning rate decays to 2e-4, the minimum learning rate is set to 1e-6, and the maximum training period is set to 200 epochs.

[0066] Based on contrastive loss and the Adam optimizer with pre-set parameters, gradient descent optimization is performed on the image encoder to update the parameters of the image encoder; when the training reaches the maximum number of iterations, training stops and a pre-trained CLIP model is obtained.

[0067] In this embodiment, the data augmentation operations include random horizontal flipping, random vertical flipping, random cropping, and random rotation.

[0068] Furthermore, the bottleneck layer includes: a first convolutional layer with 256 channels and a kernel size of 3×3, and a second convolutional layer with triplet channels and a kernel size of 1×1; the image encoding features pass through the first convolutional layer and the second convolutional layer in sequence to output triplet features.

[0069] Furthermore, the ternary decoupled attention module includes: a third convolutional layer with 64 channels and a kernel size of 3×3, a fourth convolutional layer with a kernel size of 1×1 containing instrument feature channels, a fifth convolutional layer with a kernel size of 1×1 containing motion feature channels, and a sixth convolutional layer with a kernel size of 1×1 containing target tissue feature channels; the third convolutional layer is connected to the fourth, fifth, and sixth convolutional layers respectively.

[0070] After passing through the third convolutional layer, the image coding features are input into the fourth, fifth, and sixth convolutional layers respectively, decoupling the instrument features, motion features, and target tissue features.

[0071] In this embodiment, the method further includes obtaining feature vectors of length corresponding to the channel size for each of the triplet features, instrument features, action features, and target tissue features through global max pooling, thereby compressing the spatial resolution of each feature; and calculating the second cross-entropy loss based on the classification probability output by the auxiliary classifier and the classification probability of the decoupled features (including the pooled triplet features, instrument features, action features, and target tissue features), with the following formula:

[0072] ;

[0073] in, The second cross-entropy loss is represented by n; n represents the number of laparoscopic surgery video frames in the training set. Indicates the first training set Predicted labels of decoupled features corresponding to video frames from laparoscopic surgery. The probability distribution; Indicates the first training set Labels output by the auxiliary classifier corresponding to the video frames of an abdominal surgery. The probability distribution; supervised learning of the model based on the second cross-entropy loss.

[0074] Specifically, the process of obtaining image feature representations includes:

[0075] In a partial attention head, the triplet features are mapped to query vector, key vector, and value vector respectively. Self-attention fusion is then performed based on the query vector, key vector, and value vector to obtain self-attention features.

[0076] In the remaining attention head, the instrument features, motion features, and target tissue features are combined into a second triplet feature, and the second triplet feature is mapped to a second key vector and a second value vector, respectively. Cross-attention fusion is performed based on the query vector, the second key vector, and the second value vector to obtain cross-attention features.

[0077] The self-attention features and cross-attention features from all attention heads are concatenated to obtain the concatenated features; the concatenated features are multiplied by the first linear projection parameter to obtain the image feature representation.

[0078] S3: Employ a multi-head cross-attention mechanism to fuse text feature representations and image feature representations to obtain text-enhanced image features; train a temporal convolutional network based on the text-enhanced image features to obtain a well-trained temporal convolutional network.

[0079] Specifically, the process of obtaining text-enhanced image features includes:

[0080] The image feature representation is mapped to the second query vector, and the text feature representation is mapped to the third key vector and the third value vector;

[0081] Based on the second query vector, the third key vector, and the third value vector, and through a multi-head cross-attention mechanism, the text-enhanced image features are obtained.

[0082] Furthermore, the training process for temporal convolutional networks includes:

[0083] The dataset of laparoscopic surgery video frame images was divided into a training set and a test set.

[0084] All abdominal surgery video frame images in the training set are processed through steps S2-S3 to obtain the corresponding text-enhanced image features;

[0085] The text-enhanced image features are input into a temporal convolutional network, which outputs the corresponding predicted labels.

[0086] Based on the predicted labels and the triplet labels of the laparoscopic surgery video frame images, the cross-entropy loss is calculated. The formula for calculating the cross-entropy loss is:

[0087] ;

[0088] in, The cross-entropy loss is represented by n; n represents the number of abdominal surgery video frames in the training set. Indicates the first training set Predicted labels for video frames of laparoscopic surgery The probability distribution; Indicates the first training set Triple tags for video frames of laparoscopic surgery The probability distribution;

[0089] Set the parameters of the SGD optimizer, with the initial first learning rate set to 0.001 and linear decay used. After the 200th epoch, the first learning rate decays to 0.0005, and the maximum training epoch is set to 1000 epochs.

[0090] Based on cross-entropy loss and a well-configured SGD optimizer, gradient descent optimization is performed on the temporal convolutional network to update the network parameters.

[0091] During training, training is stopped when the cross-entropy loss corresponding to all abdominal surgery video frame images in the validation set does not decrease for 10 consecutive epochs, or when the training reaches the maximum number of iterations, and the trained temporal convolutional network is obtained.

[0092] S4: The video frame image of the abdominal surgery to be identified is processed through step S2 to obtain the corresponding second image feature representation and second instrument feature; the second image feature representation is input into the trained temporal convolutional network to obtain the triplet classification result; the foreground is extracted from the second instrument feature to obtain the surgical instrument localization result. The visualized surgical instrument localization result provides information guidance for the surgery and is used to verify the triplet classification result.

[0093] In this embodiment, a threshold of 0.5 is used to extract the foreground of the second instrument feature, and the obtained minimum bounding rectangle is used as the surgical instrument positioning result.

[0094] The text-based semantic enhancement-based triplet recognition method for laparoscopic surgery videos provided in this embodiment has the following beneficial effects:

[0095] This method effectively addresses the lack of semantic information in the structured analysis of surgical videos through a text-based semantic-guided cross-modal learning mechanism, demonstrating significant application value in areas such as surgical navigation and intraoperative decision support. Furthermore, after training the CLIP model, a temporal convolutional network is used to model the temporal information of video frames, further enhancing the semantic information of the features. After training, the images of the abdominal surgery video frames to be identified are sequentially passed through an image encoder and a temporal convolutional network to obtain the triplet classification results for the abdominal surgery video, improving the model's applicability and reliability.

[0096] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for recognizing triples in laparoscopic surgery videos based on text semantic enhancement, characterized in that, include: S1: Design text prompts, which include standardized template prompts and learnable prompts. Use a text encoder to extract the standard prompt features and learnable prompt features respectively. Add the standard prompt features and learnable prompt features to obtain the text feature representation. S2: Acquire video frame images of abdominal surgery, extract image coding features of the video frame images of abdominal surgery using an image encoder, and output triplet features through a bottleneck layer; based on the designed triplet decoupled attention module, decouple the image coding features into instrument features, motion features, and target tissue features; use a multi-head hybrid attention mechanism to fuse triplet features, instrument features, motion features, and target tissue features to obtain image feature representation; S3: Employ a multi-head cross-attention mechanism to fuse text feature representations and image feature representations to obtain text-enhanced image features; train a temporal convolutional network based on the text-enhanced image features to obtain a well-trained temporal convolutional network; S4: The video frame image of the abdominal surgery to be identified is processed through step S2 to obtain the corresponding second image feature representation and second instrument feature; the second image feature representation is input into the trained temporal convolutional network to obtain the triplet classification result; the foreground is extracted from the second instrument feature to obtain the surgical instrument localization result. The visualized surgical instrument localization result provides information guidance for the surgery and is used to verify the triplet classification result.

2. The method for identifying triplets in laparoscopic surgery videos based on text semantic enhancement according to claim 1, characterized in that, The standardized template prompt is designed as "a {LABEL} laparoscopic image", and the triplet labels of each laparoscopic surgery video frame image in the laparoscopic surgery video frame image dataset are filled into {LABEL}; all the standardized templates after filling in the triplet labels are processed by a text encoder to obtain several standard prompt features; The learnable hint is a learnable parameter tensor containing 16 randomly initialized variables. The learnable hint is processed by a text encoder to obtain learnable hint features. The text feature representation is obtained by adding all standard cue features to the learnable cue features.

3. The method for identifying triplets in laparoscopic surgery videos based on text semantic enhancement according to claim 2, characterized in that, The text encoder is the text encoder in the pre-trained CLIP model, and the image encoder is the image encoder in the pre-trained CLIP model.

4. The method for identifying triples in laparoscopic surgery videos based on text semantic enhancement according to claim 1, characterized in that, The bottleneck layer includes: a first convolutional layer with 256 channels and a kernel size of 3×3, and a second convolutional layer with triplet channels and a kernel size of 1×1; the image encoding features pass through the first convolutional layer and the second convolutional layer in sequence, and output triplet features.

5. The method for identifying triplets in laparoscopic surgery videos based on text semantic enhancement according to claim 1, characterized in that, The ternary decoupled attention module includes: a third convolutional layer with 64 channels and a kernel size of 3×3, a fourth convolutional layer with a kernel size of 1×1 containing instrument feature channels, a fifth convolutional layer with a kernel size of 1×1 containing motion feature channels, and a sixth convolutional layer with a kernel size of 1×1 containing target tissue feature channels; the third convolutional layer is connected to the fourth, fifth, and sixth convolutional layers respectively; After passing through the third convolutional layer, the image coding features are input into the fourth, fifth, and sixth convolutional layers respectively, decoupling the instrument features, motion features, and target tissue features.

6. The method for identifying triples in laparoscopic surgery videos based on text semantic enhancement according to claim 1, characterized in that, The process of obtaining image feature representations includes: In a partial attention head, the triplet features are mapped to query vector, key vector, and value vector respectively. Self-attention fusion is then performed based on the query vector, key vector, and value vector to obtain self-attention features. In the remaining attention head, the instrument features, motion features, and target tissue features are combined into a second triplet feature, and the second triplet feature is mapped to a second key vector and a second value vector, respectively. Cross-attention fusion is performed based on the query vector, the second key vector, and the second value vector to obtain cross-attention features. The self-attention features and cross-attention features from all attention heads are concatenated to obtain the concatenated features; the concatenated features are multiplied by the first linear projection parameter to obtain the image feature representation.

7. The method for identifying triples in laparoscopic surgery videos based on text semantic enhancement according to claim 1, characterized in that, The process of obtaining text-enhanced image features includes: The image feature representation is mapped to the second query vector, and the text feature representation is mapped to the third key vector and the third value vector; Based on the second query vector, the third key vector, and the third value vector, and through a multi-head cross-attention mechanism, the text-enhanced image features are obtained.

8. The method for identifying triplets in laparoscopic surgery videos based on text semantic enhancement according to claim 2, characterized in that, The training process of a temporal convolutional network includes: The dataset of laparoscopic surgery video frame images was divided into a training set and a test set. All abdominal surgery video frame images in the training set are processed through steps S2-S3 to obtain the corresponding text-enhanced image features; The text-enhanced image features are input into a temporal convolutional network, which outputs the corresponding predicted labels. Cross-entropy loss is calculated based on the predicted labels and triplet labels of laparoscopic surgery video frame images. Set the parameters of the SGD optimizer, with the initial first learning rate set to 0.001 and linear decay used. After the 200th epoch, the first learning rate decays to 0.0005, and the maximum training epoch is set to 1000 epochs. Based on cross-entropy loss and a well-configured SGD optimizer, gradient descent optimization is performed on the temporal convolutional network to update the network parameters. During training, training is stopped when the cross-entropy loss corresponding to all abdominal surgery video frame images in the validation set does not decrease for 10 consecutive epochs, or when the training reaches the maximum number of iterations, and the trained temporal convolutional network is obtained.

9. The method for identifying triples in laparoscopic surgery videos based on text semantic enhancement according to claim 3, characterized in that, The training process of the CLIP model includes: Freeze the parameters of the text encoder and set the parameters of the image encoder for training; Data augmentation was performed on the abdominal surgery video frame images in the abdominal surgery video frame image dataset to obtain an augmented dataset. After augmentation, the text feature representation and triplet labels of each abdominal surgery video frame image remained unchanged. Each abdominal surgery video frame image in the augmented dataset is processed through step S2 to obtain its corresponding second image feature representation. The contrast loss is then calculated based on each second image feature representation and its corresponding text feature representation. The contrast loss calculation formula is as follows: ； ； ； in, Indicates comparative loss; This represents the expectation on the augmented dataset; Indicates KL divergence; Indicates cosine similarity; The similarity score between the second image feature representation and the text feature representation is represented. The similarity score represents the difference between the text feature representation and the second image feature representation; This represents the distribution of triplet labels from the second image feature representation to the text feature representation; This represents the distribution of triplet labels from text feature representation to second image feature representation; This represents the second image feature representation; Indicates the first in the augmented dataset The second image feature representation corresponding to the video frame image of Zhang's abdominal surgery; Indicates the first in the augmented dataset The second image feature representation corresponding to the video frame image of Zhang's abdominal surgery; Representing text features; Indicates the first in the augmented dataset Text feature representation of video frame images from an abdominal surgery procedure; Indicates the first in the augmented dataset Text feature representation of video frame images from an abdominal surgery procedure; The parameter represents temperature; D represents the augmented dataset; N represents the number of laparoscopic surgery video frame images in the augmented dataset. Set the parameters of the Adam optimizer, with the initial second learning rate set to 1e-5 and a cosine decay strategy used. During training, the second learning rate decays to 2e-4, the minimum learning rate is set to 1e-6, and the maximum training period is set to 200 epochs. Based on contrastive loss and the Adam optimizer with pre-set parameters, gradient descent optimization is performed on the image encoder to update the parameters of the image encoder; when the training reaches the maximum number of iterations, training stops and a pre-trained CLIP model is obtained.

10. The method for identifying triples in laparoscopic surgery videos based on text semantic enhancement according to claim 1, characterized in that, The surgical instrument localization results are obtained by: extracting the foreground using a threshold of 0.5 for the second instrument feature, and using the obtained minimum bounding rectangle as the surgical instrument localization result.

Images