A method and system for generating text descriptions of endoscopic surgical images across modalities.

Abstract

This invention belongs to the field of image data processing and generation technology, and specifically provides a method and system for generating text descriptions of endoscopic surgical images across modalities. It includes a scene feature extraction module for parallel processing of endoscopic surgical images, extracting global and local visual features of the surgical scene; an instance encoding module for accurately identifying key objects such as surgical instruments and organs in the endoscopic surgical images, generating corresponding classification information and encoding it into image category features; a cross-modal fusion module for performing cross-modal feature fusion of image category features and global visual features through an adaptive channel overlay method; and a text description module for decoding the cross-modal fused features to generate text descriptions of the endoscopic surgical images. This invention, through cross-modal feature fusion technology, effectively combines the semantic information of instrument categories and panoramic visual information in the surgical scene, improving the accuracy and completeness of endoscopic surgical image processing and analysis.

Description

Technical Field

[0001] This invention relates to the field of image data processing and generation technology, specifically to a method and system for generating text descriptions of endoscopic surgical images across modalities. Background Technology

[0002] During endoscopic surgery, the surgical assistant's effective understanding of the surgical process and procedures is crucial for efficient collaboration with the surgeon. Standardized interpretation of surgical procedures is also essential for improving the surgical skills of medical students, helping them clearly identify surgical steps, accurately understand instrument manipulation, and intuitively recognize anatomical structures. However, challenges remain in improving surgical collaboration efficiency and learning outcomes due to the asymmetry in processing scene information in endoscopic surgical images and the difficulty in interpreting the dynamic and complex relationships of surgical operations. Traditional endoscopic surgical scene understanding models aim at surgical process analysis, step identification, and scene segmentation, using category labels as the output of understanding, lacking complete textual descriptions and contextual information.

[0003] Currently, research on endoscopic image processing focuses on the segmentation and recognition of surgical scenes. Existing endoscopic image segmentation methods typically target specific types of instruments, organs, or small tissues, and relying solely on region features is insufficient to fully express the dynamic relationships between surgical instruments and organs. Current endoscopic image recognition methods often use triplet annotations such as surgical instruments, manipulation actions, and organs, lacking detailed descriptions of surgical procedures.

[0004] The definition and extraction mechanism of interactive actions play a crucial role in the accuracy of surgical scene understanding. However, the existing descriptions of endoscopic surgical images lack predicate verbs that connect interactive relationships, leading to confusion in the model's recognition of encoded semantic features.

[0005] The integration of computer vision and natural language processing has driven the development of natural image text description, potentially serving as a technical reference for scene understanding in endoscopic surgical images. However, due to the high similarity of surgical instruments in images, existing natural image text description models struggle to learn features from images to generate text descriptions that match endoscopic images. Furthermore, insufficient training of text models and generation of dictionary datasets for endoscopic surgical images limit the accuracy and generalization ability of scene understanding. Therefore, despite significant progress in endoscopic image scene segmentation and image text description generation techniques by researchers both domestically and internationally, text description generation for endoscopic surgical images remains a gap.

[0006] In summary, due to factors such as the lack of text description annotations for endoscopic surgical images, insufficient training of text models, and the complex interaction between surgical instruments and tissues, there is an urgent need for a method and system for generating text descriptions for endoscopic surgical images. Summary of the Invention

[0007] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0008] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0009] A method for generating text descriptions of endoscopic surgical images across modalities, comprising the following steps:

[0010] S1: Using a transferable distillation vision Transformer as the implementation path, extract global and local detail visual features in the endoscopic surgery scene;

[0011] S2: The visual Transformer model is used as the implementation path for instance classification, and the generated category features are presented in the form of one-hot encoding.

[0012] S3: Cross-modal fusion concatenates scene features and instance codes by overlaying channels to form fused features. The feature fusion module adaptively optimizes the representation of features from different channels through a squeeze-excitation attention mechanism, as shown in the formula:

[0013] ;

[0014] in, Represented as instance encoding features, Represented as global features of the scene. Represented as cross-modal fusion features, each channel of the fusion feature represents a different feature. The squeeze-excitement attention mechanism compresses the information of each feature channel into a single value to form a channel descriptor through global pooling operation. Subsequently, after transformation by a small fully connected network, the importance weights of each channel are generated.

[0015] S4: The decoder receives cross-modal fusion features from the encoder, processes and transforms these features through a multi-layer self-attention mechanism and a feedforward neural network to generate a feature sequence; it matches the feature sequence with a predefined corpus, performs matrix multiplication using the mapping matrix generated from the corpus, and generates a probability distribution of each text position mapped to each word in the vocabulary.

[0016] As a preferred embodiment of the cross-modal endoscopic surgery image text description generation method described in this invention, the specific method of step S1 is as follows: First, the teacher model loads a pre-trained visual Transformer parameter model on a large-scale natural image dataset, uses these distillation tokens to capture the global contextual information of the endoscopic image, integrates these distillation tokens with the image patch embedding vector to form a complete input sequence, transfers it to the student model of the distilled visual Transformer for processing, and generates a scene feature representation of the endoscopic image through multi-layer self-attention mechanism and feedforward neural network encoding.

[0017] As a preferred embodiment of the cross-modal endoscopic surgical image text description generation method of the present invention, in step S3, global features are used to represent the positional information of surgical instruments and tissues in the image and the interaction between instruments and tissues, providing accurate visual semantic information for fusion features; category features are used to represent the surgical instruments and tissues contained in the endoscopic image, providing accurate classification information for fusion features; by combining scene features and category features, the fusion features integrate the category information and visual information of the image.

[0018] As a preferred embodiment of the cross-modal endoscopic surgery image text description generation method described in this invention, in step S4, the Beam search method is used to select several words with the highest probability as candidate words at each time step to generate a new candidate sequence. If a candidate sequence generates an end marker, it is added to the final result list. This process continues until the generated text sequence reaches the preset maximum length, and finally, a text description for understanding the endoscopic surgery scene is generated.

[0019] A cross-modal endoscopic surgery image text description generation system, the system comprising:

[0020] Scene feature extraction module: performs image preprocessing by converting segmentation masks into classification labels, and extracts global visual features of organ anatomical structure and instrument spatial location through the scene feature extraction module;

[0021] Instance encoding module: used to identify the category information of surgical instruments in endoscopic surgical images and convert it into label encoding;

[0022] Cross-modal fusion module: used to fuse scene features with instance codes to highlight key feature representations of the device-tissue interaction area;

[0023] Text description module: used to decode cross-modal fusion features, match words and optimize word order;

[0024] Data input and output modules: The system input is endoscopic surgical images; the output module is used to generate complete text descriptions of the surgical images as the system output.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention effectively combines the semantic information of instrument categories and panoramic visual information in the surgical scene through cross-modal feature fusion technology, which improves the accuracy and completeness of endoscopic surgical image processing and analysis, and provides precise auxiliary information for applications such as surgical navigation, intraoperative decision support and surgical teaching. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0027] Figure 1 This is a flowchart illustrating the scene feature extraction process of a cross-modal endoscopic surgery image text description generation method according to the present invention.

[0028] Figure 2 This is an example coding flowchart of a cross-modal endoscopic surgery image text description generation method according to the present invention;

[0029] Figure 3 This is a cross-modal fusion flowchart of a cross-modal endoscopic surgery image text description generation method according to the present invention;

[0030] Figure 4 This is a flowchart of a cross-modal endoscopic surgical image text description generation system according to the present invention;

[0031] Figure 5 This is a visualization of the text description generation results for different models in an embodiment of the cross-modal endoscopic surgery image text description generation method and system of the present invention. Detailed Implementation

[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0035] This invention provides a cross-modal method for generating text descriptions of endoscopic surgical images. This method effectively extracts features such as surgical instruments and organs from endoscopic surgical images through cross-modal image feature processing and generation, and identifies the surgeon's operational intentions within the surgical scene. It then describes the surgical scene in text to achieve a comprehensive interpretation of the surgery. The core image processing and generation method is an encoder-decoder architecture. The encoding stage consists of scene feature extraction, instance encoding, and cross-modal fusion. The decoding stage decodes the fused features and generates a feature sequence, ultimately achieving a text description of the scene understanding using a corpus.

[0036] Specifically, a method for generating text descriptions of cross-modal endoscopic surgical images includes the following steps:

[0037] S1: Scene feature extraction can be achieved using general feature extraction models such as ResNet, VGG19, and ViT. In this invention, a transferable distillation vision Transformer is used as the implementation path to extract global and local detail visual features in endoscopic surgery scenes. Please refer to [link to relevant documentation]. Figure 1 The implementation process is as follows: First, the teacher model loads a pre-trained visual Transformer parameter model on a large-scale natural image dataset. It uses these distillation tokens to capture global contextual information of the endoscopic image. These distillation tokens are then integrated with image patch embedding vectors to form a complete input sequence, which is then transferred to the student model for processing in the distillation visual Transformer. Through multi-layer self-attention mechanisms and feedforward neural network encoding, a scene feature representation of the endoscopic image is generated.

[0038] S2: Instance encoding replaces the predefined query input commonly used in natural image scene understanding. Endoscopic image datasets lack the rich and accurate annotation information found in natural image datasets. Therefore, exploring the representation of different modal features from images is one of the main technical innovations of this invention. This process can be implemented using models such as convolutional neural networks, VGG16, and ResNet50, but this invention uses a visual Transformer model as the implementation path for instance classification (e.g., Figure 2 As shown in the figure, the generated category features are presented in the form of one-hot encoding, which can accurately obtain the main classification information involved in the text description of the endoscopic image scene. This not only improves the accuracy of the subject and object in the description text, but also ensures that the generated text is more consistent with the image content.

[0039] S3: Cross-modal fusion concatenates scene features and instance codes through channel overlay to form fused features. Global features accurately represent the location information of surgical instruments and organs in the image, as well as the interaction between instruments and tissues, providing accurate semantic information for the fused features. Classification features accurately represent the surgical instruments and organs contained in the endoscopic image, providing accurate category information for the fused features. By combining global and classification features, the fused features integrate the category and visual information of the image. Furthermore, to highlight key features and reduce redundant features during feature fusion, the feature fusion module adaptively optimizes the representation of features from different channels through a squeeze-excitation attention mechanism, as shown in the formula:

[0040] ;

[0041] in, Represented as instance encoding features, Represented as global features of the scene. Represented as cross-modal fusion features, each channel of the fusion feature represents a different feature. The squeeze-excitement attention mechanism compresses the information of each feature channel into a single value to form a channel descriptor through global pooling. Subsequently, after transformation by a small fully connected network, importance weights for each channel are generated. These weights are used to calibrate the original fusion feature generation channel enhancement features through channel-wise multiplication operations. This not only strengthens important feature channels but also suppresses repetitive feature channels, enabling the decoder to focus more accurately on the main information and improve the accuracy of feature representation.

[0042] S4: The text description decodes the fused features through a self-attention mechanism, mapping the generation probability of each word to construct a complete descriptive text, such as... Figure 3 As shown, specifically, the decoder receives cross-modal fusion features from the encoder, processes and transforms these features through a multi-layer self-attention mechanism and a feedforward neural network to generate a feature sequence; it matches the feature sequence with a predefined corpus, performs matrix multiplication using the mapping matrix generated from the corpus, and generates a probability distribution of each text position mapped to each word in the vocabulary. To optimize the generation process, the Beam search method is used to select several words with the highest probability at each time step as candidate words to generate new candidate sequences. If a candidate sequence generates an end marker, it is added to the final result list. This process continues until the generated text sequence reaches the preset maximum length, ultimately generating a text description for understanding the endoscopic surgery scene.

[0043] This invention also provides a cross-modal endoscopic surgery image text description generation system, based on the above method, mainly including a scene feature extraction module, an instance encoding module, a cross-modal fusion module, and a text description module, while also incorporating data input and output modules, such as... Figure 4 As shown, the endoscopic surgery scene understanding system constructed collaboratively by the above modules is as follows:

[0044] S1. The system input is an endoscopic surgical image, which undergoes image preprocessing to convert segmentation mask to classification label;

[0045] S2. Extract panoramic visual features such as organ anatomical structure and instrument spatial position through the scene feature extraction module;

[0046] S3. Through the instance coding module, identify the category information of surgical instruments in endoscopic surgical images and convert it into label codes;

[0047] S4. By using the cross-modal fusion module, scene features and instance codes are fused to highlight key feature representations such as the device-tissue interaction area;

[0048] S5. The cross-modal fusion features are decoded through the text description module to match words and optimize word order;

[0049] S6. Generate a complete text description of the surgical image as the system output.

[0050] Example

[0051] Currently, there is no standard dataset specifically for textual descriptions of endoscopic surgical images. In this invention, we re-annotated the Endovis2018 dataset. We reclassified components such as tool handles, clamps, and necks from the original mask into specific surgical instrument types, achieving precise entity annotation of surgical instruments in endoscopic surgical scenes. Based on the triplet annotations of tissues, organs, surgical instruments, and their interactions in each endoscopic surgical scene, we further refined the surgical scene descriptions of the endoscopic images by expanding and perfecting them into descriptive sentences to more accurately describe the scene content of the endoscopic surgical images. These expanded texts contain a maximum of 50 words, ensuring detailed descriptions while avoiding the introduction of redundant information.

[0052] Table 1. Impact of different models on text description generation

[0053]

[0054] To intuitively evaluate the performance of this model, experiments were conducted to compare it with a series of other models and methods within this framework. In terms of metrics, the BLEU-1 score primarily reflects the lexical overlap between the generated and labeled text; the BLEU-4 score measures the fluency and semantic consistency of the text. As shown in Table 1, on the expanded EndoVis2018 dataset, the cross-modal endoscopic surgery scene understanding method and system, using Transformer as the decoder and ViT as the classifier, achieved 83.25% ± 1.55% BLEU-1 and 75.53% ± 2.68% BLEU-4, significantly outperforming other model combinations. Comparison of different decoders shows that using Transformer as the decoder generally outperforms LSTM, indicating its ability to better capture the complex relationship between image features and text generation. The cross-modal endoscopic surgery image text description generation method and system can better handle key information of endoscopic surgery images (such as instrument category and organ name), generate more accurate scene text descriptions, and perform better in generating complex text structures (such as the dynamic relationship of "instrument-operation-organ") and coherent descriptions.

[0055] To visually demonstrate the effectiveness of the model, Figure 5 A comparison of text descriptions from different models shows that the text generated by the invention system is closest in meaning to the labeled text, maintaining consistency and accuracy across diverse image content, thus verifying its advantages in feature fusion and contextual understanding.

[0056] The above experimental results verify the effectiveness of the multimodal method of the present invention in the field of surgical image data processing and generation, highlight the importance of the system of the present invention in practical applications, and help medical professionals better interpret the endoscopic surgical process and provide reliable support in clinical decision-making.

[0057] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for cross-modal endoscopic surgery image text description generation, characterized in that, The method comprises the following steps: S1: taking a migratable distillation visual Transformer as an implementation path to extract global and local detail visual features in an endoscopic surgery scene; S2: taking a visual Transformer model as an implementation path of instance classification, and the generated category features are presented in one-hot encoding form; S3: cross-modal fusion is used to splice the channel superposition of scene features and instance encoding to form fusion features, and the feature fusion module adaptively optimizes the representation of different channel features through a squeeze-excitation attention mechanism, as shown in the formula: S4: the decoder receives the cross-modal fusion features from the encoder, processes and converts the features through a multi-layer self-attention mechanism and a feedforward neural network to generate a feature sequence; and the generated feature sequence is matched with a predefined corpus, a mapping matrix generated by the corpus is used for matrix multiplication operation to generate a probability distribution of each text position mapping to each word in the vocabulary. ； wherein, are represented as instance encoding features, are represented as scene global features, are represented as cross-modal fusion features, each channel of the fusion features represents a different feature, the squeeze-and-excitation attention mechanism compresses the information of each feature channel into a single value through a global pooling operation to form a channel descriptor, and then, after conversion through a small fully connected network, generates the importance weight of each channel; The specific method of the step S1 is that a teacher model loads a visual Transformer parameter model pre-trained on a large-scale natural image dataset, captures the global context information of the endoscopic image by using the distillation tokens, integrates the distillation tokens and image block embedding vectors to form a complete input sequence, migrates to a student model of the distillation visual Transformer for processing, and generates scene feature representation of the endoscopic image through encoding of a multi-layer self-attention mechanism and a feedforward neural network.

2. The cross-modality endoscopic surgery image text description generation method of claim 1, wherein, In the step S3, the global feature is used to represent the position information of surgical instruments and tissue organs in the image and the interaction between the instruments and the tissue, and provides accurate visual semantic information for the fusion feature; the category feature is used to represent the surgical instruments and tissue organs contained in the endoscopic image, and provides accurate category information for the fusion feature; and through the combination of the scene feature and the category feature, the fusion feature integrates the category information and the visual information of the image.

3. The cross-modality endoscopic surgery image text description generation method of claim 1, wherein, In the step S4, a Beam search method is used to select a plurality of words with the highest probability at each time step as candidate words to generate a new candidate sequence, if an end marker is generated in a certain candidate sequence, the candidate sequence is added to a final result list, and the process continues until the generated text sequence reaches a preset maximum length, and finally a text description of the endoscopic surgery scene understanding is generated.

4. The cross-modality endoscopic surgery image text description generation method of claim 1, wherein, The system comprises:

5. A cross-modality endoscopic surgery image text description generation system, used to implement the cross-modality endoscopic surgery image text description generation method of any one of claims 1-4, characterized in that: a scene feature extraction module: image preprocessing for segmentation mask to classification label conversion, and extraction of global visual features of organ anatomical structure and instrument spatial position through the scene feature extraction module; an instance encoding module: used for identifying the category information of surgical instruments in the endoscopic surgery image and converting the category information into label encoding; a cross-modal fusion module: used for fusing the scene feature and the instance encoding to highlight the key feature representation of the instrument-tissue interaction area; a text description module: used for decoding the cross-modal fusion feature, matching the vocabulary and optimizing the word order; a data input and output module: the system input is an endoscopic surgery image; and the output module is used for generating a complete surgical image text description as the output of the system. ​

Images