A method for generating and evaluating long text descriptions from laparoscopic surgery images

Abstract

The application discloses a generation and evaluation method for obtaining long text description from a laparoscopic surgery image, comprising the following steps: constructing a prompt word containing a laparoscopic surgery image, a short text description, a target boundary box and surgery knowledge, and constructing long text description data by using a large language model based on the prompt word; constructing an image long text description generation model, which comprises a pre-trained visual encoder, a visual feature query transformer, a language decoder and an external surgery concept vector database; performing supervised training on the image long text description generation model by using the long text description data; constructing a long text description evaluation method to evaluate the trained model; obtaining an input laparoscopic image video stream through a laparoscopic image system, and screening key frames as to-be-processed images; inputting the to-be-processed images into the trained and evaluated model to obtain an image long text description result. The application can enhance the understanding ability of surgery details in the image and generate a more comprehensive surgery image description.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and medical image processing, and in particular to a method for generating and evaluating long text descriptions obtained from laparoscopic surgical images. Background Art

[0002] Providing high-quality surgical support is a challenge in medicine, particularly in areas with limited medical resources. With technological advances, remote surgical guidance is being explored and implemented. In particular, leveraging artificial intelligence (AI) to generate natural language descriptions of surgical images and provide real-time guidance and support to surgeons has emerged as a promising solution. This technology not only reduces the workload of physicians by automatically generating descriptions of surgical steps through video analysis, but also provides essential guidance in the absence of on-site experts, such as automatically generating surgical reports and intraoperative precautions. Furthermore, the development of this technology can help standardize surgical procedures, improve surgical quality and safety, and provide new tools for surgical education and training. Despite data heterogeneity and technical challenges, surgical image description generation is expected to effectively improve the current state of the surgical field through continued research and innovation.

[0003] The field of surgical image description generation currently faces several important challenges. First, the migration from natural images to surgical images is complex, as surgical images often involve specific medical knowledge and complex scenarios, making it difficult to adapt general image processing techniques to surgical image analysis. Second, the annotation of surgical images is not only technically demanding but also costly, primarily because professional medical knowledge is required to accurately annotate the details in the images. Finally, due to the diversity of surgical types, execution methods, equipment, and patient conditions, there is a high degree of heterogeneity between different datasets, which increases the difficulty of developing universal algorithmic models. These factors collectively lead to significant challenges in the research, development, and application of surgical image description technology.

[0004] For the task of generating descriptions for surgical images, existing research solutions generally adopt the general architecture of image encoder-language decoder. The following are some of the currently adopted model architectures:

[0005] 1. Image encoder. The image encoder in surgical image description generation is primarily responsible for extracting useful visual features from the image. Different encoder models have different processing methods and advantages for the characteristics and complexity of images: the Inception model, through the design of multi-scale convolution kernels, can capture multi-scale information of the image within the same layer, making it particularly suitable for processing the rich local details in medical images; ResNet introduces residual connections to solve the gradient vanishing problem, enabling the network to learn more complex surgical scene features; ViT comprehensively considers the relationship between various parts of the image through the self-attention mechanism, better understands the global context, and effectively analyzes the interaction between tools and tissues during surgery; VideoSwin Transformer processes temporal information through hierarchical shifted windows, which can effectively capture dynamic changes in surgical videos and enhance understanding of the surgical process.

[0006] 2. Text decoder. The LSTM model can process long-term dependencies in surgical descriptions and generate coherent and logical descriptions through a gating mechanism. The GRU structure is concise and efficient when processing short sequences, saving resources. The Transformer decoder uses a self-attention mechanism to comprehensively consider the interrelationships between various operations during the surgical process, generating detailed and context-rich text. Its parallel processing capabilities make it suitable for real-time tasks.

[0007] In summary, while traditional CNN encoders such as Inception and ResNet are capable of extracting important local features when processing surgical images, they are relatively limited in resolution and detail granularity, making it difficult to effectively capture the image's global information. This is particularly evident in laparoscopic surgery, which involves numerous small instruments and intricate surgical details that are crucial to its success. In contrast, the Transformer encoder, through its self-attention mechanism, is better able to understand and process these complex global relationships, providing a more comprehensive interpretation of visual features.

[0008] Regarding text decoders, while RNN models such as LSTM and GRU perform well in certain situations, they often struggle to process longer contexts, limiting their ability to generate detailed and coherent surgical descriptions. Furthermore, the sequential nature of RNNs can lead to decreased efficiency when processing long sequences. In contrast, the Transformer decoder, with its self-attention mechanism, not only captures long-range dependencies but also significantly improves processing speed through parallel processing, making it more suitable for complex and information-intensive surgical description tasks.

[0009] Despite the many advantages of the Transformer model, its large number of model parameters and the high GPU resources required for training, which can be a major challenge in resource-limited research environments. Furthermore, large pre-trained models are at risk of overfitting, negative transfer, or catastrophic forgetting when fine-tuned on small datasets, especially in highly specialized and detail-rich applications such as surgical image captioning. Therefore, choosing an appropriate model architecture and adjusting training strategies to avoid these issues are key to achieving effective surgical image captioning.

[0010] In summary, existing description generation methods for laparoscopic surgical images have the following challenges and problems:

[0011] 1. Limitations of short text descriptions in public datasets: Image descriptions in existing datasets are usually relatively brief, and their content is often equivalent to surgical triplet (instrument, anatomical location, action) extraction tasks, such as "monopolar curved scissors are cutting the renal parenchyma". This description not only lacks sufficient details, but is actually difficult to surpass the multi-classification triplet extraction task in terms of accuracy, which makes it difficult for the trained model to accurately reflect the complex situations and nuances in the operation. Therefore, in order to improve the practical application value and accuracy of surgical image description generation technology, there is an urgent need to expand and enrich the dataset of surgical images, provide richer and more detailed descriptions, such as the location information of instruments and tissues and the interactions between them, and further, describe the significance and precautions of some surgical operations. Such improvements will help develop more accurate and practical automatic description generation systems, thereby better supporting the teaching and practice of clinical surgery.

[0012] 2. Lack of surgical concepts and hallucination problems in pre-trained models: Existing pre-trained models are usually trained based on large-scale natural language datasets. Although they perform well in natural language generation tasks, due to the lack of expertise in the surgical field, these models often have cognitive defects when processing medical image descriptions, which in turn leads to hallucination problems. Specifically, when generating descriptions of surgical images, pre-trained models may generate text that does not match the image content or actual operations, or even generate non-existent instruments or operating steps. This phenomenon is called "hallucination" and is mainly due to the model's lack of knowledge in specific fields, especially the lack of understanding of instruments, anatomical structures, and operating procedures in surgical scenarios.

[0013] 3. Limitations of Evaluation Methods for Long Text Descriptions: As the surgical image description task expands to include long text, traditional n-gram-based evaluation metrics such as BLEU and ROUGE have gradually demonstrated their limitations. These metrics primarily rely on superficial matching between the predicted text and the reference text, and are more effective for short texts with relatively fixed sentence structures. However, in complex surgical scenarios, the generated descriptions often involve multiple layers of information, such as the purpose of the surgical procedure, procedural details, and precautions. Their length and complexity far exceed traditional description paradigms. This makes simple n-gram matching incapable of accurately assessing the actual semantic accuracy of the generated text. For example, when describing a surgical procedure, there may be multiple reasonable ways to express it. Traditional evaluation metrics may focus too much on matching words or phrases, while ignoring semantic consistency. Therefore, there is an urgent need to introduce new evaluation methods that can effectively measure the semantic distance between generated and ground-truth text to ensure that the model not only generates superficially similar text but also accurately captures the deeper semantics of the surgical scene. Summary of the Invention

[0014] The present invention provides a method for generating and evaluating long text descriptions from laparoscopic surgical images, which can enhance the ability to understand surgical details in the images and generate more comprehensive surgical image descriptions.

[0015] In response to Problem 1 in the background technology, the present invention proposes a method for constructing prompt words for a long text description dataset. In this method, prompt words suitable for long text description data of surgical images are constructed by structuredly organizing information such as the bounding box coordinates of surgical images, instruments and tissues, and surgical-related knowledge. Multimodal large models such as GPT-4-Vision are then used to generate more detailed and accurate long text descriptions for surgical images. By combining structured information from images and surgical scenes, the model's ability to understand surgical details is enhanced, thereby generating detailed descriptions that conform to actual operations, significantly improving the richness and effectiveness of image descriptions.

[0016] In response to problem 2 in the background technology, the present invention proposes a long text description model structure enhanced by external retrieval. In order to solve the problem of hallucinations caused by the lack of surgical concept cognition in the pre-trained model, the present invention introduces an external surgical concept vector database. During the model training process, combined with the visual features of the surgical images, relevant surgical concepts are retrieved from the vector database, and these surgical concepts are integrated into the model through the cross-attention mechanism. This method not only improves the model's ability to understand the instruments, anatomical structures and operations in the surgical scene, but also effectively reduces the occurrence of hallucinations, ensuring that the generated descriptions have higher accuracy and reliability in the medical field.

[0017] In response to problem 3 in the background technology, the present invention proposes a series of evaluation methods based on the characteristics of long text descriptions to solve the problem that traditional n-gram indicators such as BLEU and ROUGE cannot accurately evaluate the quality of long text generation. The specific methods include fine-tuning BERTScore to evaluate the word vector similarity of the text; designing specific prompt words for the long text description of surgical images, and using a large language model to perform semantic evaluation of image text descriptions (CLAIR); through long text relationship extraction technology, extracting triple information in the description and matching it with the triples in the real description to measure the accuracy of the generated text (Long-SPICE). This method can more comprehensively evaluate the semantic consistency of surgical image descriptions and the correctness of surgical knowledge, ensuring that the generated text has a high-fidelity description of the surgical scene.

[0018] Specifically, a method for generating and evaluating long text descriptions from laparoscopic surgical images includes:

[0019] We obtain laparoscopic surgical images from a public training set and construct prompt words containing the laparoscopic surgical images, short text descriptions, target bounding boxes, and surgical knowledge. Based on these prompt words, we use a large language model to automatically construct long text description data.

[0020] Build a model for generating long text descriptions of images, which includes a pre-trained visual encoder, a visual feature query transformer, a language decoder, and an external surgical concept vector database;

[0021] Use the constructed long text description data to perform supervised training on the image long text description generation model to obtain a trained image long text description generation model;

[0022] Build a long text description evaluation method to evaluate the trained model;

[0023] The laparoscopic image system obtains the input laparoscopic image video stream, and filters the key frames as the images to be processed; the images to be processed are input into the trained and evaluated model to obtain the long text description results of the images.

[0024] Furthermore, the visual encoder utilizes the CLIP model, which has been pre-trained on a large-scale natural image-text pairing dataset to extract key visual features from laparoscopic surgical images. During the training phase of the model for generating long text descriptions of images, the CLIP weights remain frozen, leveraging the pre-trained model's advantages in visual feature extraction, reducing the computational cost of training and avoiding overfitting on small-scale surgical datasets. This frozen visual encoder also ensures the model's generalization across diverse surgical images, maintaining consistent performance across a variety of laparoscopic surgical scenarios.

[0025] Furthermore, the visual feature query transformer adopts the Q-Former in BLIP2; the Q-Former is responsible for screening the visual features extracted from the visual encoder, identifying important visual features that appear during the surgery, such as surgical instruments and anatomical structures, and passing these important visual features to the language decoder so that the language decoder generates descriptive text that matches the image content.

[0026] Furthermore, the language decoder uses vicuna-13B, which has been pre-trained on large-scale natural language data and is used to generate long text descriptions. The weights of vicuna-13B remain frozen, and only the Q-Former component of BLIP2 is trained to adapt to the text generation needs of surgical scenes. The visual features extracted by the Q-Former from the visual encoder interact with the text description generation process of the language decoder to ensure that the generated descriptions accurately reflect the details of the surgical scene. This approach not only reduces training costs but also avoids the problem of overfitting the model on small-scale medical datasets, while ensuring the coherence of the generated text and the accuracy of surgical knowledge.

[0027] Furthermore, the external surgical concept vector database includes two steps: construction and use;

[0028] Before training, a pre-trained visual feature query transformer is used to extract image vectors and corresponding surgical concepts, constructing a vector database. During the training of the image long text description generation model, visual features are retrieved from the vector database to obtain surgical concepts related to the surgical scenario. These surgical concepts are fused with the model's visual features in the form of learnable vectors through a cross-attention mechanism. The visual features and the feature vectors of the surgical concepts are combined to guide the language decoder to generate accurate and detailed long text descriptions of the surgery. The introduction of surgical concepts not only improves the model's performance in medical scenarios but also effectively reduces the generation of hallucinations that do not conform to the actual surgical situation.

[0029] Furthermore, the training process of the image long text description generation model is divided into a pre-training stage and a downstream task fine-tuning stage;

[0030] During the pre-training phase, the loss functions include image-text matching loss, image-text contrastive learning loss, and text generation loss. The image-text matching loss is used to ensure that the image features extracted by the visual encoder and the text description generated by the language decoder are globally matched. The image-text contrastive learning loss further optimizes the alignment of visual and text features through contrastive learning, improving the model's cross-modal understanding capabilities. The text generation loss is used to supervise the language decoder in generating text descriptions that are highly relevant to the image content, ensuring the accuracy and coherence of the generated text.

[0031] During the fine-tuning phase for downstream tasks, only the Q-Former is fine-tuned, with no adjustments to the language decoder. Training at this stage focuses on generating textual descriptions of the surgery from visual features. A unified cross-entropy loss for text generation is used to constrain the visual features extracted by the Q-Former, ensuring that they accurately convey details of the surgical scene, such as surgical instruments, anatomical structures, and their interrelationships. This approach not only preserves the advantages of the pre-trained large model in handling multimodal tasks, but also allows Q-Former to be fine-tuned to accommodate the specific requirements of laparoscopic surgical image description tasks, thereby improving the model's practical application.

[0032] Furthermore, the image-text matching loss is calculated as follows:

[0033]

[0034] in, is the matching probability predicted by the model, y is the actual label, 1 indicates that the image and text match, and 0 indicates no match.

[0035] Furthermore, the image-text contrastive learning loss is calculated as follows:

[0036]

[0037] Among them, I i Represents the feature vector of the i-th picture, T i Represents the feature vector of the i-th paragraph of text, sim(I i ,T i ) represents the similarity between the image and the text, and τ is the temperature hyperparameter.

[0038] Furthermore, the calculation formula of text generation loss is as follows:

[0039]

[0040] Among them, P(y t |x 1:t-1 ) means that given the previous t-1 words, the model predicts the correct word y t probability.

[0041] Furthermore, the long text description evaluation method includes BERTScore, CLAIR and Long-SPICE evaluation methods.

[0042] BERTScore, as a semantic matching evaluation method based on a pre-trained language model, can better capture the semantic similarity between texts than traditional indicators such as BLEU, ROUGE and CIDEr. Traditional BLEU, ROUGE and CIDEr indicators rely on surface matching of n-grams and are suitable for shorter, fixed-structure descriptive texts, but they show obvious limitations for complex surgical scenes or long text descriptions. For example, BLEU and ROUGE have difficulty handling synonyms and changes in expressions when calculating vocabulary or phrase overlap, while CIDEr mainly targets word frequency matching of short texts in specific fields and cannot fully capture more complex semantic information in long texts. BERTScore, by calculating the semantic similarity of sentence embedding vectors, can more effectively measure the accuracy of generated text and reference text at the semantic level, and is particularly suitable for image description tasks in long text scenarios. Furthermore, the present invention increases the score weight of specific word units in surgical scenes based on the word segmentation of BERT Tokenizer, thereby enhancing the accuracy of this indicator in evaluating long text descriptions of surgical images.

[0043] Secondly, CLAIR is based on evaluation cues specifically designed for surgical images. These cues help large language models effectively focus on surgical information within descriptions. It not only handles surface text matching but also scores by understanding the deeper semantics of both the reference and generated text, effectively assessing the plausibility of the text and the completeness of the description. This approach overcomes the inability of traditional n-gram matching-based metrics to process semantic information. Especially for long text description tasks, CLAIR can more comprehensively assess the quality of image descriptions, ensuring that the text not only matches the image content but also exhibits coherence and deep logic.

[0044] Finally, in response to the special needs of long text descriptions, the present invention introduces the Long-SPICE indicator for complex scenes. SPICE refers to the use of scene graph extraction technology to decompose the generated text into a set of semantic relationship graphs, and use these relationship graphs to evaluate the accuracy of the text description. However, the traditional SPICE method is limited by the extraction technology and is only applicable to the scene graph extraction of a single sentence. It is difficult to cope with the situation of multiple sentences or complex scenes in long texts. Therefore, the present invention further utilizes a large language model for relationship extraction, extracts multiple semantic relationships in long texts, and matches them with the actual content such as instruments and anatomical structures extracted from the reference description for evaluation. This extension enables SPICE to effectively process long text descriptions and improve its evaluation accuracy of key information in complex descriptions.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention significantly improves the accuracy and richness of surgical image description datasets by constructing prompt words tailored to surgical scenarios and fully integrating structured information about surgical scenarios. Compared to traditional short text description datasets, this invention generates more detailed long text descriptions by organizing images, short text descriptions, annotation boxes, and specific surgical knowledge. This enhances the model's ability to understand surgical details and provides surgeons with more comprehensive surgical image descriptions.

[0047] 2. This invention enhances the model's understanding of key surgical concepts by incorporating an external surgical concept vector database and utilizing a cross-attention mechanism within the query transformer, thereby reducing the occurrence of hallucinations. This design improves the professionalism and accuracy of the generated text, ensuring that the description effectively reflects the actual surgical instruments and anatomical locations present during the procedure, as well as their locations and interactions, enhancing the model's practicality in complex surgical scenarios.

[0048] 3. This paper addresses the problem that traditional n-gram metrics cannot accurately assess the quality of long text descriptions. By proposing a novel long text description evaluation method, this paper not only focuses on surface text matching but also conducts in-depth semantic evaluation through the semantic understanding capabilities of a large language model. This ensures that the generated descriptions not only accurately reflect surgical instruments and anatomical structures, but also maintain semantic consistency and correct medical knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart for constructing long text description data for surgical images according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of constructing prompt words in Chinese for a long text description according to an embodiment of the present invention;

[0051] Figure 3 This is a network structure diagram of a pre-trained query transformer according to an embodiment of the present invention;

[0052] Figure 4 This is a network structure diagram of a model for generating long text descriptions of images according to an embodiment of the present invention;

[0053] Figure 5 Generate an effect diagram for the long text description of an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the Chinese evaluation prompt words based on CLAIR according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of extracting Chinese prompt words from a scene graph based on a large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0057] A method for generating and evaluating long text descriptions from laparoscopic surgical images mainly consists of the following three parts:

[0058] The construction of long text description prompt words relies on a public image-text dataset. The images, short text descriptions, segmentation and annotation bounding boxes and other information are combined with surgery-specific knowledge to construct structured prompt words, and a long text description is generated through a multimodal large model.

[0059] The long text description model structure design, based on the characteristics of the pre-trained multimodal large model, introduces an external surgical concept knowledge base and a cross-attention mechanism, and uses the long text description dataset constructed above for training.

[0060] The design of a long text description evaluation method overcomes the shortcomings of traditional evaluation indicators in the evaluation of long text descriptions. Evaluation methods such as BERTScore, CLAIR, and Long-SPICE are proposed, and the trained models are evaluated.

[0061] To construct description prompt words, we used the image and segmentation annotations from the 2018 MICCAI segmentation competition dataset, as well as the short text annotations used in subsequent work, to jointly construct prompt words and input them into the GPT-4-Vision model for long text annotation. We manually reviewed the possible errors or hallucinations in the annotation results, summarized them as surgical knowledge, adjusted the prompt words again, and re-input them into the large model for description generation. Figure 1 As shown, specifically including:

[0062] Step 1: Obtain laparoscopic surgery images and corresponding short text descriptions from the 2018 MICCAI competition public dataset.

[0063] Step 2: Label the segmentation mask labels of 12 instruments and 4 anatomical positions and parse them into label and bounding box coordinates (upper left corner x coordinate, upper left corner y coordinate, lower right corner x coordinate, lower right corner y coordinate) data.

[0064] Step 3: Design prompt words to organize surgical images, short text descriptions, bounding box information, and surgical knowledge information, such as Figure 2 shown.

[0065] Step 4: Call the GPT-4-Vision API, enter the prompt, and get the long text description data.

[0066] Step 5: Manually review the errors and hallucinations in the long text description and further supplement the surgery-related knowledge in the prompt based on the corresponding information.

[0067] Using the above-constructed prompt words, repeat steps 3-5 until the ideal long text description dataset is obtained.

[0068] like Figure 4 As shown in the figure, the long text description model structure mainly consists of CLIP visual encoder, vicuna-13B language decoder, Q-Former visual feature query transformer, and external surgical concept vector database.

[0069] The construction and training of the long text description model mainly consists of three steps: pre-training the query transformer, constructing the surgical vector database, and fine-tuning the surgical concept vector database.

[0070] Step 1: If Figure 3 As shown in the figure, a pre-trained multimodal large model base is obtained based on the work of BLIP2. First, the input image is divided into multiple patches, and the corresponding visual features are extracted through a pre-trained visual encoder (Vision Transformer, ViT). These features encode local information in the image and represent the semantic content of the image in the form of a fixed-length vector. Then, learnable query vectors are introduced. These query vectors are randomly initialized at the beginning and continuously optimized as training progresses. During training, the query vector and the text embedding are spliced ​​and self-attention is calculated. The mask is set differently according to the subsequent pre-training objectives.

[0071] Specifically, the pre-training process includes three pre-training objectives: image-text matching (ITM), image-text contrastive learning (ITC), and text generation tasks.

[0072] The image-text matching task uses a bidirectional self-attention mask, that is, the query vector and the text vector are mutually visible. This is because the image-text matching task is a binary classification task, which requires combining the mutual information of the query vector and the text vector to determine whether the image and text match. The self-attention result continues to enter the cross-attention module, with the concatenation of the query vector and the text vector as the query, and the image vector as the key and value, and then continuously introduces the image information into the query vector. The result of the cross-attention enters the feedforward layer. This part is as follows Figure 3 The left path of the "Query Transformer" is shown, where N layers are repeatedly stacked, and the final output is connected to the classification head. The loss of this task can be expressed as the classification cross entropy loss, which is calculated as follows:

[0073]

[0074] in, is the matching probability predicted by the model, and y is the actual label, where 1 indicates a match between the image and text, and 0 indicates a mismatch. The goal of ITM is to minimize the cross-entropy loss between image and text matching. This loss ensures that the image features extracted by the visual encoder are globally consistent with the text description.

[0075] The image-text matching task uses a one-way self-attention mask, that is, the query vector and the text vector are invisible to each other. This is because in contrastive learning, it is necessary to prevent the query vector from learning directly from the text, causing information leakage. After self-attention, the query vector continues to enter the cross-attention module, with the query vector as the query and the image vector as the key and value, and then continuously introduces the image information into the query vector. On the other hand, the text vector does not need to enter the cross-attention module after self-attention. Finally, the query vector and the text vector enter the feedforward layer with different weights respectively. This part is as follows Figure 3 The two paths of the “Query Transformer” are shown in Figure 1. N layers are repeatedly stacked. The results of the two vectors are optimized using contrastive learning. The loss can be expressed as InfoNCE (Noise Contrastive

[0076] Estimation), the calculation formula is as follows:

[0077]

[0078] Among them, I i Represents the feature vector of the i-th picture, T i Represents the feature vector of the i-th paragraph of text, sim(I i ,T i ) represents the similarity between the image and text (usually using cosine similarity), and τ is a temperature hyperparameter. Through this contrastive learning, the model can further improve cross-modal understanding and align visual and textual features.

[0079] The text generation task uses a multimodal causal self-attention mask, that is, the query vector is visible to the text vector, and the text vector itself needs to mask the token after the current position; and the text vector is invisible to the query vector. This is because in the text generation task, the prediction of the next word needs to depend on the query vector and the results of the word before the current word. After self-attention, the query vector continues to enter the cross-attention module, with the query vector as the query and the image vector as the key and value, and then continuously introduces the image information into the query vector. On the other hand, the text vector of the Nth layer can obtain the image information introduced by the cross-attention of the N-1th layer in the self-attention. Finally, the text vector enters the feedforward layer. This part is as follows Figure 3The right path of the "Query Transformer" is shown, with N layers repeatedly stacked. The loss function is the standard cross entropy loss, calculated as follows:

[0080]

[0081] Among them, P(y t |x 1:t-1 ) means that given the previous t-1 words, the model predicts the correct word y t This loss is used to supervise the language model to generate descriptions related to the image content, ensuring that the generated text is coherent and relevant to the image content.

[0082] Step 2: Construction of surgical concept vector database requires the use of pre-trained BLIP-2 model to perform feature embedding extraction on the input surgical images. Figure 3 The "fine-tuning phase" on the right shows each module using weights obtained during the pre-training phase. First, the input image is divided into multiple patches, and corresponding visual features are extracted using the same visual encoder used in the pre-training phase. These features encode local information in the image and represent the image's semantic content as a fixed-length vector. The query vector optimized during the pre-training phase is then introduced. After stacking N layers of self-attention, cross-attention, and feed-forward modules, the visual vector embedding of the image is obtained.

[0083] For all images in the training set, the corresponding visual vector embeddings are extracted and their instrument category information is stored. Because surgical images may contain multiple instrument categories, the model requires multi-category annotation of the embeddings for each image. These image vector-surgical concept pairs are stored in a .pkl file.

[0084] Step 3: Fine-tuning. During training, the .pkl file obtained in Step 2 is first loaded into the Faiss vector database. For each input image, the pre-trained query transformer is used to obtain an image vector embedding, as in Step 2. Vector similarity is calculated to retrieve the top-k images most similar to the current image from the stored embedding vector set. The specific similarity calculation can be expressed as follows:

[0085]

[0086] Among them, v i Represents the embedding vector of the current input image, v j Represents the embedded vector of the stored image, sim(v i ,v j) represents the cosine similarity between the two. This similarity calculation yields the category information for the top-k most similar images. Because these images may have overlapping categories, the present invention employs a voting mechanism to extract categories whose occurrences exceed a set threshold. A threshold, T, is set; if a category's occurrences exceed T, it is selected as the category description for the current image.

[0087] This category information is used as text input and enters the attention mechanism together with the image information. First, a text query vector is defined and concatenated with the query vector extracted from the image embedding to form an image-text query. This query captures the information of the current image through the self-attention mechanism. The calculation formula of the self-attention mechanism is as follows:

[0088]

[0089] Among them, Q is the query vector, K is the key vector, and V is the value vector. is the scaling factor. In this step, the image query is combined with the self-attention mechanism to extract key information related to the current image. Next, the text query vector is used as the query vector (Query), and the text category embedding vector extracted in the previous step is used as the key-value vector (Key-Value). The two are fused through the cross-attention mechanism. The calculation formula of the cross-attention mechanism is as follows:

[0090]

[0091] Among them, Q text-former is the text query vector, K text ,V text Embed the text vector. Through the above steps, the text query captures the image information, and the image query captures the semantic information of the text category. Finally, the two query vectors are connected and linearly projected. The formula is as follows:

[0092] h=W[q img-forme ;q text-former ]+b

[0093] Among them, q img-forme is the image query vector, q text-former is the text query vector, [;] represents vector concatenation, W is the linear projection matrix, and b is the bias term. The projected result is input into the frozen large language model to generate the final text description. The training process also uses the above-mentioned cross entropy loss function. This method combines image and text information to ensure that the generated description can accurately reflect the surgical instruments and their operation scenes, effectively improving the text generation quality of laparoscopic surgical images. The generated results are as follows: Figure 5 shown.

[0094] The long text description evaluation method includes three evaluation indicators:

[0095] Indicator 1: BERTScore, an evaluation metric based on the pre-trained BERT model. BERTScore is an evaluation method based on semantic similarity. By using the pre-trained BERT model, each word in the generated text and the reference text is embedded in a high-dimensional space and the semantic similarity between them is calculated. The specific calculation formula is as follows:

[0096]

[0097] Here, X and Y represent the word embeddings of the generated and reference texts, respectively, and sim(x,y) is the cosine similarity between the word embeddings. Based on the word segmentation results of the BERT tokenizer, we further weight specific keywords in surgical scenarios N times as much as other tokens to enhance the BERTScore's performance in evaluating surgical image descriptions. The formula for weighted BERTScore is as follows:

[0098]

[0099] Among them, w(x) represents the weight of each word, ∑ x∈Z w(x) is a normalization factor that ensures comparability of total scores under different weight distributions. Compared to traditional n-gram matching-based evaluation methods such as BLEU and ROUGE, BERTScore can better capture the semantic similarity of text and is particularly suitable for evaluating long texts in complex surgical scenarios.

[0100] Indicator 2: CLAIR, an image description evaluation indicator based on a large language model. Its working principle is to construct specific prompt words for surgical image descriptions, and use the large language model to understand the deep semantic relationship between the predicted text and the labeled text, especially focusing on key information such as instruments, anatomical locations, operations, and surgical concepts, while ignoring differences in word order or expression. The prompt word template is as follows: Figure 6 In addition to providing a score, the large language model can also provide the basis for the score, further enhancing the interpretability of the evaluation results.

[0101] Indicator 3: SPICE (Semantic Propositional Image Caption Evaluation) is a semantic scene graph image description evaluation indicator for long texts.

[0102]

[0103] Where N is the total number of pictures, The semantic parsing graph representing the generated description of the i-th image, The semantic parsing graph representing the reference description of the i-th image. F1 is the F1-score based on the generated semantic graph and the reference semantic graph, and is calculated as:

[0104]

[0105] Among them, Precision represents the accuracy between the semantic elements of the generated description and the semantic elements of the reference description, and Recall represents the recall rate between the semantic elements of the generated description and the semantic elements of the reference description. The traditional SPICE method is only applicable to single-sentence scene descriptions and cannot handle multiple sentences or complex scenes in long texts. The present invention improves this by further extracting multiple semantic relationships in long texts through a large language model, and matching them with key elements in the surgical scene (such as instruments, anatomical structures, etc.), thereby improving the accuracy of the evaluation of complex surgical scenes. The prompt word template for relationship extraction is as follows: Figure 7 shown.

[0106] Table 1 compares the performance of different methods in the long text description task for laparoscopic surgical images, primarily evaluating the performance of BERTScore, CLAIR, and Long-SPICE. Overall, the multimodal large model combined with vector retrieval significantly improves the quality of generated text by incorporating external surgical knowledge, particularly in describing surgical image details and maintaining semantic consistency.

[0107] Table 1 Comparison of image long text description results

[0108]

[0109] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating and evaluating long text descriptions from laparoscopic surgical images, characterized in that: include: We obtain laparoscopic surgical images from a public training set and construct prompt words containing the laparoscopic surgical images, short text descriptions, target bounding boxes, and surgical knowledge. Based on these prompt words, we use a large language model to automatically construct long text description data. Build a model for generating long text descriptions of images, which includes a pre-trained visual encoder, a visual feature query transformer, a language decoder, and an external surgical concept vector database; The visual encoder uses the CLIP model, which is pre-trained on a large-scale natural image-text pairing dataset to extract key visual features from laparoscopic surgical images. During the training phase of the model for generating long text descriptions of images, the weights of CLIP remain frozen. The visual feature query transformer uses the Q-Former in BLIP2. The Q-Former is responsible for filtering the visual features extracted from the visual encoder, extracting important visual features that appear during the surgery, and passing these important visual features to the language decoder. The language decoder uses vicuna-13B, which has been pre-trained on large-scale natural language data and is used to generate long text descriptions. The weights of vicuna-13B remain frozen, and only the Q-Former part of BLIP2 is trained to adapt to the text generation needs of surgical scenarios. The visual features extracted by Q-Former from the visual encoder interact with the text description generation process of the language decoder. The external surgical concept vector database consists of two steps: construction and use. Before training, a pre-trained visual feature query transformer is used to extract image vectors and corresponding surgical concepts to construct the vector database. When training the image long text description generation model, visual features are retrieved from the vector database to obtain surgical concepts related to the surgical scene. These surgical concepts are fused with the model's visual features in the form of learnable vectors through a cross-attention mechanism. The visual features and the feature vectors of the surgical concepts are combined to guide the language decoder to generate accurate and detailed long text descriptions of the surgery. Use the constructed long text description data to perform supervised training on the image long text description generation model to obtain a trained image long text description generation model; Build a long text description evaluation method to evaluate the trained model; The laparoscopic image system obtains the input laparoscopic image video stream, and filters the key frames as the images to be processed; the images to be processed are input into the trained and evaluated model to obtain the long text description results of the images.

2. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 1, characterized in that: The training process of the image long text description generation model is divided into a pre-training phase and a downstream task fine-tuning phase; During the pre-training phase, the loss functions include image-text matching loss, image-text contrastive learning loss, and text generation loss. The image-text matching loss is used to ensure that the image features extracted by the visual encoder and the text description generated by the language decoder are globally matched. The image-text contrastive learning loss further optimizes the alignment of visual and text features through contrastive learning, improving the model's cross-modal understanding capabilities. The text generation loss is used to supervise the language decoder in generating text descriptions that are highly relevant to the image content, ensuring the accuracy and coherence of the generated text. During the downstream task fine-tuning stage, only the Q-Former part is fine-tuned, and the language decoder is not adjusted. At this time, the training process focuses on generating text descriptions of the surgery through visual features, and a unified text generation cross-entropy loss is used to constrain it, ensuring that the visual features extracted by Q-Former accurately convey the details of the surgical scene.

3. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 2, characterized in that: The formula for calculating text generation loss is as follows: Among them, P(y t |x 1:t-1 ) means that given the previous t-1 words, the model predicts the correct word y t probability.

4. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 1, characterized in that: Long text description evaluation methods include BERTScore, CLAIR, and Long-SPICE evaluation methods.

Images