Generation and evaluation method for obtaining long text description from laparoscopic surgery image

Abstract

The invention discloses a generation and evaluation method for acquiring long text description from a laparoscopic surgery image, which comprises the following steps of: constructing a cue word containing the laparoscopic surgery image, short text description, a target bounding box and surgery knowledge, and constructing long text description data by utilizing a large language model based on the cue word; constructing an image long text description generation model which comprises a pre-trained visual encoder, a visual feature query converter, a language decoder and an external operation concept vector database; performing supervised training on the image long text description generation model by using the long text description data; constructing a long text description evaluation method to evaluate the trained model; the method comprises the following steps: acquiring an input laparoscope image video stream through a laparoscope image system, and screening a key frame as a to-be-processed image; and inputting a to-be-processed image into the trained and evaluated model to obtain an image long text description result. According to the method, the ability of understanding operation details in the image can be enhanced, and more comprehensive operation image description can be generated.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and medical image processing, and in particular to a method for generating and evaluating long text descriptions obtained from laparoscopic surgical images. Background Art

[0002] In the medical field, there are challenges in providing high-quality surgical support, especially in areas with limited medical resources. With the advancement of technology, methods for performing surgery through remote guidance are gradually being explored and implemented. In particular, using artificial intelligence technology to generate natural language descriptions of surgical images to provide real-time guidance and support to surgeons has become a promising solution. This technology can not only reduce the workload of doctors by automatically generating descriptions of surgical steps through video analysis, but also provide necessary guidance in the absence of on-site experts, such as automatically generating surgical reports and intraoperative precautions. In addition, the development of this technology can also help standardize surgical processes and improve surgical quality and safety, while also providing new tools for surgical education and training. Despite data heterogeneity and technical challenges, surgical image description generation is expected to effectively improve the status quo in the field of surgery through continued research and innovation.

[0003] The field of surgical image description generation currently faces several important issues. First, the migration from natural images to surgical images is complex because surgical images often involve specific medical knowledge and complex scenes, which makes it difficult to adapt from general image processing techniques to surgical image analysis. Second, the annotation of surgical images is not only technically demanding but also costly, mainly because professional medical knowledge is required to accurately annotate the details in the image. Finally, due to the diversity of surgical types, execution methods, equipment, and patient conditions, there is a high degree of heterogeneity between different datasets, which increases the difficulty of developing general algorithm models. These factors together lead to significant challenges in the research and development and application of surgical image description technology.

[0004] For the task of generating descriptions of surgical images, existing research solutions generally adopt the general architecture of image encoder-language decoder. The following are some of the model architectures that have been adopted:

[0005] 1. Image encoder. The image encoder in surgical image description generation is mainly responsible for extracting useful visual features from the image. Different encoder models have different processing methods and advantages for the characteristics and complexity of images: the Inception model can capture the multi-scale information of the image in the same layer through the multi-scale convolution kernel design, which is particularly suitable for processing the rich local details in medical images; ResNet introduces residual connections to solve the gradient vanishing problem, allowing the network to learn more complex surgical scene features; ViT comprehensively considers the relationship between the various parts of the image through the self-attention mechanism, better understands the global context, and effectively analyzes the interaction between tools and tissues during surgery; VideoSwin Transformer processes time information through hierarchical shifted windows, which can effectively capture dynamic changes in surgical videos and enhance the understanding of the surgical process.

[0006] 2. Text decoder. The LSTM model can process long-term dependency information in surgical descriptions and generate coherent and logical descriptions through a gating mechanism; the GRU has a concise structure and is efficient and resource-saving when processing short sequences; the Transformer decoder uses a self-attention mechanism to fully consider the interrelationships of various operations during surgery, generates detailed and context-rich text, and has parallel processing capabilities, making it suitable for real-time tasks.

[0007] In summary, although traditional CNN encoders such as Inception and ResNet are able to extract important local features when processing surgical images, they are relatively limited in resolution and detail granularity, and it is difficult to effectively capture the global information of the image. This is particularly true in laparoscopic surgery, which involves numerous small instruments and complex surgical details that are critical to the success of the surgery. In contrast, the Transformer encoder is better able to understand and process these complex global relationships through its self-attention mechanism, thereby providing a more comprehensive visual feature parsing.

[0008] In terms of text decoders, although RNN models such as LSTM and GRU perform well in some cases, they usually have difficulty processing longer contexts, which limits their ability to generate detailed and continuous surgical descriptions. In addition, the sequence processing characteristics of RNNs may cause them to face the problem of reduced efficiency when processing long sequences. In contrast, the Transformer decoder, with its self-attention mechanism, can not only capture long-range dependencies, but also significantly improve processing speed through parallel processing, making it more suitable for complex and information-intensive surgical description tasks.

[0009] Although the Transformer model has many advantages, its model parameters are numerous, and the GPU resources required for training are large and time-consuming, which may become a major challenge in resource-limited research environments. In addition, when pre-trained large models are fine-tuned on small datasets, there is a risk of overfitting, negative transfer, or catastrophic forgetting, especially in highly specialized and detail-rich applications such as surgical image description. Therefore, choosing the right model structure and adjusting the training strategy to avoid these problems are key to achieving effective surgical image description.

[0010] In summary, existing description generation methods for laparoscopic surgical images have the following challenges and problems:

[0011] 1. Limitations of short text descriptions in public datasets: Image descriptions in existing datasets are usually relatively brief, and their content is often equivalent to surgical triplet (instrument, anatomical location, action) extraction tasks, such as "monopolar curved scissors are cutting renal parenchyma". This description not only lacks sufficient details, but is actually difficult to surpass the multi-classification triplet extraction task in terms of accuracy, which makes it difficult for the trained model to accurately reflect the complex situations and nuances in surgery. Therefore, in order to improve the practical application value and accuracy of surgical image description generation technology, it is urgent to expand and enrich the dataset of surgical images, provide richer and more detailed descriptions, such as the location information of instruments and tissues and their interactions, and further, describe some information such as the significance and precautions of surgical operations. Such improvements will help develop more accurate and practical automatic description generation systems, thereby better supporting the teaching and practice of clinical surgery.

[0012] 2. Lack of surgical concepts and hallucination problems in pre-trained models: Existing pre-trained models are usually trained based on large-scale natural language datasets. Although they perform well in natural language generation tasks, due to the lack of expertise in the surgical field, these models often have cognitive defects when processing medical image descriptions, which leads to hallucination problems. Specifically, when generating descriptions of surgical images, pre-trained models may generate text that does not match the image content or actual operations, or even generate non-existent instruments or operating steps. This phenomenon is called "hallucination" and is mainly due to the model's lack of knowledge in specific fields, especially the lack of understanding of instruments, anatomical structures, and operating procedures in surgical scenarios.

[0013] 3. Limitations of evaluation methods for long text descriptions: As the surgical image description task expands to long texts, traditional n-gram-based evaluation metrics such as BLEU and ROUGE have gradually shown their limitations. These metrics mainly rely on the surface matching between the predicted text and the reference text, and are more effective for short texts with relatively fixed sentence structures. However, in complex surgical scenarios, the generated descriptions often involve multiple levels of information, such as the purpose of the surgical operation, process details, precautions, etc., and their length and complexity far exceed the traditional description paradigm. This makes it impossible for simple n-gram matching to accurately evaluate the actual semantic accuracy of the generated text. For example, when describing the surgical process, there may be multiple reasonable ways of expression, and traditional evaluation metrics may focus too much on the matching of words or phrases, while ignoring the semantic consistency. Therefore, it is urgent to introduce new evaluation methods that can effectively measure the semantic distance between the generated text and the real text to ensure that the model can not only generate superficially similar texts, but also accurately capture the deep semantics in the surgical scene. Summary of the invention

[0014] The present invention provides a method for generating and evaluating long text descriptions from laparoscopic surgical images, which can enhance the ability to understand surgical details in images and generate more comprehensive surgical image descriptions.

[0015] In response to problem 1 in the background technology, the present invention proposes a method for constructing prompt words for a long text description data set. In this method, prompt words suitable for long text description data of surgical images are constructed by structurally organizing information such as the bounding box coordinates of surgical images, instruments and tissues, and surgery-related knowledge. Multimodal large models such as GPT-4-Vision are used to generate more detailed and accurate long text descriptions for surgical images. By combining the structured information of images and surgical scenes, the model's ability to understand surgical details is enhanced, thereby generating a detailed description that conforms to actual operations, significantly improving the richness and effectiveness of image descriptions.

[0016] In response to problem 2 in the background technology, the present invention proposes a long text description model structure enhanced by external retrieval. In order to solve the problem of hallucinations caused by the lack of surgical concept cognition in the pre-trained model, the present invention introduces an external surgical concept vector database. During the model training process, the relevant surgical concepts are retrieved from the vector database in combination with the visual features of the surgical images, and these surgical concepts are integrated into the model through the cross-attention mechanism. This method not only improves the model's ability to understand the instruments, anatomical structures, and operations in the surgical scene, but also effectively reduces the occurrence of hallucinations, ensuring that the generated descriptions have higher accuracy and reliability in the medical field.

[0017] In response to problem 3 in the background technology, the present invention proposes a series of evaluation methods based on the characteristics of long text descriptions to solve the problem that traditional n-gram indicators such as BLEU and ROUGE cannot accurately evaluate the quality of long text generation. The specific methods include fine-tuning BERTScore to evaluate the word vector similarity of the text; designing specific prompt words for the long text description of surgical images, and using a large language model to perform semantic evaluation of image text descriptions (CLAIR); extracting triple information in the description through long text relationship extraction technology, and matching it with the triples in the real description to measure the accuracy of the generated text (Long-SPICE). This method can more comprehensively evaluate the semantic consistency of surgical image descriptions and the correctness of surgical knowledge, ensuring that the generated text describes the surgical scene with high fidelity.

[0018] Specifically, a method for generating and evaluating long text descriptions from laparoscopic surgical images includes:

[0019] Obtain laparoscopic surgical images from a public training set, construct prompt words containing laparoscopic surgical images, short text descriptions, target bounding boxes, and surgical knowledge, and automatically construct long text description data based on the prompt words using a large language model;

[0020] Build a model for generating long text descriptions of images, which includes a pre-trained visual encoder, a visual feature query transformer, a language decoder, and an external surgical concept vector database;

[0021] Use the constructed long text description data to perform supervised training on the image long text description generation model to obtain a trained image long text description generation model;

[0022] Construct a long text description evaluation method to evaluate the trained model;

[0023] The laparoscopic image system obtains the input laparoscopic image video stream, and filters the key frames as the images to be processed; the images to be processed are input into the trained and evaluated model to obtain the long text description results of the images.

[0024] Furthermore, the visual encoder uses the CLIP model, which is pre-trained on a large-scale natural image-text pairing dataset and is used to extract key visual features from laparoscopic surgical images. During the training phase of the image long text description generation model, the weights of CLIP remain frozen, making full use of the advantages of the pre-trained model in visual feature extraction, reducing the computational cost of training, and avoiding overfitting problems on small-scale surgical datasets. The frozen visual encoder also ensures the generalization ability of the model when processing different surgical images, thereby maintaining consistent performance in a variety of laparoscopic surgical scenarios.

[0025] Furthermore, the visual feature query transformer adopts Q-Former in BLIP2; Q-Former is responsible for screening the visual features extracted from the visual encoder, identifying important visual features that appear during the operation, such as surgical instruments and anatomical structures, and passing these important visual features to the language decoder so that the language decoder generates a description text that matches the image content.

[0026] Furthermore, the language decoder uses vicuna-13B, which is pre-trained with large-scale natural language data and is used to generate long text descriptions; the weights of vicuna-13B remain frozen, and only the Q-Former part in BLIP2 is trained to adapt to the text generation needs in surgical scenes; the visual features extracted by Q-Former from the visual encoder interact with the text description generation process of the language decoder to ensure that the generated description accurately reflects the details in the surgical scene. This method not only reduces the training cost, but also avoids the problem of overfitting of the model on small-scale medical datasets, while ensuring the coherence of the generated text and the accuracy of surgical knowledge.

[0027] Furthermore, the external surgical concept vector database includes two steps: construction and use;

[0028] Before training, the pre-trained visual feature query transformer is used to extract the vector of the image and the corresponding surgical concept to construct a vector database; when training the image long text description generation model, the visual features are retrieved in the vector database to obtain surgical concepts related to the surgical scene; these surgical concepts are fused with the model's visual features in the form of learnable vectors through a cross-attention mechanism, and the language decoder is guided by the feature vectors of the visual features and surgical concepts to generate accurate and detailed long text descriptions of the surgery. The introduction of surgical concepts not only improves the performance of the model in medical scenarios, but also effectively reduces the phenomenon of generating hallucinations that do not conform to the actual surgical situation.

[0029] Furthermore, the training process of the image long text description generation model is divided into a pre-training stage and a downstream task fine-tuning stage;

[0030] In the pre-training stage, the loss functions include image-text matching loss, image-text contrastive learning loss and text generation loss; the image-text matching loss is used to ensure that the image features extracted by the visual encoder and the text description generated by the language decoder are globally matched; the image-text contrastive learning loss further optimizes the alignment of visual and text features through contrastive learning to improve the cross-modal understanding ability of the model; the text generation loss is used to supervise the language decoder to generate text descriptions that are highly relevant to the image content, ensuring the accuracy and coherence of the generated text;

[0031] In the downstream task fine-tuning stage, only the Q-Former part is fine-tuned, and the language decoder is not adjusted; the training process at this time focuses on generating text descriptions of the surgery through visual features, and is constrained by a unified text generation cross-entropy loss to ensure that the visual features extracted by Q-Former accurately convey the details of the surgical scene, such as surgical instruments, anatomical structures and their relationships. This not only retains the advantages of the pre-trained large model in processing multimodal tasks, but also can adapt to the special needs of laparoscopic surgery image description tasks through fine-tuning of Q-Former, thereby improving the actual application effect of the model.

[0032] Furthermore, the image-text matching loss is calculated as follows:

[0033]

[0034] in, is the matching probability predicted by the model, y is the actual label, 1 indicates that the image and text match, and 0 indicates no match.

[0035] Furthermore, the image-text contrastive learning loss is calculated as follows:

[0036]

[0037] Among them, I i Represents the feature vector of the i-th image, T i The feature vector representing the i-th paragraph of text, sim(I i ,T i ) represents the similarity between the image and the text, and τ is the temperature hyperparameter.

[0038] Furthermore, the calculation formula of text generation loss is as follows:

[0039]

[0040] Among them, P(y t |x 1:t-1 ) means that given the previous t-1 words, the model predicts the correct word y t probability.

[0041] Furthermore, the long text description evaluation method includes BERTScore, CLAIR and Long-SPICE evaluation methods.

[0042] BERTScore, as a semantic matching evaluation method based on a pre-trained language model, can better capture the semantic similarity between texts than traditional indicators such as BLEU, ROUGE and CIDEr. Traditional BLEU, ROUGE and CIDEr indicators rely on the surface matching of n-grams, which are suitable for shorter, fixed-structure description texts, but they show obvious limitations for complex surgical scenes or long text descriptions. For example, BLEU and ROUGE have difficulty handling synonyms and changes in expressions when calculating vocabulary or phrase overlaps, while CIDEr mainly matches the word frequency of short texts in specific fields and cannot fully capture more complex semantic information in long texts. BERTScore can more effectively measure the accuracy of generated text and reference text at the semantic level by calculating the semantic similarity of sentence embedding vectors, and is particularly suitable for image description tasks in long text scenarios. Furthermore, the present invention increases the score weight of specific word units in surgical scenes based on the word segmentation of BERT Tokenizer, thereby enhancing the accuracy of the indicator in evaluating long text descriptions of surgical images.

[0043] Secondly, CLAIR is based on evaluation prompts specially designed for surgical images. These prompts can help large language models effectively focus on surgical information in the description. It can not only handle surface matching of text, but also score by understanding the deep semantics of reference text and generated text, effectively judging the rationality of the text and the completeness of the description. This method makes up for the defect that traditional indicators based on n-gram matching cannot handle semantic information. Especially in long text description tasks, CLAIR can more comprehensively evaluate the quality of image description, thereby ensuring that the text not only matches the image content, but also has coherence and deep logic.

[0044] Finally, in response to the special needs of long text descriptions, the present invention introduces the Long-SPICE index for complex scenes. SPICE refers to decomposing the generated text into a set of semantic relationship graphs through scene graph extraction technology, and evaluating the accuracy of text descriptions through these relationship graphs. However, the traditional SPICE method is limited by the extraction technology and is only applicable to the scene graph extraction of a single sentence, and it is difficult to cope with the situation of multiple sentences or complex scenes in long texts. Therefore, the present invention further utilizes a large language model for relationship extraction, extracts multiple semantic relationships in long texts, and matches them with the actual content such as instruments and anatomical structures extracted from the reference description for evaluation. This extension enables SPICE to effectively process long text descriptions and improve its evaluation accuracy of key information in complex descriptions.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention significantly improves the accuracy and richness of surgical image description datasets by constructing prompt words for surgical scenes and fully combining the structured information of surgical scenes. Compared with traditional short text description datasets, this invention generates more detailed long text descriptions by organizing images, short text descriptions, annotation boxes and specific surgical knowledge, which enhances the model's ability to understand surgical details and provides surgeons with a more comprehensive description of surgical images.

[0047] 2. This invention introduces an external surgical concept vector database and uses the cross-attention mechanism in the query transformer to enhance the model's understanding of key surgical concepts, thereby reducing the occurrence of hallucinations. This design improves the professionalism and accuracy of the generated text, ensures that the description content can effectively reflect the actual surgical instruments and anatomical locations that appear during the operation, as well as their locations and interactions, and enhances the practicality of the model in complex surgical scenarios.

[0048] 3. Aiming at the problem that the traditional n-gram index cannot accurately evaluate the quality of long text description, the present invention proposes a new long text description evaluation method. The present invention not only focuses on the surface matching of the text, but also performs deep semantic evaluation through the semantic understanding ability of the large language model, so that the generated description not only needs to accurately reflect the surgical instruments and anatomical structures, but also needs to maintain semantic consistency and correct medical knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart for constructing long text description data of a surgical image according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of Chinese prompt words for constructing a long text description in an embodiment of the present invention;

[0051] Figure 3 A network structure diagram of a pre-trained query transformer according to an embodiment of the present invention;

[0052] Figure 4 A network structure diagram of a model for generating long text descriptions of images according to an embodiment of the present invention;

[0053] Figure 5 Generate an effect diagram for the long text description of an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of Chinese evaluation prompt words based on CLAIR according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of extracting Chinese prompt words based on a scene graph of a large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0057] A method for generating and evaluating long text descriptions from laparoscopic surgical images mainly includes the following three parts:

[0058] The construction of long text description prompt words relies on the public image-text dataset. The images, short text descriptions, segmentation annotation bounding boxes and other information are combined with surgery-specific knowledge to construct structured prompt words, and the long text description is generated through a multimodal large model.

[0059] The long text description model structure design introduces an external surgical concept knowledge base and a cross-attention mechanism based on the characteristics of the pre-trained multimodal large model, and uses the long text description dataset constructed above for training.

[0060] The long text description evaluation method is designed to overcome the shortcomings of traditional evaluation indicators in the evaluation of long text descriptions. Evaluation methods such as BERTScore, CLAIR and Long-SPICE are proposed, and the trained models are evaluated.

[0061] To describe the construction of prompt words, we use the image and segmentation annotations in the 2018 MICCAI segmentation competition dataset and the short text annotations used in subsequent work to jointly construct prompt words and input them into the GPT-4-Vision model for long text annotation. We manually review the possible errors or hallucinations in the annotation results, summarize them as surgical knowledge information, adjust the prompt words again, and re-input them into the large model for description generation. Figure 1 As shown, specifically including:

[0062] Step 1: Obtain laparoscopic surgery images and corresponding short text descriptions from the 2018 MICCAI competition public dataset.

[0063] Step 2: Annotate the segmentation mask labels of 12 instruments and 4 anatomical positions and parse them into label and bounding box coordinates (upper left corner x coordinate, upper left corner y coordinate, lower right corner x coordinate, lower right corner y coordinate) data.

[0064] Step 3: Design prompt words to organize surgical images, short text descriptions, bounding box information, and surgical knowledge information, such as Figure 2 shown.

[0065] Step 4: Call the GPT-4-Vision API, enter the prompt, and get the long text description data.

[0066] Step 5: Manually review the errors and hallucinations in the long text description, and further supplement the surgery-related knowledge in the prompt based on the corresponding information.

[0067] Using the above-constructed prompt words, repeat steps 3-5 until an ideal long text description dataset is obtained.

[0068] like Figure 4 As shown in the figure, the long text description model structure mainly consists of CLIP visual encoder, vicuna-13B language decoder, Q-Former visual feature query transformer, and external surgical concept vector database.

[0069] The construction and training of the long text description model mainly consists of three steps: pre-training of the query transformer, construction of the surgical vector database, and fine-tuning of the surgical concept vector database:

[0070] Step 1: If Figure 3 As shown in the figure, a pre-trained multimodal large model base is obtained based on the work of BLIP2. First, the input image is divided into multiple patches, and the corresponding visual features are extracted through a pre-trained visual encoder (Vision Transformer, ViT). These features encode local information in the image and represent the semantic content of the image in the form of a fixed-length vector. Then, learnable query vectors are introduced. These query vectors are randomly initialized at the beginning and are continuously optimized as the training progresses. During the training process, the query vector and the text embedding are spliced, and self-attention calculation is performed. The mask is set differently according to the subsequent pre-training objectives.

[0071] Specifically, the pre-training process includes three pre-training objectives, image-text matching (ITM), image-text contrastive learning (ITC), and text generation tasks.

[0072] The image-text matching task uses a bidirectional self-attention mask, that is, the query vector and the text vector are visible to each other. This is because the image-text matching task is a binary classification task, and it is necessary to combine the mutual information of the query vector and the text vector to determine whether the image and text match. The self-attention result continues to enter the cross-attention module, with the concatenated vector of the query vector and the text vector as the query, and the image vector as the key and value, and then continuously introduces the image information into the query vector. The result of the cross-attention enters the feedforward layer. This part is as follows Figure 3 As shown on the left side of the "Query Transformer", N layers are repeatedly stacked, and the final output is connected to the classification head. The loss of this task can be expressed as the classification cross entropy loss, which is calculated as follows:

[0073]

[0074] in, is the matching probability predicted by the model, y is the actual label, 1 means the image and text match, 0 means they do not match. The goal of ITM is to minimize the cross entropy loss of whether the image and text match. This loss is used to ensure that the image features extracted by the visual encoder are globally consistent with the text description.

[0075] The image-text matching task uses a one-way self-attention mask, that is, the query vector and the text vector are invisible to each other. This is because in contrastive learning, it is necessary to prevent the query vector from learning directly from the text, causing information leakage. After self-attention, the query vector continues to enter the cross-attention module, with the query vector as the query and the image vector as the key and value, thereby continuously introducing image information into the query vector. On the other hand, the text vector does not need to enter the cross-attention module after self-attention. Finally, the query vector and the text vector enter the feedforward layer with different weights respectively. This part is as follows Figure 3 The two paths of the “query transformer” are shown in Figure 1, with N layers of repeated stacking. The calculation results of the two vectors are optimized using contrastive learning, and the loss can be expressed as InfoNCE (Noise Contrastive

[0076] Estimation), the calculation formula is as follows:

[0077]

[0078] Among them, I i Represents the feature vector of the i-th image, T i The feature vector representing the i-th paragraph of text, sim(I i ,T i ) represents the similarity between the image and the text (usually using cosine similarity), and τ is a temperature hyperparameter. Through this contrastive learning, the model can further improve cross-modal understanding and align visual and text features.

[0079] The text generation task uses a multimodal causal self-attention mask, that is, the query vector is visible to the text vector, and the text vector itself needs to mask the token after the current position; the text vector is not visible to the query vector. This is because in the text generation task, the prediction of the next word needs to depend on the query vector and the results of the word before the current word. After self-attention, the query vector continues to enter the cross-attention module, with the query vector as the query and the image vector as the key and value, and then continuously introduces the image information into the query vector. On the other hand, the text vector of the Nth layer can obtain the image information introduced by the cross-attention of the N-1th layer in the self-attention. Finally, the text vector enters the feedforward layer. This part is as follows Figure 3As shown on the right side of the "Query Transformer", N layers are repeatedly stacked. The loss function is the standard cross entropy loss, calculated as follows:

[0080]

[0081] Among them, P(y t |x 1:t-1 ) means that given the previous t-1 words, the model predicts the correct word y t This loss is used to supervise the language model to generate descriptions related to the image content, ensuring that the generated text is coherent and relevant to the image content.

[0082] Step 2: Construction of surgical concept vector database requires the use of pre-trained BLIP-2 model to extract features from the input surgical images. Figure 3 As shown in the "fine-tuning phase" on the right, each module uses the weights obtained in the pre-training phase. First, the input image is divided into multiple patches, and the corresponding visual features are extracted through the same visual encoder as in the pre-training phase. These features encode the local information in the image and represent the semantic content of the image in the form of a vector of fixed length. The query vector optimized in the pre-training phase is then introduced. After stacking N layers of self-attention, cross-attention and feedforward layer modules, the visual vector embedding of the image is obtained.

[0083] For all images in the training set, the corresponding visual vector embeddings are extracted and their instrument category information is stored. Since surgical images may contain multiple instrument categories, the model needs to perform multi-category annotation on the embedding of each image. These image vector-surgical concept pairs are stored in a .pkl file.

[0084] Step 3: Fine-tuning phase. During the training process, first load the .pkl file obtained in step 2 into the Faiss vector database. For each input image, as in step 2, use the pre-trained query transformer to get the image vector embedding. By calculating the vector similarity, retrieve the top-k images that are most similar to the current image from the stored embedding vector set. The specific similarity calculation can be expressed by the following formula:

[0085]

[0086] Among them, v i Represents the embedding vector of the current input image, v j Represents the embedding vector of the stored image, sim(v i ,v j) represents the cosine similarity between the two. By similarity calculation, the category information of the top-k most similar images is obtained. Since these images may have overlapping categories, the present invention designs a voting mechanism to extract categories whose occurrence times are greater than a set threshold. Set a threshold T, if the occurrence times of a category is greater than T, then the category is selected as the category description of the current image.

[0087] This category information is used as text input and enters the attention mechanism together with the image information. First, a text query vector is defined and connected with the query vector extracted from the image embedding to form an image-text query. The query captures the information of the current image through the self-attention mechanism. The calculation formula of the self-attention mechanism is as follows:

[0088]

[0089] Among them, Q is the query vector, K is the key vector, and V is the value vector. is the scaling factor. In this step, the image query is combined with the self-attention mechanism to extract key information related to the current image. Next, the text query vector is used as the query vector (Query), and the text category embedding vector extracted in the previous step is used as the key-value vector (Key-Value). The two are fused through the cross-attention mechanism. The calculation formula of the cross-attention mechanism is as follows:

[0090]

[0091] Among them, Q text-former is the text query vector, K text ,V text is the text embedding vector. Through the above steps, the text query captures the image information, and the image query captures the semantic information of the text category. Finally, the two query vectors are connected and linearly projected. The formula is as follows:

[0092] h=W[q img-forme ;q text-former ]+b

[0093] Among them, q img-forme is the image query vector, q text-former is the text query vector, [;] represents vector concatenation, W is the linear projection matrix, and b is the bias term. The projected result is input into the frozen large language model to generate the final text description. The training process also uses the above cross entropy loss function. This method combines image and text information to ensure that the generated description can accurately reflect the surgical instruments and their operation scenarios, effectively improving the text generation quality of laparoscopic surgical images. The generated results are shown in the following example. Figure 5 shown.

[0094] The long text description evaluation method includes three evaluation indicators:

[0095] Indicator 1: BERTScore, an evaluation indicator based on the pre-trained BERT model. BERTScore is an evaluation method based on semantic similarity. By using the pre-trained BERT model, each word of the generated text and the reference text is embedded into a high-dimensional space to calculate the semantic similarity between them. The specific calculation formula is as follows:

[0096]

[0097] Among them, X and Y represent the word embedding sets of the generated text and the reference text respectively, and sim(x,y) is the cosine similarity between word vectors. According to the word segmentation results of BERT tokenizer, the weight of specific keywords in the surgical scene is further set to N times that of other words to enhance the evaluation performance of BERTScore for surgical image description. The formula of weighted BERTScore is as follows:

[0098]

[0099] Among them, w(x) represents the weight of each word, ∑ x∈Z w(x) is a normalization factor to ensure the comparability of the total scores under different weight distributions. Compared with traditional evaluation methods based on n-gram matching such as BLEU and ROUGE, BERTScore can better capture the semantic similarity of texts and is particularly suitable for long text evaluation in complex surgical scenarios.

[0100] Indicator 2: CLAIR, an image description evaluation indicator based on a large language model. Its working principle is to construct specific prompt words for surgical image descriptions, and use the large language model to understand the deep semantic relationship between the predicted text and the label text, especially focusing on key information such as instruments, anatomical locations, operations, surgical concepts, etc., while ignoring differences in word order or expression. The prompt word template is as follows: Figure 6 In addition to giving a score, the large language model can also provide the basis for the score, further enhancing the interpretability of the evaluation results.

[0101] Indicator 3: SPICE (SemanticPropositional Image Caption Evaluation) is a semantic scene graph image description evaluation indicator for long texts.

[0102]

[0103] Where N is the total number of pictures, The semantic parsing graph representing the generated description of the i-th image, The semantic parsing graph representing the reference description of the i-th image. F1 is the F1-score based on the generated semantic graph and the reference semantic graph, and the calculation formula is:

[0104]

[0105] Among them, Precision represents the precision between the semantic elements of the generated description and the semantic elements of the reference description, and Recall represents the recall between the semantic elements of the generated description and the semantic elements of the reference description. The traditional SPICE method is only applicable to single-sentence scene descriptions and cannot handle multiple sentences or complex scenes in long texts. The present invention improves this by further extracting multiple semantic relationships in long texts through a large language model, and matching them with key elements in the surgical scene (such as instruments, anatomical structures, etc.), thereby improving the accuracy of the evaluation of complex surgical scenes. The prompt word template for relationship extraction is as follows: Figure 7 shown.

[0106] Table 1 shows the comparison of indicators of different methods in the long text description task of laparoscopic surgical images, mainly evaluating the performance of BERTScore, CLAIR and Long-SPICE. In general, the method of combining multimodal large model with vector retrieval significantly improves the quality of generated text by introducing external surgical knowledge, especially in describing the details of surgical images and maintaining semantic consistency.

[0107] Table 1 Comparison of image long text description results

[0108]

[0109] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating and evaluating long text descriptions from laparoscopic surgical images, characterized in that: include: Obtain laparoscopic surgical images from a public training set, construct prompt words containing laparoscopic surgical images, short text descriptions, target bounding boxes, and surgical knowledge, and automatically construct long text description data based on the prompt words using a large language model; Build a model for generating long text descriptions of images, which includes a pre-trained visual encoder, a visual feature query transformer, a language decoder, and an external surgical concept vector database; Use the constructed long text description data to perform supervised training on the image long text description generation model to obtain a trained image long text description generation model; Construct a long text description evaluation method to evaluate the trained model; The laparoscopic image system obtains the input laparoscopic image video stream, and filters the key frames as the images to be processed; the images to be processed are input into the trained and evaluated model to obtain the long text description results of the images.

2. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 1, characterized in that: The visual encoder uses the CLIP model, which is pre-trained on a large-scale natural image-text pairing dataset and is used to extract key visual features from laparoscopic surgical images. During the training phase of the image long text description generation model, the weights of CLIP are kept frozen.

3. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 1, characterized in that: The visual feature query transformer uses the Q-Former in BLIP2. The Q-Former is responsible for screening the visual features extracted from the visual encoder, extracting the important visual features that appear during the surgery, and passing these important visual features to the language decoder.

4. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 3, characterized in that: The language decoder uses vicuna-13B, which is pre-trained on large-scale natural language data and is used to generate long text descriptions. The weights of vicuna-13B remain frozen, and only the Q-Former part in BLIP2 is trained to adapt to the text generation needs in surgical scenarios. The visual features extracted by Q-Former from the visual encoder interact with the text description generation process of the language decoder.

5. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 4, characterized in that: The external surgical concept vector database includes two steps: construction and use; Before training, the pre-trained visual feature query transformer is used to extract the image vector and the corresponding surgical concepts to construct a vector database. When training the image long text description generation model, the visual features are retrieved from the vector database to obtain surgical concepts related to the surgical scene. These surgical concepts are fused with the model's visual features in the form of learnable vectors through a cross-attention mechanism. The feature vectors of the visual features and surgical concepts are combined to guide the language decoder to generate accurate and detailed long text descriptions of the surgery.

6. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 5, characterized in that: The training process of the image long text description generation model is divided into the pre-training stage and the downstream task fine-tuning stage; In the pre-training stage, the loss functions include image-text matching loss, image-text contrastive learning loss and text generation loss; the image-text matching loss is used to ensure that the image features extracted by the visual encoder and the text description generated by the language decoder are globally matched; the image-text contrastive learning loss further optimizes the alignment of visual and text features through contrastive learning to improve the cross-modal understanding ability of the model; the text generation loss is used to supervise the language decoder to generate text descriptions that are highly relevant to the image content, ensuring the accuracy and coherence of the generated text; During the downstream task fine-tuning stage, only the Q-Former part is fine-tuned, and the language decoder is not adjusted; at this time, the training process focuses on generating text descriptions of the surgery through visual features, and is constrained by a unified text generation cross-entropy loss to ensure that the visual features extracted by Q-Former accurately convey the details of the surgical scene.

7. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 6, characterized in that: The image-text matching loss is calculated as follows: in, is the matching probability predicted by the model, y is the actual label, 1 indicates that the image and text match, and 0 indicates no match.

8. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 6, characterized in that: The image-text contrastive learning loss is calculated as follows: Among them, I i Represents the feature vector of the i-th image, T i The feature vector representing the i-th paragraph of text, sim(I i ,T i ) represents the similarity between the image and the text, and τ is the temperature hyperparameter.

9. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 6, characterized in that: The calculation formula for text generation loss is as follows: Among them, P(y t |x 1:t-1 ) means that given the previous t-1 words, the model predicts the correct word y t probability.

10. The method for generating and evaluating long text descriptions from laparoscopic surgical images according to claim 1, characterized in that: The long text description evaluation methods include BERTScore, CLAIR, and Long-SPICE evaluation methods.

Images