Endoscopic image description report generation method and device and medium

Abstract

The invention relates to the field of artificial intelligence, in particular to an endoscopic image description report generation method and device and a medium. According to the method, a multi-modal endoscope model comprising a visual encoder, a large language model and a cross-modal alignment module is constructed, image analysis and text generation are combined, integrated processing from image recognition to text description is achieved on an endoscope image, a description report is finally generated, and therefore efficient conversion from the image to the text is achieved; the speed and the automation degree of endoscopic image analysis are improved, and the time consumption of a doctor on report writing is reduced. According to the multi-modal endoscope model, the complex medical knowledge and the image analysis result are effectively combined, so that diagnosis suggestions with clinical value can be more accurately generated, namely, a more accurate description report is generated. The multi-modal endoscope model is trained by using a multi-task data set, the capabilities of different tasks can be integrated, and the prediction of a multi-task result is realized on the same endoscope picture.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device and medium for generating a description report of an endoscopic image. Background Art

[0002] At present, endoscopic image analysis and report generation mainly rely on manual operations, and endoscopists need to spend a lot of time on quality control, lesion detection, diagnosis and report writing of collected images. With the development of deep learning technology, some automated tools have begun to assist doctors in image analysis, such as the recognition of image parts and the detection of lesions through discriminant models.

[0003] However, most of these tools can only handle single tasks, such as image recognition or lesion detection, and lack the ability to comprehensively analyze endoscopic images and texts, making it impossible to automatically generate the full text of endoscopic reports. Moreover, although existing technologies have made progress in image quality control, lesion detection, and diagnosis, they still have obvious limitations in the automated generation of endoscopic reports. For example, although some systems can generate certain parts of the report, these reports are often generated in a “fill-in-the-blank” manner based on templates, lacking an in-depth understanding of the image content and personalized description. Existing technologies are inefficient in processing unstructured text data and are unable to fully utilize existing medical report data for learning and optimization.

[0004] Therefore, after acquiring endoscopic images, how to automatically realize the process from endoscopic image recognition to description report generation is an urgent problem to be solved. Summary of the invention

[0005] The present application provides a method for generating a description report of an endoscopic image. By constructing a multimodal endoscopic model, the method can automatically realize the process from image recognition to text description report generation for the endoscopic image, thereby obtaining a description report with an in-depth understanding and personalized description of the endoscopic image.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] In a first aspect, the present invention provides a method for generating a description report of an endoscopic image, the method comprising:

[0008] Collect endoscopic image data and text data as training sets;

[0009] Building a multimodal model based on a multimodal architecture, the multimodal model comprising a visual encoder, a large language model, and a cross-modal alignment module;

[0010] The visual encoder is constructed based on the ViT model to convert the input endoscopic image data into a visual feature vector;

[0011] The large language model is built based on the Transformer architecture, captures the context information of the text data, and converts it into a vector representation to generate text features;

[0012] The cross-modal alignment module is constructed based on multiple linear layers and a cross-attention mechanism to align and fuse the visual feature vector and the text feature;

[0013] Processing the endoscopic image data using a pre-trained semantic segmentation model to predict a mask image of a lesion in the endoscopic image;

[0014] In combination with the mask image, the multimodal model is trained using the training set to obtain a pre-trained multimodal model;

[0015] An endoscopic image to be processed is collected, and the endoscopic image to be processed is input into the pre-trained multimodal model to obtain a description report.

[0016] In a preferred example of the present application, it can be further configured that the collecting of endoscopic image data and text data includes:

[0017] Collect paper image data, endoscopic atlas data and endoscopic report image description data, wherein the paper image data is research papers related to upper gastrointestinal endoscopy, the endoscopic atlas data is endoscopic images in the atlas of the field of digestive endoscopy, and the endoscopic report image description data is the endoscopic report and its corresponding endoscopic image, and the endoscopic report includes a description of the endoscopic image and what is found under the microscope.

[0018] In a preferred example of the present application, it can be further configured to use a pre-trained semantic segmentation model to process the endoscopic image data and predict a mask image of the lesion site in the endoscopic image, including:

[0019] Acquiring a tumor delineation data set, a polyp delineation data set, and an inflammation delineation data set from the endoscopic image data as a lesion prediction training set;

[0020] Using the lesion prediction training set to train the UNet model, a tumor segmentation model, a polyp segmentation model, and an inflammation segmentation model are obtained;

[0021] The tumor segmentation model, the polyp segmentation model and the inflammation segmentation model are used respectively to predict all endoscopic images in the endoscopic image data, and a tumor mask image, a polyp mask image and an inflammation mask image of the endoscopic image are obtained respectively.

[0022] In a preferred example of the present application, it can be further configured that the combining the mask image and using the training set to train the multimodal model to obtain a pre-trained multimodal model includes:

[0023] In combination with the tumor mask image, the polyp mask image and the inflammation mask image, the multimodal model is trained using the paper image data, the endoscopic atlas data and the endoscopic report image description data in the training set;

[0024] The multimodal model is fine-tuned by using the LoRA technology, and the loss functions of training and verification are monitored until the multimodal model converges to obtain a pre-trained multimodal model.

[0025] In a preferred example of the present application, it can be further configured that after obtaining the description report, the following is further included:

[0026] A query is constructed based on the description report and its corresponding endoscopic image, and the query is processed using a large language model to generate examination findings.

[0027] In a preferred example of the present application, it can be further configured that after obtaining the description report, the following is further included:

[0028] Constructing an endoscopy knowledge vector database, wherein the database includes an endoscopy data set and medical text data;

[0029] Constructing a search engine, inputting the examination findings into the search engine, and searching for medical text data and historical similar cases semantically related to the examination findings in the endoscopic knowledge vector database;

[0030] Constructing a prompt template, splicing the patient's medical history information, the examination findings, and related medical text data and historical similar cases, and outputting the splicing result to the large language model end of the multimodal model;

[0031] After extracting the key fields of the splicing result, the large language model end analyzes the key fields in combination with the endoscopic knowledge vector database to obtain the inspection conclusion.

[0032] In a second aspect, the present application provides a device for generating a description report of an endoscopic image, the device comprising:

[0033] A data acquisition module is used to collect endoscopic image data and text data as a training set;

[0034] A model building module, for building a multimodal model based on a multimodal architecture, wherein the multimodal model includes a visual encoder, a large language model and a cross-modal alignment module; the visual encoder is built based on the ViT model to convert the input endoscopic image data into a visual feature vector; the large language model is built based on the Transformer architecture to capture the contextual information of the text data and convert it into a vector representation to generate text features; the cross-modal alignment module is built based on a multi-layer linear layer and a cross-attention mechanism to align and fuse the visual feature vector and the text feature;

[0035] A model training module, used to process the endoscopic image data using a pre-trained semantic segmentation model to predict a mask image of the lesion site in the endoscopic image; and train the multimodal model using the training set in combination with the mask image to obtain a pre-trained multimodal model;

[0036] The prediction module is used to collect the endoscopic image to be processed, input the endoscopic image to be processed into the pre-trained multimodal model, and obtain a description report.

[0037] In a third aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for generating a description report of an endoscopic image as described in any one of the above items are implemented.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method for generating a description report of an endoscopic image as described in any one of the above items is implemented.

[0039] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for generating a descriptive report of an endoscopic image as described in any one of the above items.

[0040] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0041] 1. By building a multimodal endoscopic model including a visual encoder, a large language model, and a cross-modal alignment module, image analysis is combined with text generation to achieve integrated processing of endoscopic images from image recognition to text description, and finally generate a description report, thereby achieving efficient conversion from image to text, improving the speed and automation of endoscopic image analysis, and reducing the time doctors spend on report writing.

[0042] 2. The multimodal endoscopic model can more accurately generate clinically valuable diagnostic opinions, that is, generate more accurate descriptive reports, by effectively combining complex medical knowledge with image analysis results.

[0043] 3. The multimodal endoscopic model is trained using a multi-task dataset and can integrate the capabilities of different tasks to achieve prediction of multi-task results on the same endoscopic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for generating a description report of an endoscopic image provided in one embodiment of the present application.

[0045] Figure 2 A structural diagram of a device for generating a description report of an endoscopic image provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] In one embodiment of the present application, a method for generating a description report of an endoscopic image is provided. Figure 1 As shown, the method includes:

[0048] S100: Collect endoscopic image data and text data as a training set.

[0049] In a specific implementation, the endoscopic image data and text data are endoscopic images collected from multiple sources and their corresponding description text data. In order to improve the subsequent model training efficiency, the training set data is preprocessed, and the preprocessing includes using LLM (Large Language Model) to perform data enhancement on the text description to obtain description text data with the same semantics but different expression texts, and data cleaning of the training set data.

[0050] In a preferred embodiment, the collecting of endoscopic image data and text data includes:

[0051] Collect paper image data, endoscopic atlas data and endoscopic report image description data, wherein the paper image data is research papers related to upper gastrointestinal endoscopy, the endoscopic atlas data is endoscopic images in the atlas of the field of digestive endoscopy, and the endoscopic report image description data is the endoscopic report and its corresponding endoscopic image, and the endoscopic report includes a description of the endoscopic image and what is found under the microscope.

[0052] In the specific implementation, in order to construct the paper image data, research papers related to upper gastrointestinal endoscopy were screened from the PubMed Central (PMC) database. These papers usually contain high-quality endoscopic images and their detailed captions. The construction process of the PMC dataset is divided into three stages: collection and preprocessing of literature images, extraction and annotation alignment of sub-images, and detection and annotation of sub-images. Image annotation pair collection: Download the PDF file, image file and XML-tagged literature text of the paper through the batch download function of NCBI, and use the multi-scale template matching method to obtain the image coordinates. Sub-image separation and annotation alignment: Match the image with the inline reference text in the text, and extract the images related to endoscopic technology by comparing with the images in the S2ORC corpus; use a set of keywords to compare with annotations and inline references, and use the Image-Net pre-trained ResNet-101 image classifier to further screen out qualified images, and screen out endoscopy-related image text annotation pairs. Sub-image detection and annotation: For composite images, the DETER model based on ResNet-34 is used as a sub-image detector to decompose the composite image into sub-images. The bounding boxes of the sub-images are then converted into embedded vectors and combined with the tag encoding generated by BERT. The label probability of each tag is predicted by a multi-layer perceptron. After the above three stages, a caption can be paired for each image.

[0053] To construct the endoscopic atlas data, based on three authoritative atlases in the field of digestive endoscopy, including Atlas of New Technologies and Treatments for Digestive Endoscopy, Atlas of Endoscopic Diagnostic Techniques for Early Gastrointestinal Cancer (Revised Edition), and Atlas of Endoscopic Cases for Early Upper Gastrointestinal Cancer, 940 high-quality endoscopic images were selected from the three atlases, covering various lesion types such as achalasia, esophageal varices, superficial esophageal cancer, early gastric cancer, early colorectal cancer, and various endoscopic diagnostic techniques such as white light endoscopy, chromoendoscopy, narrow-band spectral endoscopy, and magnifying endoscopy. Images and description texts in the digital endoscopic atlas were manually extracted, and the extraction results were quality controlled to ensure the consistency of image quality and description text.

[0054] In order to construct the image description data of the endoscopic report, endoscopic reports and corresponding endoscopic images were collected. The description of endoscopic images in the report includes a brief image description and detailed microscopic findings. First, the method of combining text rule preprocessing + fine-tuning large language model correction was used to extract images and their brief descriptions and microscopic descriptions from the endoscopic report. The text rule preprocessing method is as follows: extract the endoscopic report from the endoscopy system, associate and download related images; exclude reports with incorrect or missing image paths, screen out initial diagnosis reports, and exclude reports related to postoperative, radiotherapy, and chemotherapy; exclude reports with less than 4 images; standardize the microscopic text in the report and unify the site terms. In the microscopic findings of the endoscopic report, there will be text in the format of "{site}:{morphology and lesion description}", and the information of the image will be similar to "{site}:{short description}". Therefore, a rule-based method is used to match the site description. The rules include: directly returning the sentence directly corresponding to the site description; splicing the sentence where the site appears in the description; processing the upper, lower, and middle segments of the site. If the rule matching fails, it is marked as an error [error]. After preprocessing, in order to accelerate the revision process of standardized language of endoscopic reports, the large language model technology was used, and the 1-year endoscopic report revision data was included to construct a training data set with the format of [report endoscopic findings, original image site, original image description, revised image site, revised image description, revised image endoscopic findings]. The LoRA technology was used, the learning rate was set to 1e-4, and the learning length was 8 epochs. The large language model was fine-tuned to obtain the LLM-1y model. In actual use, [report endoscopic findings, original image site, original image description] can be input, and the model can automatically complete [revised image site, revised image description, revised image endoscopic findings], thereby assisting doctors in standardizing medical terms for endoscopic image sites and descriptions.

[0055] In this embodiment, by collecting and processing multi-source data and using it as a training set, the accuracy of the model in processing text data is improved, thereby improving the accuracy of the description report.

[0056] S200: Construct a multimodal model based on a multimodal architecture, wherein the multimodal model includes a visual encoder, a large language model and a cross-modal alignment module; the visual encoder is constructed based on the ViT model to convert the input endoscopic image data into a visual feature vector; the large language model is constructed based on the Transformer architecture to capture the contextual information of the text data and convert it into a vector representation to generate text features; the cross-modal alignment module is constructed based on multi-layer linear layers and a cross-attention mechanism to align and fuse the visual feature vector and the text feature.

[0057] Specifically, the multimodal model adopts a multimodal architecture similar to LlaVA, and the model includes a visual encoder, a large language model, and a cross-modal alignment module. The visual encoder uses a model ViT (Vision Transformer) based on the Transformer architecture, and its main task is to convert the input image data into a series of visual feature vectors. These feature vectors capture the semantic information of the image and can be aligned with the text vector of the language model through a cross-modal alignment module. The large language model is responsible for processing text input, which is based on the Transformer architecture and can be a GPT series. The large language model captures the contextual information of the text through a deep neural network and converts it into a vector representation. The cross-modal alignment module is responsible for aligning and fusing the image features generated by the visual encoder and the text features generated by the language model. This layer is usually implemented through multiple linear layers and a cross-attention mechanism to ensure that the information between different modalities can be effectively combined. The features generated by the visual encoder and the text features generated by the language model are fused through a cross-modal alignment module, and then the output of the description report is completed after decoding by the large language model. The model is pre-trained with large-scale image-text pairing data, that is, the training set, and the tasks include image-text matching, image description generation, text generation, etc.

[0058] S300: Processing the endoscopic image data using a pre-trained semantic segmentation model to predict a mask image of the lesion site in the endoscopic image.

[0059] In a preferred embodiment, the endoscopic image data is processed using a pre-trained semantic segmentation model to predict a mask image of a lesion in the endoscopic image, including:

[0060] Acquiring a tumor delineation data set, a polyp delineation data set, and an inflammation delineation data set from the endoscopic image data as a lesion prediction training set;

[0061] Using the lesion prediction training set to train the UNet model, a tumor segmentation model, a polyp segmentation model, and an inflammation segmentation model are obtained;

[0062] The tumor segmentation model, the polyp segmentation model and the inflammation segmentation model are used respectively to predict all endoscopic images in the endoscopic image data, and a tumor mask image, a polyp mask image and an inflammation mask image of the endoscopic image are obtained respectively.

[0063] In this embodiment, the lesion mask predicted by the tumor segmentation model, the polyp segmentation model, and the inflammation segmentation model is used as the input of the multimodal model, which can enhance the ability of the multimodal model to detect lesions and improve the accuracy of disease classification.

[0064] S400: In combination with the mask image, the multimodal model is trained using the training set to obtain a pre-trained multimodal model.

[0065] Specifically, first construct multiple description task training data of the multimodal model. Convert the training set data into a data format that can be used for multimodal model training, for example:

[0066] The code is expressed as {"query":"Image-1: \nImage-2: \nImage-3: \nImage-4: \nDescribe this image (the first image is the endoscopic image, the second image is the tumor mask, the third image is the polyp mask, and the fourth image is the inflammation mask), give the image location, image brief description and image detailed description","response":"**Location:**Descending part\n**Description:**No abnormality\n**Description:**No abnormality was found in the mucosa. ","images":["xxx-image.jpg","xxx-tumor-mask.jpg","xxx-polyp-mask.jpg","xxx-inflammation-mask.jpg"]}. Among them, the tumor mask, polyp mask and inflammation mask are the prediction results of the tumor segmentation model, polyp segmentation model and inflammation segmentation model respectively.

[0067] In a preferred embodiment, the step of combining the mask image and using the training set to train the multimodal model to obtain a pre-trained multimodal model comprises:

[0068] In combination with the tumor mask image, the polyp mask image and the inflammation mask image, the multimodal model is trained using the paper image data, the endoscopic atlas data and the endoscopic report image description data in the training set;

[0069] The multimodal model is fine-tuned by using the LoRA technology, and the loss functions of training and verification are monitored until the multimodal model converges to obtain a pre-trained multimodal model.

[0070] In this embodiment, the mask image can be understood as a visual prompt word, which is a model strategy proposed for the endoscopic image, and can better achieve cross-modal alignment. Moreover, the accuracy of the multimodal model prediction is improved through the fine-tuning technology.

[0071] S500: Acquire an endoscopic image to be processed, input the endoscopic image to be processed into the pre-trained multimodal model, and obtain a description report.

[0072] In this embodiment, the beneficial effects include at least:

[0073] 1. By building a multimodal endoscopic model including a visual encoder, a large language model, and a cross-modal alignment module, image analysis is combined with text generation to achieve integrated processing of endoscopic images from image recognition to text description, and finally generate a description report, thereby achieving efficient conversion from image to text, improving the speed and automation of endoscopic image analysis, and reducing the time doctors spend on report writing.

[0074] 2. The multimodal endoscopic model can more accurately generate clinically valuable diagnostic opinions, that is, generate more accurate descriptive reports, by effectively combining complex medical knowledge with image analysis results.

[0075] 3. The multimodal endoscopic model is trained using a multi-task dataset and can integrate the capabilities of different tasks to achieve prediction of multi-task results on the same endoscopic image.

[0076] In some embodiments, after obtaining the description report, the method further includes:

[0077] A query is constructed based on the description report and its corresponding endoscopic image, and the query is processed using a large language model to generate examination findings.

[0078] In specific implementation, after the endoscopist completes an endoscopic examination, all images acquired by the endoscopic system are predicted using the multimodal model to obtain a descriptive report, and images that pass quality control are retained. Then, construct a query, such as {"query":"The patient application form information is... Then, determine the type of operation. If it is a gastroscopy, the descriptions of the esophagus, cardia, fundus, body, angle of the stomach, antrum, pylorus, bulb, and descending part need to be listed separately. If a certain part has not been imaged, it is assumed that there is no abnormality in the part, and the output is in the format of "-**{part}**:{description}"; if it is a colonoscopy, the descriptions of the ileocecal valve, cecum, colon, rectum, ileum, appendix, ascending colon, transverse colon, descending colon, and sigmoid colon need to be listed separately. If a certain part has not been imaged, it is assumed that there is no abnormality in the part, and the output is in the format of "-**{part}**:{description}"; if it is an endoscopic treatment operation, the above text is organized into a coherent paragraph, and important operation information cannot be lost. "}. Finally, after processing the above query, the large language model can output the summarized text, that is, the examination findings.

[0079] In this embodiment, the doctor's work efficiency is improved by combining the description report with the large prediction model to automatically generate the examination findings. The LLM, that is, the large language model, is used to process the query, so that when processing different scenarios and report templates, different report writing methods can be achieved by simply adjusting the prompt words, thereby improving the flexibility of generating the examination findings.

[0080] In some embodiments, after obtaining the description report, the method further includes:

[0081] Constructing an endoscopy knowledge vector database, wherein the database includes an endoscopy data set and medical text data;

[0082] Constructing a search engine, inputting the examination findings into the search engine, and searching for medical text data and historical similar cases semantically related to the examination findings in the endoscopic knowledge vector database;

[0083] Constructing a prompt template, splicing the patient's medical history information, the examination findings, and related medical text data and historical similar cases, and outputting the splicing result to the large language model end of the multimodal model;

[0084] After extracting the key fields of the splicing result, the large language model end analyzes the key fields in combination with the endoscopic knowledge vector database to obtain the inspection conclusion.

[0085] In specific implementation, the large language model end of the multimodal model has the ability to match texts and summarize using prompt words. First, an endoscopic knowledge vector database is constructed, and the steps include:

[0086] ①Environment preparation: Install software and libraries, including Python, Weaviate or Langchain vector database client, to support the development and operation of the retrieval enhancement system.

[0087] ②Data collection and processing: The medical text data collected from the endoscopy dataset are preprocessed, including cleaning, segmentation, and vectorization, so that they can be processed by the embedded model.

[0088] ③ Embedding model training: Use the embedding model to generate vector embeddings for medical text data, and store these embeddings in a vector database to build a vector database in the medical field.

[0089] ④ Retriever construction: Design and implement a retriever component that can retrieve the most relevant medical information from the vector database based on the user's query (e.g., endoscopic image description) and the semantic similarity between the embedded blocks.

[0090] Next, a prompt template is constructed, which splices the medical history information, the examination findings, and the related medical text data and the entries matching the historical similar cases, and outputs the splicing result to the large language model end of the pre-trained multimodal model, and the prompt model performs a diagnosis and induction based on the input content. The prompt template consists of "patient information", "medical data", "similar cases", and "current examination findings", wherein "patient information" is obtained from the application form of the endoscope system, "current examination findings" are obtained from the examination findings, "medical data" and "similar cases" are obtained by querying the "current examination findings" using the endoscope knowledge vector database, and selecting the top k (optional k = 3) medical data and historical similar cases that are most relevant to the "current examination findings".

[0091] Finally, these text materials are spliced ​​together and sent to the large language model end of the multimodal endoscopy model. The large language model (LLM) preprocesses the medical history text and structures the key fields in the medical history data (such as disease diagnosis, treatment process, previous examination results, etc.). Then, the endoscopy knowledge vector database (including medical text data and historical similar cases) is used in combination with the structured medical history information to match semantically related medical data, obtain candidate medical knowledge terms and matching similarities, and generate diagnostic conclusions. Among them, the large language model end of the multimodal model supports direct text input.

[0092] An example of the output of the inspection conclusion is:

[0093] Site description: clearly state the healthy or diseased state of the examined site;

[0094] Lesion diagnosis: conclusions such as “no abnormality found” and “tumor location is clear”;

[0095] Diagnostic recommendations: Further investigation or treatment recommendations based on medical history.

[0096] In this embodiment, by building an endoscopy knowledge base and combining it with a large language model, relevant medical knowledge and similar cases are combined to strengthen the integration and application of medical knowledge and improve the accuracy and depth of diagnosis. Medical text data and historical similar cases are stored in the database and can be continuously updated without the need to retrain the large language model. On the one hand, this can prevent the LLM from making up its own nonsense, and on the other hand, it can also allow the stored data to be continuously updated with the continuous development of medical research and guidelines, thereby improving the accuracy of generating inspection conclusions.

[0097] This application also provides a device for generating a description report of an endoscopic image, see Figure 2 As shown, the device comprises:

[0098] The data acquisition module 100 is used to acquire endoscopic image data and text data as a training set;

[0099] A model building module 200 is used to build a multimodal model based on a multimodal architecture, wherein the multimodal model includes a visual encoder, a large language model and a cross-modal alignment module; the visual encoder is built based on the ViT model to convert the input endoscopic image data into a visual feature vector; the large language model is built based on the Transformer architecture to capture the contextual information of the text data and convert it into a vector representation to generate text features; the cross-modal alignment module is built based on a multi-layer linear layer and a cross-attention mechanism to align and fuse the visual feature vector and the text feature;

[0100] The model training module 300 is used to process the endoscopic image data using a pre-trained semantic segmentation model to predict a mask image of the lesion site in the endoscopic image; and train the multimodal model using the training set in combination with the mask image to obtain a pre-trained multimodal model;

[0101] The prediction module 400 is used to collect the endoscopic image to be processed, input the endoscopic image to be processed into the pre-trained multimodal model, and obtain a description report.

[0102] The functional implementation of each module in the above-mentioned endoscopic image description report generation device corresponds to the steps in the above-mentioned endoscopic image description report generation method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0103] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for generating a description report of an endoscopic image as described in any of the above embodiments are implemented.

[0104] The present application also provides a computer-readable storage medium, on which a program is stored, wherein the computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of a method for generating a description report of an endoscopic image, which will not be repeated here.

[0105] The application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for generating a descriptive report of an endoscopic image as described in any of the above embodiments.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several deformations and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. A method for generating a description report of an endoscopic image, characterized in that: include: Collect endoscopic image data and text data as training sets; Building a multimodal model based on a multimodal architecture, the multimodal model comprising a visual encoder, a large language model, and a cross-modal alignment module; The visual encoder is constructed based on the ViT model to convert the input endoscopic image data into a visual feature vector; The large language model is built based on the Transformer architecture, captures the context information of the text data, and converts it into a vector representation to generate text features; The cross-modal alignment module is constructed based on multiple linear layers and a cross-attention mechanism to align and fuse the visual feature vector and the text feature; Processing the endoscopic image data using a pre-trained semantic segmentation model to predict a mask image of a lesion in the endoscopic image; In combination with the mask image, the multimodal model is trained using the training set to obtain a pre-trained multimodal model; An endoscopic image to be processed is collected, and the endoscopic image to be processed is input into the pre-trained multimodal model to obtain a description report.

2. The method for generating a description report of an endoscopic image according to claim 1, characterized in that: The collecting of endoscopic image data and text data includes: Collect paper image data, endoscopic atlas data and endoscopic report image description data, wherein the paper image data is research papers related to upper gastrointestinal endoscopy, the endoscopic atlas data is endoscopic images in the atlas of the field of digestive endoscopy, and the endoscopic report image description data is the endoscopic report and its corresponding endoscopic image, and the endoscopic report includes a description of the endoscopic image and what is found under the microscope.

3. The method for generating a description report of an endoscopic image according to claim 2, characterized in that: The endoscopic image data is processed using a pre-trained semantic segmentation model to predict a mask image of a lesion in the endoscopic image, including: Acquiring a tumor delineation data set, a polyp delineation data set, and an inflammation delineation data set from the endoscopic image data as a lesion prediction training set; Using the lesion prediction training set to train the UNet model, a tumor segmentation model, a polyp segmentation model, and an inflammation segmentation model are obtained; The tumor segmentation model, the polyp segmentation model and the inflammation segmentation model are used respectively to predict all endoscopic images in the endoscopic image data, and a tumor mask image, a polyp mask image and an inflammation mask image of the endoscopic image are obtained respectively.

4. The method for generating a description report of an endoscopic image according to claim 3, characterized in that: The step of combining the mask image and using the training set to train the multimodal model to obtain a pre-trained multimodal model includes: In combination with the tumor mask image, the polyp mask image and the inflammation mask image, the multimodal model is trained using the paper image data, the endoscopic atlas data and the endoscopic report image description data in the training set; The multimodal model is fine-tuned by using the LoRA technology, and the loss functions of training and verification are monitored until the multimodal model converges to obtain a pre-trained multimodal model.

5. The method for generating a description report of an endoscopic image according to claim 1, characterized in that: After obtaining the description report, it also includes: A query is constructed based on the description report and its corresponding endoscopic image, and the query is processed using a large language model to generate examination findings.

6. The method for generating a description report of an endoscopic image according to claim 5, characterized in that: After obtaining the description report, it also includes: Constructing an endoscopy knowledge vector database, wherein the database includes an endoscopy data set and medical text data; Constructing a search engine, inputting the examination findings into the search engine, and searching for medical text data and historical similar cases semantically related to the examination findings in the endoscopic knowledge vector database; Constructing a prompt template, splicing the patient's medical history information, the examination findings, and related medical text data and historical similar cases, and outputting the splicing result to the large language model end of the multimodal model; After extracting the key fields of the splicing result, the large language model end analyzes the key fields in combination with the endoscopic knowledge vector database to obtain the inspection conclusion.

7. A device for generating a description report of an endoscopic image, characterized in that: include: A data acquisition module is used to collect endoscopic image data and text data as a training set; A model building module, used to build a multimodal model based on a multimodal architecture, wherein the multimodal model includes a visual encoder, a large language model, and a cross-modal alignment module; the visual encoder is built based on the ViT model to convert the input endoscopic image data into a visual feature vector; The large language model is built based on the Transformer architecture, captures the context information of the text data, and converts it into a vector representation to generate text features; The cross-modal alignment module is constructed based on multiple linear layers and a cross-attention mechanism to align and fuse the visual feature vector and the text feature; A model training module, used to process the endoscopic image data using a pre-trained semantic segmentation model to predict a mask image of the lesion site in the endoscopic image; and train the multimodal model using the training set in combination with the mask image to obtain a pre-trained multimodal model; The prediction module is used to collect the endoscopic image to be processed, input the endoscopic image to be processed into the pre-trained multimodal model, and obtain a description report.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a description report of an endoscopic image according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, wherein when the program is executed by a processor, the method for generating a description report of an endoscopic image according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method for generating a description report of an endoscopic image according to claims 1 to 6.

Images