Digestive endoscopy report generation method and device based on multi-modal large language model

Abstract

The invention relates to the technical field of computer vision and medical image processing, and discloses a digestive endoscopy report generation method and device based on a multi-modal large language model. The method specifically comprises the following steps: for a plurality of upper gastrointestinal endoscopy images, extracting a first image feature through a visual encoder to perform anatomical part identification; extracting a second image feature from the first image feature by using a self-attention mechanism to perform background gastric mucosa classification; generating description about digestive tract diseases through a large language model, and under the guidance of the description, extracting third image features from the first image features to perform digestive tract disease classification; the constructed text instruction about report generation and the third image feature are combined and then input into a multi-mode decoder, and a digestive endoscopy report is generated. According to the method, a plurality of upper gastrointestinal endoscopy image analysis tasks can be processed at the same time, and a digestive endoscopy report which does not depend on a fixed template and has rich context content can be generated with high efficiency and high accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and medical image processing, in particular to a method and device for generating a digestive endoscopy report based on a multi-modal large language model. BACKGROUND

[0002] Gastrointestinal diseases are among the most common diseases worldwide, affecting approximately 20% of the global population each year. Upper gastrointestinal endoscopy (EGD) is a fundamental clinical procedure for diagnosing digestive tract diseases. During the endoscopy workflow, endoscopists carefully examine each anatomical site of the digestive tract (e.g., esophagus, cardia, duodenum) and record their observations with detailed written descriptions. However, it has been reported that the accuracy of the examination conclusion section in endoscopy reports is suboptimal, especially for endoscopists with less than 10 years of clinical experience, with a misdiagnosis rate of approximately 25% for early gastric cancer and early esophageal cancer. Given that the incidence of gastric cancer and esophageal cancer exceeds 1.4 million worldwide each year and continues to rise significantly, there is an urgent need for an automatic method that can significantly improve diagnostic accuracy.

[0003] Artificial intelligence (AI) techniques have shown superior performance in various upper gastrointestinal endoscopy (EGD) image analysis tasks. Some studies have attempted to decompose this task into multiple subtasks, such as site identification, lesion detection, and disease classification. Each subtask is accomplished by training separate deep neural networks (DNNs) (e.g., VGG-16, ResNet-50), and their outputs are subsequently aggregated into a structured template to generate the final endoscopy report. Although these modular approaches have achieved a certain degree of automation, they have significant limitations. First, due to the overhead of running multiple DNNs, efficiency is significantly lower, making them unsuitable for real-time clinical applications. Second, the completeness and authenticity of the generated report depend on the performance of each DNN. Errors in any model propagate throughout the system, reducing the reliability and clinical efficacy of the final report. Furthermore, the reliance on structured templates also limits their flexibility in generating contextually rich descriptions for complex cases. SUMMARY

[0004] The present application provides a method for generating a digestive endoscopy report based on a multi-modal large language model to solve the problems of unreliable, low efficiency, and difficulty in generating contextually rich descriptions for complex cases in the prior art.

[0005] Correspondingly, the present application also provides a device for generating a digestive endoscopy report based on a multi-modal large language model, an electronic device, and a computer-readable storage medium for ensuring the implementation and application of the above method.

[0006] To solve the above technical problems, the application discloses a method for generating a digestive endoscopy report based on a multi-modal large language model, which comprises the following steps:

[0007] For a plurality of upper digestive endoscopy images, first image features are extracted by a visual encoder, and anatomical site recognition is performed using the first image features;

[0008] Second image features are extracted from the first image features using a self-attention mechanism, and background gastric mucosa classification is performed using the second image features;

[0009] A description of a digestive tract disease is generated by a large language model, third image features are extracted from the first image features using an interactive module under the guidance of the description, and digestive tract disease classification is performed using the third image features;

[0010] Text instructions for generating a digestive endoscopy report are constructed, and the text instructions and the third image features are combined and input into a decoder to generate the digestive endoscopy report.

[0011] The application also discloses a device for generating a digestive endoscopy report based on a multi-modal large language model, which comprises the following modules:

[0012] An anatomical region recognition module is configured to extract first image features from a plurality of upper digestive endoscopy images by a visual encoder, and to perform anatomical site recognition using the first image features;

[0013] A background gastric mucosa classification module is configured to extract second image features from the first image features using a self-attention mechanism, and to perform background gastric mucosa classification using the second image features;

[0014] A digestive tract disease classification module is configured to generate a description of a digestive tract disease by a large language model, to extract third image features from the first image features using an interactive module under the guidance of the description, and to perform digestive tract disease classification using the third image features;

[0015] A report generation module is configured to construct text instructions for generating a digestive endoscopy report, to combine the text instructions and the third image features, and to input the combination into a decoder to generate the digestive endoscopy report.

[0016] The application also discloses an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements one or more methods described in the application when executing the program.

[0017] The application also discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement one or more methods described in the application.

[0018] The present application can simultaneously process multiple upper gastrointestinal endoscopy image analysis tasks, including: for multiple upper gastrointestinal endoscopy images, extracting first image features through a visual encoder, identifying anatomical sites, and facilitating subsequent generation of accurate examination results and descriptions for each anatomical site. On this basis, second image features are extracted from the first image features using a self-attention mechanism to classify the background gastric mucosa; a large language model is used to generate a description of the digestive tract disease, and under the guidance of the description, an interactive module learns relevant medical knowledge to enhance the model's understanding of disease-related visual features, extract third image features, and perform digestive tract disease classification based on the features to improve the accuracy of disease detection. Finally, a text instruction for generating a digestive endoscopy report is constructed, and after combining the text instruction and the third image features, a multi-modal decoder is input for self-recursive decoding, which can efficiently and accurately generate a digestive endoscopy report that is not dependent on a fixed template and has rich contextual content.

[0019] Additional aspects and advantages of the present application will be given in the following description section, which will become apparent from the following description or will be understood by those skilled in the art through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a digestive endoscopy report generation method based on a multi-modal large language model provided by an embodiment of the present application;

[0022] Figure 2 A model structure diagram provided by an embodiment of the present application;

[0023] Figure 3 A structure diagram of a digestive endoscopy report generation device based on a multi-modal large language model provided by an embodiment of the present application;

[0024] Figure 4 A structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0026] As those skilled in the art will readily appreciate from the disclosure herein, terms used herein are commonly used and benefit from customary meaning within the context of the relevant art and will be applied herein consistent with such customary meanings. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this disclosure and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0027] As those skilled in the art will readily appreciate from the disclosure herein, terms used herein are commonly used and benefit from customary meaning within the context of the relevant art and will be applied herein consistent with such customary meanings. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this disclosure and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0028] The terminal and the server can be connected directly or indirectly through wired or wireless communication manner, and the present application does not make any limitation here. For the technical problems existing in the prior art, the method and device for generating a digestive endoscopy report based on a multi-modal large language model provided by the present application aims to solve at least one of the technical problems of the prior art.

[0029] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0030] The embodiments of the present application provide a possible implementation manner, such as Figure 1As shown, a flowchart of a multi-modal large language model-based digestive endoscopy report generation method is provided, which can be executed by any electronic device, and can be executed on a server side or a terminal device.

[0031] As shown in Figure 1 and Figure 2 The method can include the following steps:

[0032] Step 101, for a plurality of upper gastrointestinal endoscopy images, first image features are extracted by a visual encoder, and anatomical site recognition is performed using the first image features.

[0033] In the embodiments of the present application, a pre-trained multi-modal large model (MedCLIP) visual encoder is used to extract necessary first image features from upper gastrointestinal endoscopy images, and anatomical region recognition is performed based on the features, which is used for subsequent generation of accurate examination results and descriptions for each anatomical site.

[0034] Step 102, second image features are extracted from the first image features using a self-attention mechanism, and background gastric mucosa classification is performed using the second image features.

[0035] Step 103, a description of the digestive tract disease is generated by a large language model, under the guidance of the description, third image features are extracted from the first image features using an interactive module, and digestive tract disease classification is performed using the third image features.

[0036] In the embodiments of the present application, medical knowledge is embedded into the processed first image features, gradually guiding the model to learn relevant medical knowledge and enhancing the model's understanding of disease-related visual features. Then, the third image features related to the lesion are extracted from the image features containing rich medical knowledge, so that the model can learn the complex relationship between the gastrointestinal endoscopy images and the disease, and perform multi-label digestive tract disease classification to generate reliable classification results.

[0037] Step 104, a text instruction for generating a digestive endoscopy report is constructed, and the text instruction and the third image features are combined and input into a multi-modal decoder to generate a digestive endoscopy report.

[0038] The third image features embedded with medical knowledge and the text instruction are spliced and input into the pre-trained BLIP-2 multi-modal decoder for autoregressive decoding, which fully utilizes the powerful context ability of the large language model, so that the generated digestive endoscopy report can not rely on a fixed template and has rich context content.

[0039] The embodiments of the present application can simultaneously process multiple upper gastrointestinal endoscopy (EDG) image analysis tasks, including anatomical site recognition, background gastric mucosa classification, digestive tract disease diagnosis, and digestive endoscopy report generation, and can classify various background gastric mucosa (including chronic non-atrophic gastritis, chronic atrophic gastritis (C1), chronic atrophic gastritis (C2), chronic atrophic gastritis (C3), chronic atrophic gastritis (O1), and active gastritis) and various representative digestive tract diseases (including erosive fundus polyps, gastric body polyps, antral ulcers, early gastric cancer, advanced gastric cancer, and others). The background gastric mucosa classification result is used as the first part of the "examination conclusion" part of the generated digestive endoscopy report, and the digestive tract disease classification result is used as the second part of the "examination conclusion" part, to generate a detailed and comprehensive result description.

[0040] In steps 101 to 104 described above, the embodiments of the present application use anatomical site classification loss, background gastric mucosa classification loss, digestive tract disease classification loss, and language modeling loss to optimize the model as a whole.

[0041] In the embodiments of the present application, for multiple upper gastrointestinal endoscopy images, the first image features are extracted by a visual encoder for anatomical site recognition, which facilitates the subsequent generation of accurate examination results and descriptions for each anatomical site. On this basis, the second image features are extracted from the first image features using a self-attention mechanism for background gastric mucosa classification; a large language model is used to generate a description of the digestive tract disease, and under the guidance of the description, an interactive module is guided to learn relevant medical knowledge, enhancing the model's understanding of disease-related visual features, extracting third image features, and performing digestive tract disease classification based on the features, which can improve the accuracy of disease detection. Finally, a text instruction for generating a digestive endoscopy report is constructed, and after combining the text instruction and the third image features, the multi-modal decoder is input for self-recursive decoding, which can efficiently and accurately generate a digestive endoscopy report that is not dependent on a fixed template and has rich context content.

[0042] In an optional embodiment, for multiple upper gastrointestinal endoscopy images, first image features are extracted by a visual encoder, and the first image features are used for anatomical site recognition, including:

[0043] The visual encoder of the multi-modal large model extracts features from multiple upper gastrointestinal endoscopy images to obtain first image features; the first image features include labels representing global features;

[0044] According to the labels, a linear layer is used to predict the anatomical sites in the first image features.

[0045] In the embodiments of the present application, given a set of upper gastrointestinal endoscopy (EDG) images, first, the EDG images are encoded into first image features using the visual encoder of the pre-trained multi-modal large model (MedCLIP). Each first image feature contains a [CLS] token to represent its global feature. The first image features are denoted as where [CLS] represents the [CLS] token, n is the number of local feature blocks, and d is the feature dimension.

[0046] Next, the V c A linear layer is applied to predict the anatomical parts of the images. In the embodiments of the present application, eight anatomical parts in the digestive tract are considered: esophagus, cardia, fundus, body, angle, antrum, duodenal bulb, and descending duodenum. This anatomical part recognition task is a side task that can help the visual encoder capture information about specific parts, which is crucial for generating accurate descriptions for each part in the subsequent stage.

[0047] In an optional embodiment, the second image features are extracted from the first image features using a self-attention mechanism, and the background gastric mucosa classification is performed using the second image features, which includes:

[0048] The number of first image features is unified using the temporal adaptive pooling, obtaining first image features of the same dimension, denoted as:

[0049] In an optional embodiment, the second image features are extracted from the first image features using a self-attention mechanism, and the background gastric mucosa classification is performed using the second image features, which includes:

[0050] The first image features are processed using the self-attention mechanism to obtain attention features;

[0051] The values of the specified number (M) of [CLS] tokens with the maximum value along the token dimension are taken from the attention features to obtain the second image features

[0052] The background gastric mucosa classification results are obtained by predicting from the second image features using a linear layer.

[0053] In the embodiments of the present application, the first image features of the same dimension obtained after processing the first image features using the temporal adaptive pooling On this basis, the second image features v global are extracted using the self-attention mechanism. Subsequently, the second image features v global ​The linear layer is applied to classify the background gastric mucosa. The background gastric mucosa represents the overall state of the gastric mucosa, and the diagnosis result constitutes the first part of the "inspection conclusion" section of the generated report.

[0054] In an optional embodiment, a description about the digestive tract disease is generated by a large language model, under the guidance of the description, a third image feature is extracted from the first image feature using an interaction module, and a digestive tract disease classification is performed using the third image feature, including:

[0055] generating a description about the digestive tract disease by a large language model;

[0056] extracting text features of the description using a text encoder;

[0057] extracting a third image feature from the first image feature under the guidance of the text feature using an interaction module;

[0058] predicting a digestive tract disease classification result from the third image feature using a linear layer.

[0059] In the implementation of the present application, the same dimension of the first image feature obtained after using the time sequence adaptive pooling processing on the first image feature

[0060] In the embodiment of the present application, the large language model uses the GPT-4 model, first generates descriptions of 19 representative digestive tract diseases by GPT-4 using the following prompts: "Please describe the general visual appearance details of [disease] in the upper digestive tract endoscopy image in 30 words", and the obtained descriptions are as follows: "The shape of the polyp varies, some are round and smooth, some are oval or even irregular…". The obtained description T∈{t k |k=1,2,...,K} is then used to refine the feature where K=19. These descriptions have been confirmed by endoscopy experts and have been modified as necessary. They contain rich medical knowledge and can enhance the model's understanding of disease-related visual features, especially for diseases with visually similar features. Next, the text features of these descriptions are extracted using the text encoder in the pre-trained model MedCLIP Then, under the guidance of f T , a third image feature is extracted from

[0061] In an optional embodiment, under the guidance of the text feature, a third image feature is extracted from the first image feature using an interaction module, including:

[0062] Calculate self-attention within the first image features to obtain self-attention features;

[0063] Cross-attention is used to achieve the interaction between self-attention features and text features, and the output obtains a third image feature.

[0064] In this embodiment of the application, the interaction module includes three interaction blocks, each of which first calculates... Internal self-attention, to obtain Then cross-attention was used to facilitate and f T The interaction between them allows us to obtain the visual features of the text prompts. This process allows Enhance visual information related to disease textual information. The third interactive block ultimately outputs third image features. Subsequently, all local feature blocks (M×n) are connected together to perform multi-label gastrointestinal disease diagnosis. The diagnostic results will be included in the generated report as the second part of the "Examination Conclusion" section.

[0065] In an optional embodiment, a text instruction for generating a digestive endoscopy report is constructed, and the text instruction is combined with third image features and input into a multimodal decoder to generate a digestive endoscopy report, including:

[0066] Construct text instructions for generating digestive endoscopy reports;

[0067] Extract text instruction features from text instructions using a text encoder;

[0068] The text instruction features and third-party image features are combined and then input into a multimodal decoder to generate a digestive endoscopy report.

[0069] In this embodiment of the application, the text instruction T of the multimodal model is first constructed. ins For example, "Please generate an upper gastrointestinal endoscopy report based on the provided visual features. Assistant:". Then, the BLIP-2 text encoder is used to convert this text instruction into text features. Where L represents the length of the instruction. Then, F is then... ins and The tags are connected to create a multimodal cue, which is then input into a multimodal decoder to generate a description. This process is automated and recursive; each tag is generated sequentially based on the multimodal cue and all previously generated tags. The resulting text description constitutes the "Examination Results" section of the gastrointestinal endoscopy report. This embodiment optimizes the model by calculating the visual-linguistic contrast loss between the real report and third-party image features, as well as the report generation loss between the real report and the generated gastrointestinal endoscopy report.

[0070] In the embodiments of the present application, a large language model (GPT-4) is used to generate a description of a digestive tract disease, and medical knowledge is embedded into the model through a text encoder and an interaction module. In addition, by designing multiple esophagogastroduodenoscopy (EDG) image analysis tasks, including anatomical site recognition, background gastric mucosa classification, digestive tract disease diagnosis, and digestive endoscopy report generation, and designing corresponding loss functions for overall optimization of the model, the model is gradually guided to learn relevant medical knowledge, deepening the model's understanding of anatomical sites and corresponding diseases, enabling the model to learn the complex relationship between endoscopic images and diseases, and thus having the ability to distinguish similar anatomical regions and diseases and generate reliable and accurate examination conclusions. Through the embedding of medical knowledge, the embodiments of the present application can give accurate examination conclusions and descriptions, and are superior to existing methods in terms of accuracy in disease detection. In generating digestive endoscopy reports, the powerful contextual capabilities of the large language model are fully utilized, making the generated digestive endoscopy reports superior to existing methods in terms of completeness and content length. The embodiments of the present application successfully overcome the problems of low accuracy, low efficiency, dependence on templates, and lack of contextual content in generating endoscopy reports by existing methods.

[0071] Based on the same principles as the method provided by the embodiments of the present application, the embodiments of the present application also provide a digestive endoscopy report generation device based on a multi-modal large language model, as shown in Figure 3 The device comprises:

[0072] An anatomical region recognition module 301 is configured to, for a plurality of esophagogastroduodenoscopy images, extract first image features through a visual encoder, and use the first image features for anatomical site recognition.

[0073] A background gastric mucosa classification module 302 is configured to extract second image features from the first image features using a self-attention mechanism, and use the second image features for background gastric mucosa classification.

[0074] A digestive tract disease classification module 303 is configured to generate a description of a digestive tract disease through a large language model, and under the guidance of the description, extract third image features from the first image features using an interaction module, and use the third image features for digestive tract disease classification.

[0075] A report generation module 304 is configured to construct a text instruction for digestive endoscopy report generation, combine the text instruction and the third image features, and input them into a decoder to generate a digestive endoscopy report.

[0076] In this embodiment, for multiple upper gastrointestinal endoscopic images, a visual encoder extracts first image features for anatomical site identification, facilitating the generation of accurate examination results and descriptions for each anatomical site. Based on this, a self-attention mechanism is used to extract second image features from the first image features for background gastric mucosa classification. A large language model generates descriptions of gastrointestinal diseases, guiding the interactive module to learn relevant medical knowledge, enhancing the model's understanding of disease-related visual features, and extracting third image features. Based on these features, gastrointestinal disease classification is performed, improving disease detection accuracy. Finally, text instructions for generating gastrointestinal endoscopy reports are constructed. These text instructions, combined with the third image features, are input into a multimodal decoder for autoregressive decoding, enabling the efficient and accurate generation of gastrointestinal endoscopy reports that are independent of fixed templates and possess rich contextual content.

[0077] The digestive endoscopy report generation device based on a multimodal large language model provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0078] The digestive endoscopy report generation device based on a multimodal large language model in this application embodiment can execute the digestive endoscopy report generation method based on a multimodal large language model provided in this application embodiment. The implementation principle is similar. The actions performed by each module and unit in the digestive endoscopy report generation device based on a multimodal large language model in each embodiment of this application correspond to the steps in the digestive endoscopy report generation method based on a multimodal large language model in each embodiment of this application. For detailed functional descriptions of each module of the digestive endoscopy report generation device based on a multimodal large language model, please refer to the descriptions in the corresponding digestive endoscopy report generation method based on a multimodal large language model shown above, which will not be repeated here.

[0079] Based on the same principles as the methods shown in the embodiments of this application, this application also provides an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the digestive endoscopy report generation method based on a multimodal large language model shown in any optional embodiment of this application by calling the computer program. Compared with the prior art, the digestive endoscopy report generation method based on a multimodal large language model provided in this application, for multiple upper gastrointestinal endoscopic images, extracts first image features through a visual encoder to identify anatomical locations, facilitating the generation of accurate examination results and descriptions for each anatomical location. Based on this, a self-attention mechanism is used to extract second image features from the first image features for background gastric mucosa classification; a description of digestive tract diseases is generated through a large language model; under the guidance of the description, the interactive module is guided to learn relevant medical knowledge, enhancing the model's understanding of disease-related visual features, extracting third image features, and classifying digestive tract diseases based on these features, which can improve the accuracy of disease detection. Finally, a text instruction for generating gastrointestinal endoscopy reports is constructed. The text instruction and third-party image features are combined and then input into a multimodal decoder for autoregressive decoding. This enables the efficient and accurate generation of gastrointestinal endoscopy reports that do not rely on fixed templates and have rich contextual content.

[0080] In an alternative embodiment, an electronic device, such as Figure 4 As shown, Figure 4 The illustrated electronic device 400 can be a server, including a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the electronic device 400 may also include a transceiver 404. It should be noted that in practical applications, the transceiver 404 is not limited to one unit, and the structure of this electronic device 400 does not constitute a limitation on the embodiments of this application.

[0081] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0082] Bus 402 may include a pathway for transmitting information between the aforementioned components. Bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 402 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0083] The memory 403 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0084] The memory 403 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the foregoing method embodiments.

[0085] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0086] The server provided in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0087] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0088] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0089] It should be noted that the computer-readable storage medium described above in this application can also be a computer-readable signal medium or a combination of computer-readable storage media and computer-readable storage media. Computer-readable storage media can be, for example,—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0090] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0091] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0092] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the digestive endoscopy report generation method and apparatus based on a multimodal large language model provided in the various optional implementations described above.

[0093] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the anatomical region recognition module can also be described as "an anatomical region recognition module for extracting first image features from multiple upper gastrointestinal endoscopic images using a visual encoder and using the first image features to identify anatomical locations."

[0096] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for generating digestive endoscopy reports based on a multimodal large language model, characterized in that, The method includes: For multiple upper gastrointestinal endoscopy images, the first image features are extracted by a visual encoder, and the first image features are used to identify anatomical sites. A second image feature is extracted from the first image feature using a self-attention mechanism, and the second image feature is used to classify the background gastric mucosa. A description of digestive tract diseases is generated using a large language model. Guided by the description, an interactive module is used to extract third image features from the first image features, and the third image features are used to classify digestive tract diseases. A text instruction for generating a digestive endoscopy report is constructed, and the text instruction is combined with the third image feature and input into a decoder to generate a digestive endoscopy report.

2. The method for generating digestive endoscopy reports based on a multimodal large language model according to claim 1, characterized in that, Before extracting second image features from the first image features using a self-attention mechanism and using the second image features for background gastric mucosa classification, the method further includes: Temporal adaptive pooling is used to unify the number of the first image features, thereby obtaining the first image features of the same dimension.

3. The method for generating digestive endoscopy reports based on a multimodal large language model according to claim 1, characterized in that, The process of extracting first image features from multiple upper gastrointestinal endoscopic images using a visual encoder and then using these first image features for anatomical site identification includes: The first image features are obtained by extracting features from multiple upper gastrointestinal endoscopy images using a multimodal large-scale visual encoder; the first image features include labels representing global features. Based on the markers, anatomical sites in the first image features are predicted using a linear layer.

4. The method for generating digestive endoscopy reports based on a multimodal large language model according to claim 3, characterized in that, The step of extracting second image features from the first image features using a self-attention mechanism and using the second image features for background gastric mucosa classification includes: The first image features are processed using a self-attention mechanism to obtain attention features; The second image feature is obtained by taking the maximum value of a specified number of labels along the label dimension from the attention features; The background gastric mucosa classification result is obtained by predicting from the second image features using a linear layer.

5. The method for generating digestive endoscopy reports based on a multimodal large language model according to claim 1, characterized in that, The process involves generating a description of digestive tract diseases using a large language model, and under the guidance of this description, using an interactive module to extract third image features from the first image features, and then using these third image features to classify digestive tract diseases. This includes: Generate descriptions of digestive tract diseases using large language models; Use a text encoder to extract the text features of the description; Guided by the text features, the third image features are extracted from the first image features using the interactive module; The classification results of digestive tract diseases are predicted from the third image features using a linear layer.

6. The method for generating digestive endoscopy reports based on a multimodal large language model according to claim 5, characterized in that, Under the guidance of the description, the method of extracting third image features from the first image features using the interactive module includes: Calculate the self-attention within the first image features to obtain the self-attention features; Cross-attention is used to achieve the interaction between the self-attention feature and the text feature, and the third image feature is obtained by outputting it.

7. The method for generating digestive endoscopy reports based on a multimodal large language model according to claim 1, characterized in that, The process of constructing text instructions for generating a digestive endoscopy report, combining the text instructions with the third image features, and inputting the combined text instructions into a multimodal decoder to generate a digestive endoscopy report includes: Construct the text instructions for generating the digestive endoscopy report; The text instruction features are extracted from the text instruction using a text encoder; The text instruction features and the third image features are combined and input into a multimodal decoder to generate the digestive endoscopy report.

8. A digestive endoscopy report generation device based on a multimodal large language model, characterized in that, The device includes: The anatomical region recognition module is used to extract first image features from multiple upper gastrointestinal endoscopy images through a visual encoder and use the first image features to identify anatomical regions. The background gastric mucosa classification module is used to extract second image features from the first image features using a self-attention mechanism, and to use the second image features to classify the background gastric mucosa. The digestive tract disease classification module is used to generate descriptions of digestive tract diseases through a large language model. Guided by the descriptions, the interaction module extracts third image features from the first image features and uses the third image features to classify digestive tract diseases. The report generation module is used to construct text instructions for generating a digestive endoscopy report. The text instructions are combined with the third image features and then input into the decoder to generate a digestive endoscopy report.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images