Real-time endoscope image analysis method and system based on multi-modal large model

Abstract

The invention relates to a real-time endoscope image analysis method and system based on a multi-modal large model, and the method comprises the following steps: collecting endoscope image data in real time, inputting the endoscope image data into the multi-modal large model for processing, and obtaining a lesion segmentation map, the multi-modal large model is executed by the following steps of: performing preliminary interpretation on the basis of the endoscope image data to obtain an endoscope image type and a lesion probability; based on the endoscope image type and the lesion probability, performing deep processing by adopting an expert model to obtain an initial lesion segmentation map; and summarizing the initial lesion segmentation maps, and performing comprehensive reasoning to obtain a final lesion segmentation map. Compared with the prior art, the method has the advantages of integrating multiple tasks, providing more accurate basis for diagnosis of doctors and the like.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a real-time endoscopic image analysis method based on a multimodal large model. Background Art

[0002] With the rapid development of medical imaging technology, endoscopy is playing an increasingly important role in the diagnosis of gastrointestinal diseases. Endoscopic imaging technology provides doctors with rich visual information by directly observing the internal conditions of the digestive tract, facilitating early detection of lesions, improving diagnostic accuracy, and formulating treatment plans. However, endoscopic imaging data often contains a large amount of complex information, making manual analysis of this data inefficient and potentially risky for misdiagnosis or omission.

[0003] In recent years, deep learning-based image processing techniques have made significant progress in the field of medical image analysis. Deep learning, particularly convolutional neural networks (CNNs), has demonstrated outstanding performance in tasks such as image recognition, segmentation, and classification. These techniques can automatically learn features from large amounts of data, reducing reliance on manual annotation and improving analysis accuracy and efficiency.

[0004] Despite this, existing medical image analysis systems often focus on a single task, such as disease detection or lesion segmentation. While these systems excel in their respective tasks, they lack the ability to integrate multiple tasks. Therefore, a multi-task fusion approach is needed to perform multiple tasks and provide doctors with more comprehensive information to assist in their diagnostic decisions. Summary of the Invention

[0005] The purpose of the present invention is to provide a real-time endoscopic image analysis method based on a multimodal large model that provides comprehensive reference information.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A real-time endoscopic image analysis method based on a multimodal large model comprises the following steps:

[0008] Endoscopic image data is collected in real time and input into a large multimodal model for processing to obtain a lesion segmentation map, wherein the large multimodal model includes an expert model. The execution steps of the large multimodal model include:

[0009] Based on the endoscopic image data, a preliminary interpretation is performed to obtain the endoscopic image type and lesion probability;

[0010] Based on the endoscopic image type and lesion probability, an expert model is used to perform deep processing to obtain an initial lesion segmentation map;

[0011] The initial lesion segmentation maps are summarized and subjected to comprehensive reasoning to obtain a final lesion segmentation map.

[0012] Furthermore, the endoscopic image types include digestive endoscopic images, bronchial endoscopic images, laparoscopic images and thoracic endoscopic images.

[0013] Furthermore, the multimodal large model adopts a prompt word fine-tuning method to identify various features in the endoscopic image data, and further identifies the endoscopic image type and obtains the probability of lesions.

[0014] Furthermore, the expert model includes a part localization sub-model for outputting a specific part of the image, a disease sign detection sub-model for outputting a lesion location bounding box and a corresponding lesion category, and a lesion segmentation sub-model for outputting a fine semantic segmentation mask of the lesion area, wherein:

[0015] The part localization sub-model is composed of a pre-trained medical code and a multi-layer perceptron classification head, and is obtained by supervised training and fine-tuning through images of different parts and their part labels;

[0016] The disease symptom detection sub-model is obtained by fine-tuning the detection sub-model through supervised training using the disease location bounding box and the corresponding lesion category;

[0017] The lesion segmentation sub-model takes the disease location bounding box as input, and the lesion segmentation sub-model is obtained by fine-tuning the segmentation sub-model using a refined semantic segmentation mask of the lesion area.

[0018] Furthermore, the expert model is trained using a variety of loss functions to optimize multiple tasks, where the task types include classification tasks, multi-label classification tasks, and segmentation tasks.

[0019] Furthermore, a cross entropy loss function is used for the classification task to measure the difference between the predicted label and the true label, wherein the expression of the cross entropy loss function is:

[0020]

[0021] Where, L CE is the cross entropy loss, N is the number of samples, y i is the true label, is the predicted probability.

[0022] Furthermore, a binary cross entropy loss function is used for the multi-label classification task to evaluate the classification result of each label, wherein the expression of the binary cross entropy loss function is:

[0023]

[0024] Where, L BCE is the binary cross entropy loss, N is the number of samples, y i is the true label, is the predicted probability.

[0025] Furthermore, the Dice loss function is used for the segmentation task to evaluate the overlap between the predicted area and the true area, wherein the expression of the Dice loss function is:

[0026]

[0027] Where, L Dice is the Dice loss, A is the predicted area, and B is the true area.

[0028] Furthermore, the final lesion segmentation map includes a lesion site segmentation map, a suspected lesion site segmentation map and a potential lesion area.

[0029] The present invention also provides a real-time endoscopic image analysis system based on a multimodal large model, comprising:

[0030] Endoscopic image acquisition module: used to collect endoscopic image data in real time;

[0031] A preliminary interpretation and task routing module is used to input the endoscopic image data into the multimodal large model for preliminary interpretation, and to obtain the endoscopic image type and lesion probability;

[0032] Expert model module: used for performing in-depth processing using an expert model based on the endoscopic image type and lesion probability to obtain an initial lesion segmentation map;

[0033] Result summary and interpretation module: used to summarize the initial lesion segmentation map and perform comprehensive reasoning to obtain the final lesion segmentation map;

[0034] Wherein, the multimodal large model includes an expert model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The present invention uses a multimodal large model to automatically interpret different endoscopic images and route them to multiple expert models based on image features for targeted lesion detection and segmentation, thereby solving multiple task requirements and providing doctors with more comprehensive analytical reference information.

[0037] (2) The present invention has multiple expert models to cope with the processing of different endoscopic images, and each expert model is equipped with a site positioning sub-model, a disease symptom detection sub-model and a lesion segmentation sub-model, which can process different types of endoscopic images and provide rich processing results, so that the invention is not limited to a single task and helps to improve the diagnostic efficiency of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the method flow of the present invention;

[0039] Figure 2 This is a schematic diagram of image annotation of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] Example 1

[0042] This embodiment provides a real-time endoscopic image analysis method based on a multimodal large model, such as Figure 1 As shown, the method includes the following steps:

[0043] Step 1: Real-time acquisition of endoscopic image data.

[0044] Endoscopic imaging data of patients is collected in real time through endoscopic equipment. The endoscopic imaging data includes various types, such as digestive endoscopy, bronchial endoscopy, laparoscopic endoscopy and thoracic endoscopy.

[0045] Step 2: Preliminary interpretation and task routing.

[0046] The collected endoscopic images are fed into a multimodal large model for preliminary interpretation, identifying the image type and the probability of lesions, thereby determining the likelihood of lesions. The multimodal large model is fine-tuned using prompt words from medical imaging data, with different prompt words corresponding to image type judgments and lesion determinations.

[0047] This large multimodal model was trained using a dataset of expert detailed image readings. This dataset contains multi-angle interpretations of patient images, including descriptions of body parts and symptoms, broken down into different tasks (such as body part QA and symptom QA). Through prompt-tuning, the model can effectively identify and interpret various features in endoscopic images. The results of the descriptive component analysis are used in routing and visualization interface calls of the expert model.

[0048] Based on the interpretation results, it is routed to the corresponding expert model.

[0049] Step 3: Expert model performs in-depth processing.

[0050] In this embodiment, the multimodal large model includes multiple expert models, each of which processes different types of images and preliminary interpretation results.

[0051] The training process of the expert model includes: each expert model is trained using a specific professional data set. Taking colonoscopy as an example, the data set includes detailed part annotations (such as Figure 2 As shown in Figure 2, the annotated data contains the following information:

[0052] Location marking: such as ileocecal region, terminal cecum, colon, etc.

[0053] Symptom segmentation and labeling: including mask segmentation of lesion areas and labeling of specific lesion categories (such as ulcers, polyps, etc.).

[0054] The expert model training task is carried out through multi-task combination optimization such as classification task (cross entropy loss), multi-label classification task (binary cross entropy loss), and segmentation task (Dice loss).

[0055] in:

[0056] Cross-Entropy Loss: used for classification tasks to measure the difference between the prediction and the true label.

[0057]

[0058] Among them, y i is the true label, is the predicted probability.

[0059] Binary Cross-Entropy Loss: used for multi-label classification tasks to evaluate the classification effect of each label.

[0060]

[0061] Dice Loss: used for segmentation tasks to evaluate the overlap between the predicted area and the true area.

[0062]

[0063] Among them, A is the predicted area and B is the real area.

[0064] In this embodiment, each expert model includes three sub-models, namely, a site location sub-model, a disease symptom detection sub-model, and a lesion segmentation sub-model. Specifically:

[0065] The expert model performs tasks such as site localization, disease sign detection, and lesion segmentation based on image features. Each sub-model outputs its analysis results, including information such as lesion location, lesion type, and potential lesion area. The site location sub-model consists of a classification sub-model, which is composed of a pre-trained medical code and a multi-layer perceptron classification head. It is fine-tuned through supervised training with images of different parts and their part labels, and outputs the specific part of the image (such as the ileocecal region in small intestine endoscopy images). The disease sign detection sub-model consists of a detection sub-model, which is fine-tuned through supervised training with symptom location bounding boxes and symptom categories, and outputs symptom location bounding boxes and symptom categories. The lesion segmentation sub-model uses the bounding boxes of disease sign detection as input, which is further input into the segmentation sub-model. The segmentation sub-model is fine-tuned with lesion fine semantic segmentation data and outputs a fine semantic segmentation mask of the lesion location.

[0066] Step 4: Summarize and interpret the results.

[0067] The output of each expert model is summarized and further interpreted by the multimodal large model. Specifically, the expert model's output of site identification, lesion classification, and lesion segmentation for each medical image is input. Leveraging the comprehensive and reasoning capabilities of the large language model, the original assessment information is further refined and analyzed for valid information. This information is then formatted according to a fixed case format to generate a comprehensive analysis report. This report includes image interpretation, lesion diagnosis references, and treatment recommendations.

[0068] Example 2

[0069] Endoscopic image acquisition module: used to collect endoscopic image data in real time;

[0070] A preliminary interpretation and task routing module is used to input the endoscopic image data into the multimodal large model for preliminary interpretation, and to obtain the endoscopic image type and lesion probability;

[0071] Expert model module: used for performing in-depth processing using an expert model based on the endoscopic image type and lesion probability to obtain an initial lesion segmentation map;

[0072] Result summary and interpretation module: used to summarize the initial lesion segmentation map and perform comprehensive reasoning to obtain the final lesion segmentation map;

[0073] Wherein, the multimodal large model includes an expert model.

[0074] In addition, the system also includes a visualization display module to display the final analysis results through a visualization interface. Doctors can intuitively view the images and their analysis results to assist in clinical decision-making.

[0075] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0076] It will be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0077] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0080] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0081] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A real-time endoscopic image analysis method based on a multimodal large model, characterized in that: The following steps are involved: Endoscopic image data is collected in real time and input into a large multimodal model for processing to obtain a lesion segmentation map, wherein the large multimodal model includes an expert model. The execution steps of the large multimodal model include: Based on the endoscopic image data, a preliminary interpretation is performed to obtain the endoscopic image type and lesion probability; Based on the endoscopic image type and lesion probability, an expert model is used to perform deep processing to obtain an initial lesion segmentation map; The initial lesion segmentation maps are summarized and subjected to comprehensive reasoning to obtain a final lesion segmentation map.

2. A real-time endoscopic image analysis method based on a multimodal large model according to claim 1, characterized in that: The endoscopic image types include digestive endoscopy images, bronchoscopy images, laparoscopy images and thoracic endoscopy images.

3. The real-time endoscopic image analysis method based on a multimodal large model according to claim 1, characterized in that: The multimodal large model adopts a prompt word fine-tuning method to identify various features in endoscopic image data, and further identifies the endoscopic image type and obtains the probability of lesions.

4. The method for real-time endoscopic image analysis based on a multimodal large model according to claim 1, characterized in that: The expert model includes a part localization sub-model for outputting a specific part of the image, a disease sign detection sub-model for outputting a lesion position bounding box and a corresponding lesion category, and a lesion segmentation sub-model for outputting a fine semantic segmentation mask of the lesion area, wherein: The part localization sub-model is composed of a pre-trained medical code and a multi-layer perceptron classification head, and is obtained by supervised training and fine-tuning through images of different parts and their part labels; The disease symptom detection sub-model is obtained by fine-tuning the detection sub-model through supervised training using the disease location bounding box and the corresponding lesion category; The lesion segmentation sub-model takes the disease location bounding box as input, and the lesion segmentation sub-model is obtained by fine-tuning the segmentation sub-model using a refined semantic segmentation mask of the lesion area.

5. The real-time endoscopic image analysis method based on a multimodal large model according to claim 1, characterized in that: The expert model is trained using a variety of loss functions to optimize multiple tasks, including classification tasks, multi-label classification tasks, and segmentation tasks.

6. The real-time endoscopic image analysis method based on a multimodal large model according to claim 5, characterized in that: The cross entropy loss function is used for the classification task to measure the difference between the predicted label and the true label, wherein the expression of the cross entropy loss function is: Where, L CE is the cross entropy loss, N is the number of samples, y i is the true label, is the predicted probability.

7. The real-time endoscopic image analysis method based on a multimodal large model according to claim 5, characterized in that: A binary cross entropy loss function is used for the multi-label classification task to evaluate the classification result of each label, wherein the expression of the binary cross entropy loss function is: Where, L BCE is the binary cross entropy loss, N is the number of samples, y i is the true label, is the predicted probability.

8. The method for real-time endoscopic image analysis based on a multimodal large model according to claim 5, characterized in that: The Dice loss function is used for the segmentation task to evaluate the overlap between the predicted area and the true area, wherein the expression of the Dice loss function is: Where, L Dice is the Dice loss, A is the predicted area, and B is the true area.

9. The method for real-time endoscopic image analysis based on a multimodal large model according to claim 1, characterized in that: The final lesion segmentation map includes a lesion site segmentation map, a suspected lesion site segmentation map and a potential lesion area.

10. A real-time endoscopic image analysis system based on a multimodal large model, characterized in that: include: Endoscopic image acquisition module: used to collect endoscopic image data in real time; A preliminary interpretation and task routing module is used to input the endoscopic image data into the multimodal large model for preliminary interpretation, and to obtain the endoscopic image type and lesion probability; Expert model module: used for performing in-depth processing using an expert model based on the endoscopic image type and lesion probability to obtain an initial lesion segmentation map; Result summary and interpretation module: used to summarize the initial lesion segmentation map and perform comprehensive reasoning to obtain the final lesion segmentation map; Wherein, the multimodal large model includes an expert model.

Images