Multimodal large model design method and system with ability to simultaneously process multiple medical visual language tasks

Abstract

The application discloses a multimodal large model design method and system with the ability of simultaneously processing multiple medical visual language tasks, constructs a benchmark data set MedVision-MT, covers five core tasks such as medical question and answer and radiographic image report generation, improves data quality through data filtering and instruction pool design, adopts a phased field adaptation training strategy, first warms up on a natural image data set, and then migrates the multi-task ability of MedVision-MT to the medical field, a visual encoder is trained in the second stage, and a multi-attribute expert prompt generation method is proposed, the model system comprises a basic module and a network structure comprising an image coding module, a vocabulary embedding module and the like, the application solves the problems of data of a multimodal large model in the medical field, cross-modal alignment and the like, improves task generalization and application ability, and promotes clinical application of medical artificial intelligence.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal large models and the field of medical image processing, and in particular to a multi-modal large model design method and system capable of simultaneously processing multiple medical visual language tasks. BACKGROUND

[0002] In recent years, multi-modal large language models (MLLMs) such as GPT-4 and LLaVA have shown excellent performance in multiple fields. In the field of text generation and visual understanding, they have strong generalization ability and can achieve good generalization effect across different tasks. Current research on multi-modal large language models focuses on general knowledge adaptation and model optimization. By aligning visual features and large language models with a large number of natural images and detailed annotations, the ability to improve dialogue and reasoning tasks with human instructions is enhanced, and excellent performance is achieved in general tasks such as image classification and text generation, with strong cross-task adaptability. However, when applied to the medical field, due to the unique difficulties of medical images and semantic levels, it faces challenges that are significantly different from those in the natural field.

[0003] Difficulty in obtaining medical image datasets: Developing an efficient multi-modal large language model requires a large amount of medical data to align visual features and large language models. However, due to patient privacy protection policies, high medical costs, and other restrictions, the number of medical image datasets is limited and difficult to obtain. In addition, the quality of medical data is low, and the granularity of image datasets is coarse, with insufficient information accuracy and details. Compared with rich and detailed image datasets in the natural field (such as large-scale Internet datasets), the challenge is enormous.

[0004] Complex and rigorous correspondence between different modalities of medical images: Medical image report terms and descriptions are professional and accurate, and key terms have special semantics. Important information cannot be replaced arbitrarily like natural image descriptions. If important information is misunderstood, it may lead to misdiagnosis and other serious consequences. Therefore, the medical field requires multi-modal large language models not only to understand images and text, but also to accurately handle medical image details to ensure accurate information transmission.

[0005] Lack of comprehensive evaluation benchmarks in the medical field: There is a perfect multi-modal large language model evaluation system in the natural field, but there is a lack of comprehensive evaluation benchmarks for medical multi-modal large language models in the medical field. Most evaluation systems are still in the early stages and have not formed a comprehensive and standardized evaluation framework. Existing research on medical multi-modal large language models has made preliminary progress in transferring original models to the medical field, but it mainly focuses on solving specific tasks and lacks flexibility and adaptability in handling a wider range of biomedical tasks. It is difficult to efficiently perform cross-task and cross-domain tasks in the face of diverse and complex medical tasks.

[0006] Some recent studies explore the potential of multi-modal large language models in medical applications and have made some progress, but they mostly focus on single-task solutions and lack the ability to design flexible and general models to handle multiple biomedical tasks. The medical field has a wide range of complex tasks, and the model's understanding, reasoning, and adaptability are highly demanding. Current medical multi-modal large language models need to further improve their task generalization and practical application capabilities.

[0007] In summary, although there have been some advances in the research of multi-modal large language models in the medical field, there are still many challenges such as data acquisition, cross-modal alignment accuracy, integration of professional knowledge, and task generalization ability. To address these issues, the present invention proposes a multi-modal large model design method and system that can handle multiple medical visual language tasks simultaneously. SUMMARY

[0008] The purpose of the present invention is to provide a multi-modal large model design method and system that can handle multiple medical visual language tasks simultaneously, breaking through the application bottleneck in the medical field and improving the task generalization and execution capabilities.

[0009] To achieve the above-mentioned purpose, the present invention is implemented by the following technical solutions:

[0010] The multi-modal large model system of the present invention can handle multiple medical visual language tasks simultaneously, and includes a multi-task end-to-end multi-modal medical large model (which is based on a multi-task medical visual dataset and a phased progressive training strategy). The multi-task end-to-end multi-modal medical large model includes basic modules and network structures, and the network structures include an image encoding module, a vocabulary embedding module, a linear projection layer module, a large language model module, and a multi-attribute expert prompt module.

[0011] The image encoding module supports image modality input and encodes the input image features using the image encoding module. The vocabulary embedding module supports text modality input and encodes the input text features using the vocabulary embedding module. The linear projection layer module aligns the image modality features and the text modality features.

[0012] The large language model module supports smooth text output and decodes the input text and image features using the large language model to obtain the final text output. The multi-attribute expert prompt module selects all pathological categories as medical entities based on the data tuples of the MedVision-MT training set (benchmark dataset MedVision-MT) to generate expert knowledge.

[0013] Preferably, each medical entity is designed with four attributes, including shape [SHAPE], size [SIZE], location [LOCATION] and symptom [SYMPTOM], and represented by A1, A2, A3 and A4 respectively, wherein the three attributes of shape [SHAPE], size [SIZE] and location [LOCATION] represent the low-level information of each medical entity in the corresponding image, and the last attribute of symptom [SYMPTOM] represents high-level information.

[0014] Preferably, in order to obtain expert-level prompts, a large language model is used to generate key attributes of each medical entity, and two generation strategies are adopted: visual question answering (VQA) and prediction (Pre); wherein in the VQA scheme (it is worth noting that the present application does not limit the response generated by the VQA scheme to a single word, which can enhance the diversity of attribute values of each medical entity), the query template is designed as: “What is the [medical entity]’s [A i ]?”; while in the Pre scheme, the sentence to be predicted is: “[medical entity]’s [A i ] is [MASK]”, wherein A i is any one of the four attributes; the response of the VQA query and the [MASK] part represent the key attribute values obtained by the two generation strategies, respectively.

[0015] Preferably, the two generation strategies are applied to each object to obtain an attribute pool, each entity containing eight different attribute values; the final expert prompt process is summarized by the following formula:

[0016] ;

[0017] In the above formula: VQA represents the association between medical images and knowledge by asking questions and obtaining answers through the question and answer strategy; χ represents the input image, which is the basic data object for visual analysis and knowledge extraction by the model; Q represents the query question, specifically: Q A1 ,…,Q AN : query questions based on medical entity attributes (such as [SHAPE], [SIZE], etc.), used to obtain attribute information of entities in medical images through the VQA strategy, and N can be understood as the number of attribute-related questions.

[0018] Pre represents the association between attributes and knowledge by predicting the mask ([MASK]) part in the medical entity attribute-related text through the prediction strategy; S A1 to S ANObject: medical entity, i.e. pathological categories, organs and other objects involved in medical images, k is used to distinguish different medical entities.

[0019] ExpertPrompt: the final generated expert prompt for inputting into the large language model, assisting the model in understanding professional knowledge and features in medical images.

[0020] Template n : template for constructing expert prompts, n can represent the number or type of templates, by filling the sampled X and medical entities (Object k ) into the template, the prompt text conforming to the model input requirements is formed.

[0021] Object k : medical entity, i.e. pathological categories, organs and other objects involved in medical images, k is used to distinguish different medical entities.

[0022] The present application has the design method of a multi-modal large model capable of simultaneously processing multiple medical visual language tasks, which comprises the following steps:

[0023] Step 1): a unified data set containing multiple medical visual tasks designed for multi-modal large model system training, specifically including:

[0024] Step 1.1) New medical visual task setting: collect and reorganize popular medical visual data sets to form a benchmark data set MedVision-MT, which includes five core tasks: medical question answering Med-VQA, radiology report generation RRG, disease localization DL, clinical diagnosis CD and local clinical diagnosis LCD;

[0025] Disease localization DL aims to determine the location of the disease; radiology report generation RRG aims to generate medical reports according to images; CD refers to disease diagnosis based on medical images; Med-VQA focuses on answering questions based on images; local clinical diagnosis LCD is the inverse process of disease localization DL, which realizes the classification of the given disease after the bounding box is given; through the above comprehensive and diverse multi-task setting, the benchmark data set MedVision-MT can provide a basis for further exploration in the field of medical MLLM;

[0026] Step 1.2) Multi-task medical image data set: a number of popular medical multi-modal data sets are reorganized to form a multi-task benchmark data set MedVision-MT. Step 2): phased adaptive training; specifically including:

[0027] Step 2.1) Multi-task warm-up training phase: first, preheat the model on a variety of large-scale natural image datasets, and then transfer the multi-task processing capability of the model from the general field to the medical field; in the multi-task preheating phase, the projection matrix W is pre-trained, and the large language model is adjusted on various natural image datasets according to the needs to realize the ability of the preheated model to process different tasks under human queries;

[0028] Step 2.2) Biomedical information adaptation phase: after completing the multi-task preheating, the ability of the model to process different visual tasks in the general field is preliminarily established; then, the model is adapted to the medical field using the multi-task benchmark dataset MedVision-MT (multi-task medical dataset MedVision-MT).

[0029] Preferred: Step 2.1) specifically includes: replacing biomedical tasks with five natural visual tasks respectively, wherein image caption generation replaces radiology report generation RRG, visual question answering replaces medical question answering Med-VQA, positioning replaces disease localization DL, classification replaces clinical diagnosis CD, and object classification replaces local clinical diagnosis LCD; gradually transfer the multi-task capability from the natural field to the medical field, and establish semantic relationships between general vocabulary and complex biomedical terminology through expert prompts to realize more effective capture of medical image features.

[0030] Preferred: In step 1.2), the sub-datasets collected by medical question answering Med-VQA include VQA-Med-2019, VQA-Med-2021, SLAKE, VQA-RAD and Path-VQA; the sub-datasets collected by radiology report generation RRG include MIMIC-CXR and IU-X-Ray datasets; the sub-datasets collected by clinical diagnosis CD include NIH datasets; the sub-datasets collected by disease localization DL and local clinical diagnosis LCD include ChestXray and MS-CXR datasets.

[0031] Preferred: In order to effectively screen meaningful and high-quality data from the sub-datasets, further data filtering is performed on both image and text modalities, and only suitable image-text pairs are retained to form the operational dataset

[0032] wherein represents an image-text pair (Image-Text Pair), which is the basic unit constituting the operational dataset, represents the text (Text) part in the image-text pair, which contains semantic information such as medical reports, questions, and diagnosis results.

[0033] Preferably, the allocation proportions of the sub-datasets ChestXray, MS-CXR, VQA-RAD, VQA-Med-2021, SLAKE, IU-X-Ray, VQA-Med-2019, NIH, Path-VQA, and MIMIC-CXR are 5.5%, 0.88%, 1.3%, 2.2%, 5.0%, 6.6%, 7.3%, 16.3%, 26.1%, and 28.8%, respectively.

[0034] Beneficial effects: 1) Data set advantage: The MedVision-MT data set constructed covers multiple medical visual tasks, and the comprehensive and diverse task settings provide a solid foundation for medical MLLM research, solve the problem of data scarcity in the medical field, and improve data quality and pertinence through data filtering and instruction pool design, which helps the model to better learn medical knowledge. 2) Training strategy advantage: The phased field adaptation training strategy first uses natural field data to warm up, and then migrates to the medical field, effectively avoiding the problem of medical data scarcity, and gradually improving the model's ability to process medical tasks. The second stage training visual encoder and multi-attribute expert prompt generation method enhances the model's capture and semantic understanding of medical visual features, bridges the field gap, and improves the model's performance in medical tasks. 3) Model advantage: The network structure of the multi-task end-to-end multi-modal medical large model, with clear division of labor among modules, the multi-attribute expert prompt module designed for the characteristics of the medical field, improves the alignment effect of visual features and large language models, enabling the model to handle multiple medical visual language tasks simultaneously, enhancing task generalization and practical application capabilities, and is expected to promote the widespread application of medical artificial intelligence in clinical practice. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is one of the medical visual multi-task example set graphs in the embodiments of the present application.

[0036] Figure 2 is the second medical visual multi-task example set graph in the embodiments of the present application.

[0037] Figure 3 is the third medical visual multi-task example set graph in the embodiments of the present application.

[0038] Figure 4 is the fourth medical visual multi-task example set graph in the embodiments of the present application.

[0039] Figure 5 is a multi-task warm-up stage schematic diagram in the embodiments of the present application.

[0040] Figure 6 is a biomedical information adaptation stage schematic diagram in the embodiments of the present application.

[0041] Figure 7Fig. 1 is a schematic diagram of a multi-attribute expert prompt module in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] The present application realizes an end-to-end complete multi-modal medical large model, specifically as follows:

[0044] 1. Unified dataset (MedVision-MT) construction

[0045] New medical visual task setting: collect and reorganize popular medical visual datasets to form the benchmark dataset MedVision-MT, covering five core tasks of medical question answering (Med-VQA), radiological report generation (RRG), disease localization (DL), clinical diagnosis (CD) and local clinical diagnosis (LCD). Each task has a clear objective, DL determines the disease location; RRG generates medical reports according to images; CD diagnoses the disease based on medical images; Med-VQA answers questions according to images; and LCD is the inverse process of DL, which classifies given disease bounding boxes. Comprehensive and diverse task settings provide a foundation for the exploration of the medical MLLM field.

[0046] Multi-task medical image dataset construction: reorganize popular medical multi-modal datasets to form MedVision-MT, collect corresponding sub-datasets for each task, filter image and text modal data, and retain suitable image-text pairs to form operation datasets Design five instruction pools for each task to construct instruction datasets.

[0047] 2. Stage-based field adaptation training strategy

[0048] Multi-task warm-up training stage: to solve the problem of insufficient medical field datasets, first preheat the model on a large-scale natural image dataset, replace biomedical tasks with natural visual tasks (image caption generation, visual question answering, etc.), pretrain the projection matrix W and fine-tune the large language model, freeze the visual encoder originally trained on natural images, gradually transfer multi-task capabilities, establish semantic connections between general vocabulary and biomedical terminology through expert prompts, and effectively capture medical image features.

[0049] Biomedical Information Adaptation Stage: The MedVision-MT dataset is used to adapt the model to the medical field. The visual encoder module is further trained to enhance the perception of medical visual features. A multi-attribute expert prompt generation method is proposed to extract expert-level medical knowledge from a large language model, bridging the semantic and visual gap between the natural and medical fields.

[0050] 3. Multi-task end-to-end multimodal medical large model

[0051] Basic modules: Improved based on the commonly used TransformerBlock and neural network modules.

[0052] Network structure: Based on the classic LLaVA model structure improvement, it includes an image encoding module (encoding input image features), a vocabulary embedding module (encoding input text features), a linear projection layer module (aligning image and text modal features), a large language model module (decoding text and image features and outputting text), and a multi-attribute expert prompting module (based on the MedVision-MT training set, selecting pathology as the medical entity to generate expert knowledge, designing four attributes for each medical entity, using VQA and Pre generation strategies to generate expert prompts, forming an attribute pool, and summarizing the expert prompting process through formulas).

[0053] Implementation method:

[0054] 1. Implementation of the MedVision-MT dataset

[0055] A new medical vision task was designed and implemented: Medical vision-related datasets were collected, such as VQA-Med-2019 and VQA-Med-2021 datasets for Medical Question Answering (Med-VQA); and MIMIC-CXR and IU-X-Ray datasets for Radiographic Report Generation (RRG). Based on the task definitions, task rules were clarified for DL ​​(Disease Location), RRG (Medical Report Generation), CD (Disease Diagnosis), Med-VQA (Image Question Answering), and LCD (Bounding Box Classification). The datasets were labeled and organized to form the MedVision-MT dataset for subsequent model training.

[0056] like Figure 1 The following is a specific embodiment:

[0057] RRG (Radiology Report Generation): Input a chest X-ray image, and the model should generate a medical report (e.g., the heart, lungs, and mediastinum are all within the normal range). Simulate the process of a radiologist writing a diagnostic report based on the image, and test the model's ability to generate medical text and understand images.

[0058] like Figure 2The following is a specific implementation: DL (Disease Localization): For chest X-ray images, by asking the question Q: Where is the effusion?, the model outputs the answer A: the coordinates of the lesion location ({57,46,78,68}, representing bounding box information), thus realizing the spatial localization of the disease in the image.

[0059] like Figure 3 The following is a specific implementation example, CD (Clinical Diagnosis): Input a chest X-ray image, and ask the question Q: "Are any diseases found?" The model outputs the answer A: "Atelectasis," simulating a scenario in clinical diagnosis where diseases are determined based on images.

[0060] like Figure 4 The following is a specific embodiment:

[0061] VQA (Medical Visual Question Answering): Using X-ray images of joints, the question Q is: Which organ system is shown in the image? The output answer A is: Musculoskeletal system. This tests the model's ability to combine image analysis with medical knowledge to answer questions.

[0062] Implementation of a multi-task medical image dataset: The collected subset datasets are integrated, for example, the ChestXray and MS-CXR datasets are combined for DL ​​and LCD tasks. Then, image and text modalities are filtered, removing blurry or incomplete images and semantically unclear or erroneous text, retaining suitable image-text pairs to construct the operational dataset. Five different instructions are designed for each task to form an instruction pool. For example, the instruction pool for the medical question-answering task includes instructions such as "What is the answer to the question corresponding to this medical image?" and "Answer the question based on this medical image", which enriches the diversity of instructions for model training.

[0063] 2. Implementation of phased training strategy

[0064] Multi-task warm-up training phase implementation: Select large-scale natural image datasets (such as the COCO dataset) and perform pre-training on these datasets. Pre-train the projection matrix W, adjusting the matrix parameters to initially align image and text features. Replace biomedical tasks with natural vision tasks. Input image captioning generation task data into the model and train it to generate text similar to medical reports (simulating the RRG task); use visual question answering task data to train the model to answer similar medical questions (simulating the Med-VQA task), etc. Freeze the visual encoder and train only the language model part, enabling the model to learn the ability to handle multiple tasks in the natural domain, preparing it for transfer to the medical field.

[0065] likeFigure 5 As shown, the multi-task warm-up stage (Stage-1: Multi-Task Warming-Up) aims to let the model first practice in the general field and accumulate multi-task processing ability, and then migrate to the medical scene. Module analysis: Pre-trained Large Language Model (such as GPT) has the basic ability of general text understanding and generation. Visual Encoder: Visual encoder is responsible for extracting image features (input natural image, output visual features). Projection W: Projection layer aligns the dimensions of visual features and text features, so that the model can integrate image and text information. Text Embedding: Text embedding module converts text (such as task instructions, questions) into vectors that the model can understand. Task replacement logic: replace medical tasks with natural scene tasks to reduce the impact of medical data scarcity: image caption generation (Caption) replaces RRG (radiology report generation) to learn the ability of “image to text description”; general visual question and answer (VQA) replaces medical VQA to accumulate “image and text question and answer” foundation; target positioning (Localization) replaces DL (disease positioning) to practice “image to spatial coordinate output”; image classification (Classification) replaces CD (clinical diagnosis) to master “image to category judgment”; object classification (Object Classification) replaces LCD (local clinical diagnosis) to strengthen fine-grained classification ability. Training characteristics: freeze VisualEncoder (because it has been pre-trained on natural images), only fine-tune the language model and the projection layer, and quickly migrate general multi-task ability.

[0066] To Figure 5Detailed explanation: I. Core process and module overview Model input: natural image (Image) + text task (Text, such as Caption, VQA); Core modules: visual encoder (Visual Encoder), projection layer (Projection W), text embedding module (Text Embedding), pre-trained large language model (Pre-trained Large Language Model); Goal: let the model learn the multi-task mapping of "image → text" (such as image caption generation, visual question answering), and accumulate general multi-modal capabilities. II. Element-by-element detailed explanation 1. Input layer: Image + Text Image (image): refers to natural scene images (such as the outdoor scene photo in the figure), which is the "visual material" for model analysis, representing general field image data (different from medical images). Text (text): contains multi-task instructions / data, corresponding to two types of general multi-modal tasks: Task 1: Caption (Task 1 image caption generation): requires the model to generate descriptive text for the image (such as "a group of people playing skateboards in the park"), and learn the mapping of "image → text description". Task 2: VQA (Task 2 visual question answering): requires the model to answer questions about the image (such as "how many people are in the picture?"), and learn the reasoning of "image + question → answer". 2. Feature encoding layer: Visual Encoder + Text Embedding Visual Encoder (visual encoder): function: extract visual features of natural images (convert images into numerical feature vectors). Meaning: let the model "understand" the image content (such as recognizing skateboards, people, park scenes), which is the "visual basis" of multi-modal fusion. Text Embedding (text embedding): function: convert "text task instructions / data" (such as image caption generation description, VQA question) into text feature vectors. Meaning: let the model "understand" the text semantics (such as distinguishing the task differences between "generating captions" and "answering questions"), which is the "text basis" of multi-modal fusion. 3. Feature alignment layer: Projection W Projection W (projection layer): function: align the dimensions of visual features (from Visual Encoder) and text features (from text embedding module Text Embedding). Meaning: solve the problem of "image feature dimension ≠ text feature dimension", so that both can be input into the large language model (such as both converted into 1024-dimensional vectors).4. Reasoning Core: Pre-trained Large Language Model (PLLM): Function: Fuse "visual features + text features" and perform multi-task reasoning (e.g., generate captions, answer questions). Input: Aligned fusion features (visual + text) after projection layer ProjectionW. Output: Multi-task results (e.g., text description for image caption generation, answers for VQA). 5. Stage Goal: Multi-task warm-up → Let the model learn the basic ability of "image understanding, text generation, image-text reasoning" in a general scenario (natural images + text tasks). Value: It can lay the foundation for subsequent "medical scenario adaptation", first master general multi-modal logic, then migrate to the medical field (e.g., medical tasks in Figure 6). That is, the ability is migrated to medical tasks (disease positioning, report generation, etc.), which is a typical multi-modal model training idea of "first general, then special".

[0067] Biomedical information adaptation stage implementation: Input MedVision-MT dataset into the model after warm-up training, train the visual encoder module, and adjust its parameters to better extract medical image features (e.g., disease features, organ features, etc.). At the same time, start the multi-attribute expert prompt generation method, select pathological categories (e.g., pneumonia, fracture, etc.) as medical entities based on the training set, and design four attributes [SHAPE] (e.g., the shape of the pneumonia shadow), [SIZE] (the size of the shadow), [LOCATION] (the location of the shadow), and [SYMPTOM] (the corresponding symptoms) for each entity. Adopt VQA strategy to build query template "[pneumonia] what is the shape of", let the model generate response; adopt Pre strategy to build sentence "[pneumonia] the size is [MASK]", [MASK] is used for classification mark, let the model predict [MASK] part. Collect attribute values generated by the two strategies to form an attribute pool for each entity. Use the expert prompt process induced by the formula to convert the information in the attribute pool into expert prompt input into the model, so that the model learns the professional knowledge and feature association in the medical field, and improves the accuracy and effectiveness of processing medical tasks.

[0068] As Figure 6As shown, the biomedical information adaptation stage (Stage-2: Biomedical Information Adaptation) aims to let the model "learn" medical professional knowledge and adapt to medical scenarios, which is a key step from a "generalist" to a "medical expert". The module is newly added and strengthened: Multi-Attribute Expert-Prompt Module: Core innovation: Solving the problem of "difficult alignment of medical terminology with general knowledge". By injecting medical entity attributes (such as disease shape, location, symptoms), it bridges the gap between general semantics and medical semantics. Task instruction refinement: Clearly define medical task types, such as: DL task: locate the location of pneumonia; RRG task: help generate image report; and VQA, CD, etc. Task instructions let the model accurately understand medical needs. Training features: Unlock VisualEncoder training to let it learn medical image-specific features (such as lesion texture, organ shape); at the same time, fine-tune with MedVision-MT medical dataset to strengthen medical knowledge.

[0069] To Figure 6 Detailed description: Figure 6 The multi-modal medical large model's reasoning process architecture diagram in the "biomedical information adaptation stage (Stage-2: Biomedical Information Adaptation)", which shows how the model processes medical images + language instructions to complete multi-task reasoning (such as disease positioning, report generation).

[0070] Specifically: I. Core flow and module overview Model input: medical image (Visual Inputs) + language instruction (Language Instruction) Core modules: visual encoder (Visual Encoder), projection layer (Projection W), text embedding module (Text Embedding), multi-attribute expert prompt module (Multi-Attribute Expert-Prompt Module), pre-trained large language model (Pre-trained Large Language Model); output: multi-task result (Chat-MedGen Response, including DL, RRG, etc. Task answer). II. Detailed explanation of each element 1. Input layer: Visual Inputs + Language Instruction Visual Inputs: refers to medical images (such as the chest X-ray image in the figure), which are the "visual materials" for model analysis, representing real clinical data. Language Instruction: natural language instructions for different medical tasks, corresponding to 5 types of core tasks (Figure 1- Figure 4Task 1 (Disease Localization): Instruction: Requires the model to locate the position of "Pneumonia" in the image. Task 2 (Radiology Report Generation): Instruction: Requires the model to generate a diagnostic report for the medical image. M-VQA (Medical Visual Question Answering): Example instruction: Requires the model to answer questions related to the medical image. CD (Clinical Diagnosis): Example instruction: Requires the model to make a disease diagnosis based on the image. LCD (Local Clinical Diagnosis): Example instruction: Requires the model to diagnose the "bounding box annotated area" in the image. 2. Feature Encoding Layer: Visual Encoder + Text Embedding + Projection W Visual Encoder (Visual Encoder): Function: Extracts visual features of the medical image (output sequence, representing visual feature vectors). Significance: Transforms "medical images" into numerical features that the model can understand, capturing information such as lesions and organ morphology. Text Embedding (Text Embedding): Function: Converts "language instructions," "expert prompts," and other text into text feature vectors (output sequence). Significance: Allows the model to understand the semantics of natural language instructions, such as distinguishing between "localize pneumonia" and "generate report." Projection W (Projection Layer): Function: Aligns the dimensions of visual features (from Visual Encoder) and text features (from Text Embedding), allowing them to be integrated. Significance: Solves the problem of "image feature dimensions ≠ text feature dimensions" and enables multi-modal information fusion. 3. Knowledge Enhancement Layer: Multi-Attribute Expert-Prompt Module Function: Integrates the "expert prompts" generated in Figure 7, supplementing the model with attribute knowledge of medical entities (such as disease shape, location, symptoms). Significance: Bridges the gap between "general semantics" and "medical professional semantics," allowing the model to understand medical details in the image (such as distinguishing between "pneumonia shadow shape" and ordinary shadows). 4. Reasoning Core: Pre-trained Large Language Model Function: Integrates "visual features + text features + expert prompts" and performs multi-task reasoning.Input: Visual features (sequence, processed by Projection W), Text features (sequence, from Text Embedding), Expert hints (implied in Multi-attribute Expert Hint module, assist semantic understanding); Output: Multi-task results (Chat-MedGen Response), corresponding to different tasks: DL task: output disease location (e.g. {60,58,70,76}, representing bounding box coordinates). RRG task: output medical report, e.g. heart size normal, both lungs clear, no focal consolidation, no pleural effusion. 5. Output layer: Multi-task results are the final responses of the model to medical tasks, outputting multi-task results: DL results: {60,58,70,76} -> location coordinates of diseases (e.g. pneumonia). RRG results: heart size normal -> medical imaging diagnosis report. Closed-loop logic: output can be fed back for model optimization, or directly as "AI assisted diagnosis suggestion" to support clinical decision-making. Figure 6 The complete display of the invention involves the scheme: "Medical scene practical process" of multi-modal medical large model: input medical image + task instruction -> 2. Encode image-text features -> 3. Inject medical attribute knowledge (expert hints) -> 4. Large model inference -> 5. Output multi-task results (positioning, report, etc.). Through "multi-modal feature fusion + medical knowledge enhancement", the general large model is adapted to the medical scene, solving the tasks of image diagnosis, report generation, etc. in real clinical practice, which is the key link of "model from general to medical special purpose".

[0071] i1 to i n : visual feature sequence (from visual encoder Visual Encoder); e1 to e n : text embedding feature sequence (from text embedding module Text Embedding); t1 to t n : final fused multi-modal feature sequence (input to large language model).

[0072] 3. Multi-task end-to-end multi-modal medical large model implementation

[0073] Basic module and network structure building: improve the basic module based on TransformerBlock, adjust the attention mechanism, feedforward network and other parameters to adapt to the needs of medical image and text processing. Build the network structure, connect the image encoding module (use convolutional neural network to encode the image), the word embedding module (use pre-trained word vector to encode the text), the linear projection layer module (map the image and text features to the same dimensional space), the large language model module (such as based on open source large language model fine-tuning), and the multi-attribute expert prompt module (integrate attribute generation and prompt construction functions) in sequence to ensure the communication connection of each module and complete the medical visual language task processing.

[0074] As Figure 7 shown, the multi-attribute expert prompt module theme: medical knowledge "precise feeding" mechanism core logic: supplement the model with "attribute details" of medical entities, let the model understand the mapping of "medical image → professional knowledge", and solve the problem of medical semantic alignment. Process disassembly: medical entity and attribute definition: select pathological categories (such as pneumonia, fracture, etc.) as medical entities, design 4 types of attributes for each entity: [SHAPE] (shape), [SIZE] (size), [LOCATION] (location): describe the "low-level visual features" of medical entities in the image; [SYMPTOM] (symptom): associate "high-level clinical knowledge" (such as the image shape of pneumonia → cough, fever, etc. symptoms). Attribute generation strategy: use large language model (such as GPT4) to generate attribute values, in two ways: VQA strategy: ask the template "[medical entity] [attribute] is what?" (such as "what is the shape of pneumonia?"), let the model generate an open answer, preserve attribute diversity; Pre strategy: fill in the template "[medical entity] [attribute] is [MASK]" (such as "the size of pneumonia is [MASK]"), let the model predict the mask content, and strengthen the accuracy of the attribute. Attribute pool and expert prompt construction: the attributes generated by the two strategies (each entity produces 8 attribute values) are stored in the "attribute pool", and then through template filling, the attributes and medical entities are combined to generate expert prompts, which are fed to the large language model, so that the model learns the association of "medical image → attribute → knowledge".

[0075] To Figure 7Detailed explanation: 1. Input and core object input: medical image (medical image [IMG] represented by dashed box, image identification), and medical entity (understood as disease, organ, etc., such as pneumonia, fracture corresponding to abstract mark). Core goal: for each medical entity, generate multi-dimensional attribute knowledge including shape (SHAPE), size (SIZE), location (LOCATION), and symptom (SYMPTOM), build "expert prompt" to feed the model, and bridge the gap between general semantics and medical professional semantics. 2. Two strategies for knowledge mining (VQA+Pre) module uses GPT4 as "knowledge engine" to mine four attributes (shape, size, location, and symptom) for medical entities through visual question answering (VQA) and prediction (Pre) strategies: (1) Visual question answering (VQA) strategy (upper half of the process) Question template: for, design four questions covering four attributes (CLS classification mark): Q A1 : Ask shape (SHAPE); Q A2 : Ask size (SIZE); Q A3 : Ask location (LOCATION); Q A4 : Ask symptom (SYMPTOM); Knowledge output: GPT4 answers these questions, generating four VQA attribute results, stored in AttributePool (attribute pool) and marked as: A1 VQA shape, A2 VQA size, A3 VQA location, A4 VQA symptom; (2) Prediction (Pre) strategy (lower half of the process) Fill in the blank template: for, design four sentences containing [MASK], also covering four attributes: S A1 : Fill in the shape; S A2 : Fill in the size; S A3 : Fill in the location; S A4 : Fill in the symptom; Knowledge output: GPT4 predicts the content of [MASK], generating four Pre attribute results, also stored in the attribute pool and marked as: A1 Pre shape, A2 Pre size, A3 Pre location, A4 Pre symptom; 3. Expert prompt generation Sampling (X): from the eight results in the attribute pool (four VQA and four Pre), randomly select one (i.e. X) using uniform distribution. Template filling: fill the selected X and medical entity into the preset template to generate expert prompts. Example logic: if X is A1 VQA (such as the answer "the shape is a piece of shadow"), the template may be "the shape of the medical entity is [X content], please analyze the image based on this", finally forming a prompt text for the large model to understand medical attributes.

[0076] Model training and testing: The processed MedVision-MT dataset is used to train the built model, adjust the parameters of each module, and optimize the loss function (such as cross-entropy loss). After training, test tasks are designed, such as inputting medical images, testing the performance of the model in medical question answering, report generation, disease positioning and diagnosis, and evaluating the model's multi-task processing ability, accuracy, generalization ability and other indicators in the medical field. According to the test results, the model is further optimized.

[0077] The whole architecture goes through the three steps of "general warming-up, medical adaptation, and attribute reinforcement", enabling the multi-modal model to grow from "being able to handle general image-text tasks" to "understanding medical professional knowledge and being able to solve complex medical visual language tasks", covering report generation, disease positioning, diagnosis and other scenarios, and providing technical support for medical AI-assisted diagnosis.

[0078] Finally, it should be noted that the present application is not limited to the above embodiments, but can have many variations. All variations that can be directly derived or inferred from the disclosed content by those of ordinary skill in the art should be considered within the scope of the present application.

Claims

1. A multimodal large model system with the ability to handle multiple medical visual language tasks simultaneously, characterized in that: The multi-modal large model system comprises a multi-task end-to-end multi-modal medical large model, the multi-task end-to-end multi-modal medical large model comprising a basic module, a network structure, the network structure comprising an image coding module, a vocabulary embedding module, a linear projection layer module, a large language model module, and a multi-attribute expert prompt module connected in communication; The image coding module supports image modality input, and the input image is feature-encoded by using the image coding module; the vocabulary embedding module supports text modality input, and the input text is feature-encoded by using a vocabulary embedding module; the linear projection layer module aligns the image modality features and the text modality features; The large language model module supports fluent text output, and the input text and image features are decoded by using the large language model to obtain the final text output; the multi-attribute expert prompt module selects all pathological categories as medical entities based on the data tuples of the MedVision-MT training set to generate expert knowledge; The MedVision-MT training set comprises five core tasks: medical question answering Med-VQA, radiographic report generation RRG, disease localization DL, clinical diagnosis CD, and local clinical diagnosis LCD; Four attributes are designed for each medical entity, specifically including shape [SHAPE], size [SIZE], location [LOCATION], and symptom [SYMPTOM], and are represented by A1, A2, A3, and A4, respectively, wherein the shape [SHAPE], size [SIZE], and location [LOCATION] three attributes represent low-level information of each medical entity in the corresponding image, and the last attribute symptom [SYMPTOM] represents high-level information; To obtain expert-level hints, the key attributes of each medical entity are generated by a large language model, and two generation strategies are adopted: visual question answering (VQA) and prediction (Pre). In the VQA scheme, the query template is designed as: "[medical entity]'s [A i ] is what"; while in the Pre scheme, the sentence to be predicted is: "[medical entity]'s [A i ] is [MASK]", where A i is any of the four attributes; the response of the VQA query and the [MASK] part represent the key attribute values obtained by the two generation schemes, respectively. Two generation strategies are applied to each object to obtain an attribute pool, and each entity contains eight different attribute values; the final expert prompt process is summarized by the following formula: ； In the formula, VQA represents the question-answering strategy; χ represents the input image; Q represents the query question; Pre represents the prediction strategy; S AN Sentences corresponding to different predictions; Q A1 To Q AN : Query questions based on medical entity attributes are used to obtain attribute information of entities in medical images through VQA strategies. N is the number of attribute-related questions; Pre represents the mining of attribute knowledge by having the model predict the masked portion in the text related to medical entity attributes through a prediction strategy; S A1 To S AN Represents the sentences corresponding to different predictions, where N is the number of attribute-related texts; ExpertPrompt: the final generated expert prompt, used as input to the large language model to assist the model in understanding the professional knowledge and features in medical images; Template n : Templates used to construct expert tips; n can represent the template number or type; X is the sampled data; Object k Medical entities are objects in medical images, including pathological categories and organs. k is used to distinguish different medical entities.

2. A method for designing a multi-modal large model with the ability to simultaneously handle multiple medical visual language tasks, characterized in that The design method comprises the following steps: Step 1): a unified data set comprising multiple medical vision tasks designed for multi-modal large model system training, specifically comprising: Step 1.1) new medical vision task setting: collecting and reorganizing popular medical vision data sets to form a benchmark data set MedVision-MT, the benchmark data set MedVision-MT comprising five core tasks: medical question answering Med-VQA, radiographic report generation RRG, disease localization DL, clinical diagnosis CD, and local clinical diagnosis LCD; Disease localization DL aims to determine the location of the disease; radiographic report generation RRG aims to generate a medical report according to the image; CD refers to disease diagnosis based on medical images; Med-VQA focuses on answering questions according to images; local clinical diagnosis LCD is the inverse process of disease localization DL, which realizes classification of the given disease after the bounding box is obtained; through the above comprehensive and diverse multi-task setting, the benchmark data set MedVision-MT can provide a basis for further exploration in the field of medical MLLM; Step 1.2) Multi-task medical image dataset: A multi-task benchmark dataset MedVision-MT is formed by reorganizing several popular medical multi-modal datasets; Step 2) Phase-based adaptive training; Specifically including: Step 2.1) Multi-task warm-up training phase: first, preheat the model on a large-scale natural image dataset, and then migrate the multi-task processing capability of the model from the general field to the medical field; In the multi-task preheating phase, the projection matrix W is pre-trained, and the large language model is adjusted on various natural image datasets according to the needs to realize the ability of the preheated model to process different tasks under human queries; Step 2.2) Biomedical information adaptation phase: after completing the multi-task preheating, the model's ability to process different visual tasks in the general field is preliminarily established; Subsequently, the model is adapted to the medical field using the multi-task benchmark dataset MedVision-MT.

3. The method of claim 2, wherein the method further comprises: determining a plurality of medical visual language tasks; and determining a plurality of medical visual language task types corresponding to the plurality of medical visual language tasks. Step 2.1) Specifically includes: replace the biomedical tasks with five natural visual tasks, where image caption generation replaces radiology report generation RRG, visual question answering replaces medical question answering Med-VQA, positioning replaces disease localization DL, classification replaces clinical diagnosis CD, and object classification replaces local clinical diagnosis LCD; Gradually migrate the multi-task capability from the natural field to the medical field, and establish semantic relationships between general vocabulary and complex biomedical terminology through expert prompts to more effectively capture medical image features.

4. The method of claim 2, wherein the method further comprises: In step 1.2), the sub-datasets collected by medical question answering Med-VQA include VQA-Med-2019, VQA-Med-2021, SLAKE, VQA-RAD, and Path-VQA; The sub-datasets collected by radiology report generation RRG include MIMIC-CXR and IU-X-Ray datasets; The sub-datasets collected by clinical diagnosis CD include the NIH dataset; The sub-datasets collected by disease localization DL and local clinical diagnosis LCD include the ChestXray and MS-CXR datasets.

5. The method of claim 4, wherein the method further comprises: determining a plurality of medical visual language tasks; and determining a plurality of medical visual language task types corresponding to the plurality of medical visual language tasks. In order to effectively screen meaningful and high-quality data from the sub-datasets, further data filtering is performed on both image and text modalities, and only suitable image-text pairs are retained to form the operation dataset wherein denotes an image-text pair, and represent the image and the text, respectively.

6. The design method of a multi-modal large model capable of simultaneously processing multiple medical visual language tasks according to claim 4 or 5, characterized in that: The allocation proportions of the sub-datasets ChestXray, MS-CXR, VQA-RAD, VQA-Med-2021, SLAKE, IU-X-Ray, VQA-Med-2019, NIH, Path-VQA, and MIMIC-CXR are 5.5%, 0.88%, 1.3%, 2.2%, 5.0%, 6.6%, 7.3%, 16.3%, 26.1%, and 28.8%, respectively.

Images