Multi-modal large model training method and device, equipment and storage medium

Abstract

One or more embodiments of the invention provide a multi-modal large model training method, apparatus and device, and a storage medium, and the multi-modal large model comprises a visual coding layer used for generating image features corresponding to an image, and a large language model used for generating a reply text based on the image features and a query text; the method comprises the following steps: aiming at aligning an image feature space of a visual coding layer and a text representation space of a large language model, adjusting a multi-modal large model to obtain an aligned multi-modal large model; based on pre-training samples which are constructed in a data synthesis mode and correspond to various tasks in the at least one task in the medical scene, pre-training the aligned multi-modal large model to obtain a pre-trained multi-modal large model; and performing fine tuning on the pre-trained multi-modal large model based on a fine tuning sample which is obtained through a manual labeling mode and corresponds to the target task in the medical scene to obtain the multi-modal large model used for executing the target task.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a training method, apparatus, device, and storage medium for a multimodal large model. Background Art

[0002] A Large Language Model (LLM) is a natural language processing model based on deep learning technology, boasting powerful language understanding and generation capabilities. LLMs are increasingly being used. For example, intelligent dialogue systems (IDSs) based on LLMs are widely used in a variety of fields, including customer service, legal aid, online education, and medical consultation. In the medical consultation field, for example, IDSs have become a key tool for improving the quality and efficiency of medical services. By simulating human communication, they provide users with medical-related conversations, Q&A, and query services, such as disease diagnosis, treatment recommendations, and medication instructions. IDSs are a specific user-facing application of LLMs. They are designed to understand and answer questions posed by users in natural language and generate concise and clear answers. Specifically, IDSs are based on LLMs, which understand and answer user questions and generate corresponding answers.

[0003] Many real-world tasks require combining different types of information to arrive at the optimal solution. For example, in healthcare scenarios, doctors may need to combine a patient's medical records and medical images to make a diagnosis, while patients may seek self-help medical advice in text form based on their own medical images or photos of their medical test reports. With technological advancements and growing practical needs, multimodal large language models (MLLMs), with their unique advantages, are gradually replacing or supplementing traditional large language models, becoming a key force in driving the development of artificial intelligence to higher levels. Multimodal large language models can process and understand data from multiple modalities (such as text, images, audio, and video). They combine technologies from fields such as natural language processing and computer vision to achieve deep fusion and understanding of multimodal data, and enable reasoning, generation, and interaction across different modalities. In practical applications, how to train multimodal large models to achieve the performance required to provide specific services has become a pressing challenge. Summary of the Invention

[0004] One or more embodiments of the present application provide the following technical solutions: The present application provides a method for training a large multimodal model; wherein the large multimodal model includes a visual encoding layer and a large language model; the visual encoding layer is used to generate image features corresponding to an image; and the large language model is used to generate a response text based on the image features and a query text; The method comprises: Adjusting the multimodal large model with the goal of aligning the image feature space of the visual encoding layer with the text representation space of the large language model to obtain an aligned multimodal large model; Obtaining pre-training samples constructed by data synthesis, and pre-training the aligned multimodal large model based on the pre-training samples to obtain a pre-trained multimodal large model; wherein the pre-training samples include medical images and query texts corresponding to various tasks in at least one task, and response texts generated based on the medical images and the query texts; Obtain fine-tuning samples obtained by manual annotation, and fine-tune the pre-trained multimodal large model based on the fine-tuning samples to obtain a multimodal large model for performing the target task; wherein the fine-tuning samples include medical images and query text corresponding to the target task, and response text annotated for the medical images and the query text.

[0005] The present application also provides a training device for a multimodal large model; wherein the multimodal large model includes a visual encoding layer and a large language model; the visual encoding layer is used to generate image features corresponding to the image; the large language model is used to generate a response text based on the image features and the query text; The device comprises: an alignment module, which adjusts the multimodal large model with the goal of aligning the image feature space of the visual encoding layer with the text representation space of the large language model to obtain an aligned multimodal large model; a pre-training module that obtains pre-training samples constructed by data synthesis and pre-trains the aligned multimodal large model based on the pre-training samples to obtain a pre-trained multimodal large model; wherein the pre-training samples include medical images and query text corresponding to various tasks in at least one task, and response text generated based on the medical images and the query text; A fine-tuning module obtains fine-tuning samples obtained by manual annotation, and fine-tunes the pre-trained multimodal large model based on the fine-tuning samples to obtain a multimodal large model for performing the target task; wherein the fine-tuning samples include medical images and query text corresponding to the target task, and response text annotated for the medical images and the query text.

[0006] The present application also provides an electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the steps of any of the above methods by running the executable instructions.

[0007] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0008] In the above technical solution, the multimodal large model may include a visual coding layer for generating image features corresponding to an image, and a large language model for generating a response text based on the image features and the query text; when training the multimodal large model, the image feature space of the visual coding layer and the text representation space of the large language model may be aligned first, and then the aligned multimodal large model may be pre-trained based on pre-training samples corresponding to various tasks in at least one task in a medical scenario constructed by data synthesis, and finally the pre-trained multimodal large model may be fine-tuned based on fine-tuning samples corresponding to specific tasks in the medical scenario obtained by manual annotation to obtain a multimodal large model for performing the specific task.

[0009] By adopting the above method, by aligning the image feature space of the visual coding layer in the multimodal large model with the text representation space of the large language model, the multimodal large model can be helped to establish the association relationship between the image and the descriptive text; by pre-training the multimodal large model on various types of medical service tasks, the visual coding layer in the multimodal large model can fully understand the characteristics and layout of the medical image, and the large language model in the multimodal large model can have certain medical expertise and reasoning and summarizing capabilities; by labeling a small amount of high-quality specific task data according to actual medical scenarios, and fine-tuning the multimodal large model on specific types of medical service tasks, the multimodal large model can learn to follow the user's instructions and output a format that meets the medical service requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The following is a description of the accompanying drawings required for describing the exemplary embodiments, in which: Figure 1 It is a schematic diagram of a multimodal large model shown in an exemplary embodiment of the present application.

[0011] Figure 2 It is a flowchart of a training method for a multimodal large model shown in an exemplary embodiment of the present application.

[0012] Figure 3It is a schematic diagram of another multimodal large model shown in an exemplary embodiment of the present application.

[0013] Figure 4 It is a schematic diagram of a training process of a multimodal large model shown in an exemplary embodiment of the present application.

[0014] Figure 5 It is a structural diagram of a device shown in an exemplary embodiment of the present application.

[0015] Figure 6 It is a block diagram of a training device for a multimodal large model shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Rather, they are merely examples consistent with certain aspects of one or more embodiments of the present application.

[0017] It should be noted that, in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the method may include more or fewer steps than those described in this application. In addition, a single step described in this application may be broken down into multiple steps for description in other embodiments; and multiple steps described in this application may be combined into a single step for description in other embodiments.

[0018] In the field of medical consultation, intelligent dialogue systems based on large language models have become a key tool for improving the quality and efficiency of medical services. By simulating human communication methods, intelligent dialogue systems provide patients with consulting services, including disease diagnosis, treatment recommendations, and drug instructions.

[0019] An intelligent dialogue system is an interactive system developed using artificial intelligence technology that provides services to users by simulating human communication. An intelligent dialogue system is a specific application of a large language model for users. It aims to understand and answer questions posed by users in natural language and generate concise and clear answers. Specifically, the intelligent dialogue system is based on a large language model, which understands and answers user questions and generates corresponding answers.

[0020] A large language model is a natural language processing model based on deep learning technology, with powerful language understanding and generation capabilities. Large language models typically refer to deep learning models trained using large amounts of text data. They can be used to understand the meaning of natural language text or generate natural language text. Large language models can handle a variety of natural language tasks, such as text classification, named entity recognition (NER), question answering, and conversation, and are an important path to artificial intelligence.

[0021] In the field of natural language processing (NLP), large-scale text datasets are often referred to as corpora. Corpora can contain a variety of text data, such as literary works, academic papers, legal documents, news reports, everyday conversations, emails, and online forum posts. By learning from the text data in a corpus, large language models can acquire and understand the patterns and regularities of natural language, enabling effective processing and generation of human language.

[0022] Large language models typically use the Transformer architecture, meaning they are deep learning models based on the Transformer architecture. Transformer-based deep learning models are a type of neural network model that excels in fields like natural language processing.

[0023] Transformer is a neural network model used for sequence-to-sequence modeling. Transformer does not rely on recursive structures and can parallelize training and inference, speeding up model processing. In deep learning models based on the Transformer architecture, a multi-layer Transformer encoder is typically used to extract features from the input sequence, and a Transformer decoder is used to convert the extracted features into an output sequence. Furthermore, such models often employ a self-attention mechanism to capture long-range dependencies in the input sequence, as well as residual connections and regularization methods to accelerate training and improve model performance.

[0024] A pretrained model is a large language model pretrained on large amounts of unlabeled text data. Pretrained models are general-purpose models; they are not designed or optimized for specific tasks. To adapt pretrained models to specific application scenarios and task requirements, they require fine-tuning to improve their performance on specific tasks. The large language model that is ultimately put into use is typically a pretrained model that has been further fine-tuned, performing supervised learning on labeled text data. Pretraining and fine-tuning are complementary processes: pretraining enables the model to acquire broad language understanding capabilities, while fine-tuning makes the model more specialized and accurate for specific tasks.

[0025] In other words, the training process of a large language model can be divided into two stages: pre-training and fine-tuning. During the pre-training stage, unsupervised learning (e.g., self-supervised learning) can be used on large, unlabeled text datasets (e.g., online encyclopedias, online articles, books, etc.). Pre-training can be done by predicting missing parts or the next word based on context, learning statistical laws and language structures such as semantics and syntax. Back-propagation and optimization algorithms (e.g., gradient descent) are used to minimize prediction loss, iteratively updating model parameters and gradually improving the model's understanding of language. During the fine-tuning phase, you can select corresponding supervised learning tasks (such as text classification, named entity recognition, question-answering systems, dialogue systems, etc.) based on the specific application scenario and task requirements, and prepare task-specific text datasets. Then, you can use the pre-trained model as the starting point for fine-tuning, using supervised learning to fine-tune on the task-specific text dataset. Specifically, you can perform the task based on this text dataset and minimize the loss used to measure the model's performance in handling specific tasks through backpropagation and optimization algorithms (such as gradient descent). The model parameters are iteratively updated to gradually improve the model's performance on specific tasks. In practical applications, fine-tuning can flexibly choose supervised, unsupervised, or semi-supervised learning methods based on the specific application scenario and the type of available data.

[0026] The language comprehension ability learned by the large language model during the pre-training and fine-tuning stages enables the large language model to perform logical inference, knowledge reasoning, or problem-solving by understanding, analyzing, and integrating text information when faced with complex problems or tasks. This ability is usually referred to as the reasoning ability of the large language model.

[0027] In practical applications, the pre-trained large language model is usually called the base model of the large language model, and the fine-tuned large language model is called the serving model of the large language.

[0028] Large language models are typically guided or stimulated by prompts (also known as prompts) to perform specific tasks. A prompt can be an initial text or text fragment provided to the large language model, such as a sentence, a question, or a conversation, intended to guide or stimulate the model to produce the corresponding output. Prompts are a key tool for guiding model output and can range from very simple to quite complex, including instructions, examples, and descriptions of the desired output format. Prompts explicitly indicate the task expected of the large language model, such as answering a question, simulating a conversation, writing an article, or translating text. Prompts also provide the large language model with essential background information and context, enabling it to understand the logic, style, theme, or stance that should be followed when generating content. Prompts can also inspire the large language model to demonstrate its inherent knowledge or specific language abilities, such as explaining complex concepts, citing regulations, or emulating the writing style of a specific author.

[0029] Since large language models are primarily used to understand and generate human language based on text processing, prompts usually appear in the form of text. However, in practical applications, large language models can also accept other forms of input as prompts, such as images, audio, and even video, provided that the large language model is designed or trained to process multimodal data.

[0030] In medical scenarios, doctors may need to combine a patient's medical records and medical images to make a diagnosis. Patients may also require self-service medical consultations in text format, based on their own medical images or photos of their medical test reports. With the advancement of technology and growing practical needs, multimodal big models, with their unique advantages, are gradually replacing or supplementing traditional big language models, becoming a key force in driving the development of artificial intelligence to higher levels. Multimodal big models can process and understand data from multiple modalities (such as text, images, audio, and video). They combine technologies from fields such as natural language processing and computer vision to achieve deep integration and understanding of multimodal data, and enable reasoning, generation, and interaction across different modalities. In practical applications, how to train multimodal big models to achieve the performance required to provide specific services has become a pressing issue.

[0031] One or more embodiments of the present application provide a technical solution for implementing the training of a multimodal large model. In this technical solution, the multimodal large model may include a visual coding layer for generating image features corresponding to an image, and a large language model for generating a reply text based on the image features and a query text. When training the multimodal large model, the image feature space of the visual coding layer and the text representation space of the large language model may be aligned first. Then, the aligned multimodal large model may be pre-trained based on pre-training samples corresponding to various tasks in at least one task in a medical scenario constructed by data synthesis. Finally, the pre-trained multimodal large model may be fine-tuned based on fine-tuning samples corresponding to specific tasks in the medical scenario obtained by manual annotation to obtain a multimodal large model for performing the specific task.

[0032] By adopting the above method, by aligning the image feature space of the visual coding layer in the multimodal large model with the text representation space of the large language model, the multimodal large model can be helped to establish the association relationship between the image and the descriptive text; by pre-training the multimodal large model on various types of medical service tasks, the visual coding layer in the multimodal large model can fully understand the characteristics and layout of the medical image, and the large language model in the multimodal large model can have certain medical expertise and reasoning and summarizing capabilities; by labeling a small amount of high-quality specific task data according to actual medical scenarios, and fine-tuning the multimodal large model on specific types of medical service tasks, the multimodal large model can learn to follow the user's instructions and output a format that meets the medical service requirements.

[0033] Please refer to Figure 1 , Figure 1 It is a schematic diagram of a multimodal large model shown in an exemplary embodiment of the present application.

[0034] Specifically, the multimodal large model can perform reasoning based on images and query text (which can be called Query or Question), so as to understand and answer the questions raised by the query text in combination with the image, and generate corresponding reply text (which can be called Response or Answer).

[0035] It should be noted that the query text can be regarded as a special prompt. The query text describes the specific problem that the user wants to solve, and the problem is usually expressed through a carefully designed prompt.

[0036] like Figure 1 As shown, the above-mentioned multimodal large model can include a visual encoding layer and a large language model.

[0037] The visual encoding layer can be used to generate image features corresponding to an input image. For example, the visual encoding layer can perform feature extraction on the input image to extract image features corresponding to the image.

[0038] In practical applications, the visual encoding layer can specifically be a vision encoder.

[0039] The large language model can be used to generate corresponding response text based on the image features corresponding to the input image and the input query text output by the visual encoding layer. For example, the large language model can perform embedding processing on the input query text to generate text embedding features corresponding to the query text, and then perform inference based on the image features corresponding to the input image and the generated text embedding features output by the visual encoding layer to generate response text corresponding to the image and the query text.

[0040] Please combine Figure 1 ,refer to Figure 2 , Figure 2 It is a flowchart of a training method for a multimodal large model shown in an exemplary embodiment of the present application.

[0041] like Figure 2 As shown, the training method of the above multimodal large model may include the following steps: Step 202: With the goal of aligning the image feature space of the visual coding layer with the text representation space of the large language model, the multimodal large model is adjusted to obtain an aligned multimodal large model.

[0042] In this embodiment, during the complete training process of the above-mentioned multimodal large model, the multimodal large model can first be adjusted with the goal of aligning the image feature space of the visual coding layer in the multimodal large model with the text representation space of the large language model in the multimodal large model, so as to obtain the aligned multimodal large model.

[0043] Alignment generally refers to mapping data from different modalities (e.g., images and text) into a common feature space so that they can be compared or combined in that feature space.

[0044] Specifically, related data from different modalities (for example, an image and a text describing it) should have similar representations in the feature space. This similarity isn't just about numerical proximity; more importantly, these representations must capture the semantic information behind the data. For example, a picture of a sleeping cat and a text describing a cat sleeping should be close to each other in the feature space after being processed by their respective encoders, as they express the same or similar semantic content.

[0045] In practical applications, alignment enables cross-modal information retrieval tasks, such as finding the most relevant text description from an image input, or vice versa, finding matching images from a piece of text. Furthermore, alignment also involves generation tasks, such as generating descriptive text based on a given image, or generating a corresponding image based on a text prompt. In this case, features from different modalities are not only spatially aligned but also transformable to some extent.

[0046] For the aforementioned large multimodal model, the feature space of the visual encoding layer is typically called the image feature space. The visual encoding layer can map images to its own image feature space. The feature space of the large language model is typically called the text representation space. The large language model can map text to its own text representation space. Therefore, to enable the large multimodal model to process and understand both images and text, the image feature space of the visual encoding layer and the text representation space of the large language model can be made substantially consistent, that is, aligned.

[0047] In some embodiments, achieving this alignment typically involves training a multimodal model that learns how to transform different types of input (such as images and text) into a common embedding vector space and ensures that the relative positions of these vectors in this space reflect the semantic relationship between them.

[0048] That is, in order to align the image feature space of the visual coding layer in the above-mentioned multimodal large model with the text representation space of the large language model, a number of samples (which may be called alignment samples) for training the multimodal large model can be prepared. Among them, an alignment sample may include an image, and a descriptive text related to the image; for example, an alignment sample may include a picture containing "a sleeping cat", and a text describing "a cat sleeping". After obtaining the aligned samples, the multimodal large model can be trained based on the aligned samples. During the training process, the model parameters of the multimodal large model are adjusted to minimize the loss (Loss), thereby achieving alignment between the image feature space of the visual coding layer in the multimodal large model and the text representation space of the large language model.

[0049] For example, contrastive learning can be used to construct a similarity matrix based on the image features output by the visual encoding layer and the text features output by the large language model. InfoNCE loss (also known as temperature-scaled cross-entropy loss) is then used to maximize the similarity between the image and text features of positive samples (e.g., an aligned sample containing an image and its description text) and minimize the similarity between the image and text features of negative samples (e.g., a sample consisting of an image and description text extracted from two aligned samples, where the image and description text are unrelated). In practical applications, generative training, a hybrid of contrastive learning and generative training, and other approaches can also be used to train the multimodal large language model based on the aligned samples.

[0050] By aligning the image feature space of the visual encoding layer in the above-mentioned multimodal large model with the text representation space of the large language model, the multimodal large model can be helped to establish the association relationship between the image and the descriptive text.

[0051] Step 204: Obtain pre-training samples constructed by data synthesis, and pre-train the aligned multimodal large model based on the pre-training samples to obtain a pre-trained multimodal large model; wherein the pre-training samples include medical images and query texts corresponding to various tasks in at least one task, and response texts generated based on the medical images and the query texts.

[0052] In this embodiment, after completing the alignment of the image feature space of the visual coding layer in the above-mentioned multimodal large model and the text representation space of the large language model, several pre-training samples can be obtained. These pre-training samples can be pre-training samples corresponding to at least one task.

[0053] It should be noted that the multimodal large model in this application can be used to generate answers in medical scenarios by combining medical images (e.g., medical record photos or scanned images, medical examination report photos or scanned images, medical imaging images, etc.) with related questions. However, medical images are large in size and do not contain label information, and document reports issued by different hospitals or institutions often have different format layouts. Manual annotation is very costly, so data synthesis can be used to construct the above-mentioned pre-training samples. That is, a pre-training sample can include a medical image and a query text, as well as a response text generated based on the medical image and the query text; the medical image, the query text, and the response text can correspond to a task in at least one of the above-mentioned tasks. In the pre-training sample, the medical image and the query text can serve as sample content, and the response text can serve as a sample label.

[0054] In some embodiments, in order to enable the model to learn the overall structure of the medical image while focusing on local details, pre-training samples corresponding to global-level tasks and pre-training samples corresponding to regional-level tasks can be constructed. That is, the at least one task mentioned above can include global-level tasks and / or regional-level tasks. The global-level task is the task of generating a response text based on the entire medical image (i.e., the global image) and the query text, and the regional-level task is the task of generating a response text based on the region in the medical image (i.e., the local image) and the query text.

[0055] In some embodiments, the global-level tasks may include OCR (Optical Character Recognition) tasks, summarization tasks, etc., and the regional-level tasks may include key information extraction tasks, table extraction tasks, etc.

[0056] In some embodiments, when at least one of the aforementioned tasks includes an OCR task, optical character recognition can be performed on a captured medical image, and the optical character recognition result can be determined as a response text. The medical image, the query text used to guide the large language model to perform the OCR task, and the response text can be determined as a pre-training sample corresponding to the OCR task. The optical character recognition result can include not only the text displayed on the medical image, but also relevant information about the text bounding box.

[0057] For example, in a pre-training sample corresponding to an OCR task, the query text may be "Please identify all text content and corresponding text detection boxes in the image", and the reply text may be "<|object_ref_start|>Right foot injury, swelling and pain for half a month<|object_ref_end|><|box_start|>(61,18),(183,37)<|box_end|><|object_ref_start|>Treatment advice: <|object_ref_end|><|box_start|>(228,69),(777,89)<|box_end|>……(not all shown)". Among them, "<|object_ref_start|>" and "<|object_ref_end|>" respectively indicate the start and end of the text, and "<|box_start|>" and "<|box_end|>" respectively indicate the start and end of the text bounding box; the first coordinate used to indicate the text bounding box is the coordinate of the upper left vertex of the text bounding box, and the second coordinate is the coordinate of the lower right vertex of the text bounding box.

[0058] In some embodiments, when at least one of the above tasks includes a summarization task, optical character recognition can be performed on a obtained medical image, and based on the optical character recognition result, the text displayed on the medical image can be summarized, so that the summary result can be determined as a reply text, and the medical image, the query text used to guide the large language model to perform the summarization task, and the reply text are determined as a pre-training sample corresponding to the summarization task.

[0059] For example, in a pre-training sample corresponding to a summarization task, the query text may be "Please summarize the content presented in this picture", and the response text may be "This is an emergency medical record, which contains the patient's basic information and an overview of the condition. It mainly includes: basic patient information, such as outpatient medical record number, name, gender, date of birth, ethnicity, marital status, occupation, work unit, address, etc., but this information has been processed to maintain privacy. History of drug allergies: No allergies. Consultation time and department: Shown as the Emergency Orthopedics Department of the East Hospital. Chief complaint: right foot trauma, swelling and pain for half a month. Current medical history: describes symptoms such as accidental sprain of the right foot, pain, and limited movement, and mentions that corresponding examinations and treatments have been performed in other hospitals. Past medical history: Healthy, no history of smoking. Epidemiological history: No recent history of contact with epidemic areas. Physical examination: Good plaster fixation and good blood circulation in the extremities. Auxiliary examination results: CT examination. Preliminary diagnosis: fracture of the right foot apse (multiple locations). Diagnosis and treatment advice: It is recommended to continue plaster fixation, prohibit weight bearing on the affected limb, and provide pain management recommendations. The doctor's signature section is empty."

[0060] In some embodiments, when at least one of the above tasks includes a key information extraction task, optical character recognition can be performed on a specified area in a obtained medical image, and key information extraction can be performed on the text displayed on the identified specified area, so that the key information extraction result can be determined as a reply text, and the medical image, the query text used to guide the large language model to perform the key information extraction task, and the reply text are determined as a pre-training sample corresponding to the key information extraction task.

[0061] For example, in a pre-training sample corresponding to a key information extraction task, the query text could be "What is the current medical history of the user in the image?" (the query text specifies the area in the medical image used to display text related to the current medical history), and the response text could be "I accidentally sprained my right foot, and the affected area was painful and my movements were limited. I underwent corresponding examinations and plaster fixation treatments in our hospital. Four hours later, I accidentally injured my right foot again."

[0062] In some embodiments, when at least one of the above tasks includes a table extraction task, optical character recognition can be performed on a table area in a captured medical image, and the text in the table displayed on the recognized table area can be organized into a structured text having the layout of the table, so that the structured text can be determined as a reply text, and the medical image, the query text used to guide the large language model to perform the table extraction task, and the reply text are determined as a pre-training sample corresponding to the table extraction task.

[0063] Taking a medical examination report as an example, the medical examination results in the medical examination report can be presented in the form of a table. For example, in a routine blood test report, each examination item and its results are displayed in a table with a total of 21 rows and 4 columns; in a CT examination report, the medical examination results (including "examination items," "key images," "radiological findings," and "radiological opinions") can be considered to be displayed in a table with a total of 4 rows and 1 column. In this case, the text in the table displayed in the table area identified in the table area in the routine blood test report image can be organized into structured text representing a table with 21 rows and 4 columns; the text in the table displayed in the table area identified in the table area in the CT examination report image can be organized into structured text representing a table with 4 rows and 1 column.

[0064] In practical applications, the text in the table displayed on the table area identified in the medical image can be organized into text in HTML (HyperText Markup Language) format. Although HTML is a standard markup language for creating web pages, HTML can also be used to define the structure and content of text through various tags (for example, title, <h1> arrive< / h1> <h6>),paragraph( ), list (unordered list , ordered list and list items ) to conveniently represent structured text with a specific layout. Specifically, use Tags and their related elements (such as lines , cell ), HTML can create complex table layouts.

[0065] For example, in a pre-training sample corresponding to a table extraction task, the query text can be "Please identify all the table contents in the image" and the response text can be " Serial number Report code Project Name result Reference interval unit Inspection instruments Inspection methods 1 PT Prothrombin time (Stago) 11.60 11--14 Second STAGO-2 Solidification method 2 PTA Prothrombin activity 122.00 84--128 % STAGO-2 Solidification method 3 PTR Plasma prothrombin time ratio 0.92↓ 0.94--1.24 STAGO-2 Solidification method 4 INR Prothrombin international normalized ratio 0.89↓ 0.94--1.30 STAGO-2 Solidification method 5 APTT Activated partial thromboplastin time (Stago) 33.60 28--43.5 Second STAGO-2 Solidification method 6 APPTTR activated partial thromboplastin ratio 0.99 STAGO-2 Solidification method 7 TT Thrombin time 19.20 14--21 Second STAGO-2 Solidification method 8 TTR Thrombin time>ratio 1.13 STAGO-2 Solidification method 9 FIB Fibrinogen content 3.94 2.00--4.00 g / L STAGO-2 Solidification method 10 DD D-dimer measurement 0.62 0--1 mg / L CS5100-2 Immunoassay>Turbidimetry 11 FDP Fibrinogen degradation products 1.40 0--5 mg / L CS5100-2 Immunoturbidimetry ”.

[0066] In this embodiment, after obtaining the above-mentioned pre-training samples, the aligned above-mentioned multimodal large model can be pre-trained based on the pre-training samples. During the pre-training process, the model parameters of the multimodal large model are adjusted so that the loss is reduced as much as possible to complete the pre-training of the multimodal large model.

[0067] By pre-training the above-mentioned multimodal large model on various types of medical service tasks, the visual encoding layer in the multimodal large model can fully understand the characteristics and layout of medical images, and the large language model in the multimodal large model can have certain medical expertise and reasoning and summarizing capabilities.

[0068] Step 206: Obtain fine-tuning samples obtained by manual annotation, and fine-tune the pre-trained multimodal large model based on the fine-tuning samples to obtain a multimodal large model for performing the target task; wherein the fine-tuning samples include medical images and query text corresponding to the target task, and response text annotated for the medical images and the query text.

[0069] In this embodiment, after pre-training the multimodal large model, several fine-tuning samples can be obtained. These fine-tuning samples can be fine-tuning samples corresponding to a specific task (referred to as a target task). The target task can be selected based on actual needs.

[0070] It should be noted that the purpose of fine-tuning the aforementioned large multimodal model is to enable the model to learn to follow user instructions. Therefore, the aforementioned fine-tuning samples can be obtained through manual annotation. That is, a fine-tuning sample can include a medical image and a query text corresponding to the aforementioned target task, as well as annotated response text for the medical image and the query text. In this case, the annotated response text can be determined by an annotator or an annotation service based on historical data, taking the medical image and the query text as the basis.

[0071] After obtaining the above-mentioned fine-tuning samples, the pre-trained multimodal large model can be fine-tuned based on the fine-tuning samples. During the fine-tuning process, the model parameters of the multimodal large model are adjusted to reduce the loss as much as possible to complete the fine-tuning of the multimodal large model.

[0072] The essence of fine-tuning the above-mentioned multimodal large model for specific types of medical service tasks is to fine-tune the instructions based on a small amount of high-quality specific task data labeled according to actual medical scenarios, so that the model output is in a format that meets the requirements of medical services.

[0073] After completing the fine-tuning of the multimodal model, the multimodal model is considered fully trained. At this point, the fully trained multimodal model can be used to perform the target tasks described above.

[0074] exist Figure 1 Based on the Figure 3 , Figure 3 It is a schematic diagram of another multimodal large model shown in an exemplary embodiment of the present application.

[0075] like Figure 3 As shown in FIG, in addition to the visual encoding layer and the large language model, an additional alignment layer can be introduced in the above multimodal large model. The alignment layer can be a linear transformation layer or a more complex adapter.

[0076] The visual encoding layer can be used to generate image features corresponding to the input image. The alignment layer can be used to map the image features in the image feature space of the visual encoding layer to the text representation space of the large language model. The large language model can be used to generate corresponding response text based on the mapped features corresponding to the input image output by the alignment layer and the input query text.

[0077] That is, the alignment layer can be used to map the image features output by the visual encoding layer into the text representation space of the large language model to generate alignment features corresponding to the image features, and the large language model can be used to generate response text based on the alignment features and the query text.

[0078] It should be noted that the alignment layer provides greater flexibility. It allows for efficient mapping of image features in the image feature space of the visual encoding layer to the text representation space of the large language model without significantly changing the structure or parameters of the visual encoding layer and / or the large language model itself.

[0079] Please combine Figure 3 ,refer to Figure 4 , Figure 4 It is a schematic diagram of a training process of a multimodal large model shown in an exemplary embodiment of the present application.

[0080] like Figure 4 As shown, when the above-mentioned multimodal large model is trained based on the above-mentioned aligned samples (which can be regarded as the first stage of training), the model parameters of the large language model in the multimodal large model can be frozen, that is, the model parameters of the large language model are not adjusted during the training process, and the multimodal large model is trained based on the aligned samples, so as to adjust the model parameters of the visual coding layer and the alignment layer in the multimodal large model during the training process, so that the loss is reduced as much as possible, thereby achieving alignment of the image feature space of the visual coding layer in the multimodal large model with the text representation space of the large language model.

[0081] During the pre-training of the aligned large multimodal model based on the pre-training samples (which can be regarded as the second stage of training), the model parameters of the visual encoding layer, alignment layer and large language model in the large multimodal model can be adjusted to minimize the loss as much as possible to complete the pre-training of the large multimodal model.

[0082] When fine-tuning the pre-trained multimodal large model based on the fine-tuning samples (which can be regarded as the third stage of training), the model parameters of the visual coding layer in the multimodal large model can be frozen, that is, the model parameters of the visual coding layer are not adjusted during the fine-tuning process, and the multimodal large model is fine-tuned based on the fine-tuning samples, so as to adjust the model parameters of the large language model and the alignment layer in the multimodal large model during the fine-tuning process, so as to reduce the loss as much as possible to complete the fine-tuning of the multimodal large model.

[0083] During the first phase of training, the goal is to align the image feature space of the visual encoding layer with the text representation space of the pre-trained large language model. The main reasons for freezing the model parameters of the large language model at this time are as follows: Maintaining language understanding capabilities: Large language models have been fully trained on large-scale text datasets and have strong language understanding and generation capabilities. If their parameters are allowed to update at this stage, these existing capabilities may be destroyed, especially in the absence of sufficient relevant image-text pairs to support such cross-modal adjustments.

[0084] Focus on initial alignment of vision and language: This stage focuses on learning how to effectively map image features from the image feature space of the visual encoding layer to the text representation space of the large language model. By fixing the model parameters of the large language model, resources can be concentrated on optimizing the visual encoding layer and alignment layer, so that the image features extracted from the image can be accurately mapped to the text representation space of the large language model.

[0085] After the first and second phases of training, the model has already developed strong image understanding capabilities. In the third phase of training, the main task is to further fine-tune the alignment layer and the large language model layer to enable the model to better respond to user commands. The main reasons for freezing the model parameters of the visual encoding layer at this time are as follows: Stability of image feature representation: After the first two stages of learning, the visual encoding layer has been trained to a high level of maturity and can effectively extract key features from images. Further adjustments to these parameters at this stage may cause unnecessary changes in the image feature representation, impacting model performance.

[0086] The focus shifts to task-specific adaptation: The goal at this stage is to improve the model's ability to generate responses based on user commands, which involves improving language understanding and generation capabilities. Therefore, the focus should be on adjusting the alignment layer and the language model itself to ensure that they can more accurately output the desired content based on visual input and user commands.

[0087] In the above technical solution, the multimodal large model may include a visual coding layer for generating image features corresponding to an image, and a large language model for generating a response text based on the image features and the query text; when training the multimodal large model, the image feature space of the visual coding layer and the text representation space of the large language model may be aligned first, and then the aligned multimodal large model may be pre-trained based on pre-training samples corresponding to various tasks in at least one task in a medical scenario constructed by data synthesis, and finally the pre-trained multimodal large model may be fine-tuned based on fine-tuning samples corresponding to specific tasks in the medical scenario obtained by manual annotation to obtain a multimodal large model for performing the specific task.

[0088] By adopting the above method, by aligning the image feature space of the visual coding layer in the multimodal large model with the text representation space of the large language model, the multimodal large model can be helped to establish the association relationship between the image and the descriptive text; by pre-training the multimodal large model on various types of medical service tasks, the visual coding layer in the multimodal large model can fully understand the characteristics and layout of the medical image, and the large language model in the multimodal large model can have certain medical expertise and reasoning and summarizing capabilities; by labeling a small amount of high-quality specific task data according to actual medical scenarios, and fine-tuning the multimodal large model on specific types of medical service tasks, the multimodal large model can learn to follow the user's instructions and output a format that meets the medical service requirements.

[0089] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0090] Please refer to Figure 5 , Figure 5 : is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508 and a non-volatile memory 510, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of the present application do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0091] Please refer to Figure 6 , Figure 6 It is a block diagram of a training device for a multimodal large model shown in an exemplary embodiment of the present application.

[0092] The above multimodal large model training device can be applied to Figure 5 The device shown in the figure implements the technical solution of the present application. The multimodal large model includes a visual encoding layer and a large language model; the visual encoding layer is used to generate image features corresponding to the image; and the large language model is used to generate a response text based on the image features and the query text. The device may include: An alignment module 602 adjusts the multimodal large model with the goal of aligning the image feature space of the visual coding layer with the text representation space of the large language model to obtain an aligned multimodal large model; A pre-training module 604 acquires pre-training samples constructed by data synthesis, and pre-trains the aligned multimodal large model based on the pre-training samples to obtain a pre-trained multimodal large model; wherein the pre-training samples include medical images and query text corresponding to various tasks in at least one task, and response text generated based on the medical images and the query text; The fine-tuning module 606 obtains fine-tuning samples obtained by manual annotation, and fine-tunes the pre-trained multimodal large model based on the fine-tuning samples to obtain a multimodal large model for performing the target task; wherein the fine-tuning samples include medical images and query text corresponding to the target task, and response text annotated for the medical images and the query text.

[0093] In some embodiments, adjusting the multimodal large model with the goal of aligning the image feature space of the visual coding layer with the text representation space of the large language model to obtain an aligned multimodal large model includes: Acquire alignment samples, and train the multimodal large model based on the alignment samples to align the image feature space of the visual encoding layer with the text representation space of the large language model; wherein the alignment samples include images and descriptive text related to the images.

[0094] In some embodiments, the multimodal large language model further includes an alignment layer; the alignment layer is used to map the image features into the text representation space of the large language model to generate alignment features corresponding to the image features; the large language model is used to generate a reply text based on the alignment features and the query text.

[0095] In some embodiments, the training of the multimodal large model based on the aligned samples includes: The model parameters of the large language model are frozen, and the multimodal large model is trained based on the aligned samples to adjust the model parameters of the visual encoding layer and the alignment layer.

[0096] In some embodiments, fine-tuning the pre-trained multimodal large model based on the fine-tuning sample includes: The model parameters of the visual encoding layer are frozen, and the pre-trained multimodal large model is fine-tuned based on the fine-tuning sample to adjust the model parameters of the alignment layer and the large language model.

[0097] In some embodiments, the at least one task includes a global level task and / or a regional level task; wherein the global level task is a task of generating a reply text based on the entire medical image and the query text, and the regional level task is a task of generating a reply text based on the region in the medical image and the query text.

[0098] In some embodiments, the at least one task comprises an optical character recognition task that is a global level task; The device further includes an optical character recognition task sample construction module, which is used to: Acquiring a medical image and performing optical character recognition on the medical image; An optical character recognition result is determined as a reply text, and the medical image, the query text for guiding the large language model to perform the optical character recognition task, and the reply text are determined as the pre-training samples.

[0099] In some embodiments, the at least one task further comprises a summary task belonging to a global level task; The device also includes a module for summarizing and constructing task samples, which is used to: Summarizing the text displayed on the medical image according to the optical character recognition result; The summarization result is determined as the reply text, and the medical image, the query text for guiding the large language model to perform the summarization task, and the reply text are determined as the pre-training samples.

[0100] In some embodiments, the at least one task includes a key information extraction task belonging to a regional level task; The device also includes a key information extraction task sample construction module, which is used to: Acquiring a medical image and performing optical character recognition on a designated area in the medical image; Extracting key information from the text displayed on the identified designated area; The key information extraction result is determined as the reply text, and the medical image, the query text for guiding the large language model to perform the key information extraction task, and the reply text are determined as the pre-training samples.

[0101] In some embodiments, the at least one task includes a table extraction task that is a region-level task; The device also includes a table extraction task sample construction module, which is used to: Acquiring a medical image, and performing optical character recognition on a table area in the medical image; Organizing text in the table displayed on the identified table area into structured text having the layout of the table; The structured text is determined as the reply text, and the medical image, the query text for guiding the large language model to perform the table extraction task, and the reply text are determined as the pre-training samples.

[0102] For the device embodiments, they basically correspond to the method embodiments, so for relevant details, please refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of this application.

[0103] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0104] In a typical configuration, a computer includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0105] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0106] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0107] It should be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, product, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, product, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus that includes the element.

[0108] The above description is of specific embodiments of the present application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] The terms used in one or more embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present application. The singular forms "a," "an," "the," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. The term "and / or" refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0110] The terms "one embodiment," "some embodiments," "example," "specific example," or "one implementation method" used in one or more embodiments of the present application mean that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic descriptions of these terms do not necessarily refer to the same embodiment. Moreover, the specific features or characteristics described can be combined in appropriate ways in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics of different embodiments can be combined unless they are mutually inconsistent.

[0111] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of", "when" or "in response to determining".

[0112] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0113] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. < / h6>

Claims

1. A training method for a large multimodal model; wherein: The multimodal large model includes a visual encoding layer and a large language model; the visual encoding layer is used to generate image features corresponding to the image; The large language model is used to generate a response text based on the image features and the query text; The method comprises: Adjusting the multimodal large model with the goal of aligning the image feature space of the visual encoding layer with the text representation space of the large language model to obtain an aligned multimodal large model; Obtaining pre-training samples constructed by data synthesis, and pre-training the aligned multimodal large model based on the pre-training samples to obtain a pre-trained multimodal large model; wherein the pre-training samples include medical images and query texts corresponding to various tasks in at least one task, and response texts generated based on the medical images and the query texts; Obtain fine-tuning samples obtained by manual annotation, and fine-tune the pre-trained multimodal large model based on the fine-tuning samples to obtain a multimodal large model for performing the target task; wherein the fine-tuning samples include medical images and query text corresponding to the target task, and response text annotated for the medical images and the query text.

2. The method according to claim 1, wherein adjusting the multimodal large model with the goal of aligning the image feature space of the visual encoding layer with the text representation space of the large language model to obtain an aligned multimodal large model comprises: Acquire alignment samples, and train the multimodal large model based on the alignment samples to align the image feature space of the visual encoding layer with the text representation space of the large language model; wherein the alignment samples include images and descriptive text related to the images.

3. According to the method according to claim 2, the multimodal large language model also includes an alignment layer; the alignment layer is used to map the image features into the text representation space of the large language model to generate alignment features corresponding to the image features; the large language model is used to generate a reply text based on the alignment features and the query text.

4. The method according to claim 3, wherein training the multimodal large model based on the aligned samples comprises: The model parameters of the large language model are frozen, and the multimodal large model is trained based on the aligned samples to adjust the model parameters of the visual encoding layer and the alignment layer.

5. The method according to claim 3, wherein fine-tuning the pre-trained multimodal large model based on the fine-tuning sample comprises: The model parameters of the visual encoding layer are frozen, and the pre-trained multimodal large model is fine-tuned based on the fine-tuning sample to adjust the model parameters of the alignment layer and the large language model.

6. The method according to claim 1, wherein the at least one task comprises a global level task and / or a regional level task; The global-level task is a task of generating a response text based on the entire medical image and the query text, and the regional-level task is a task of generating a response text based on the region in the medical image and the query text.

7. The method according to claim 6, wherein the at least one task comprises an optical character recognition task belonging to a global level task; The method further comprises: Acquiring a medical image and performing optical character recognition on the medical image; An optical character recognition result is determined as a reply text, and the medical image, the query text for guiding the large language model to perform the optical character recognition task, and the reply text are determined as the pre-training samples.

8. The method according to claim 7, wherein the at least one task further comprises a summary task belonging to a global level task; The method further comprises: Summarizing the text displayed on the medical image according to the optical character recognition result; The summarization result is determined as the reply text, and the medical image, the query text for guiding the large language model to perform the summarization task, and the reply text are determined as the pre-training samples.

9. The method according to claim 6, wherein the at least one task comprises a key information extraction task belonging to a regional level task; The method further comprises: Acquiring a medical image and performing optical character recognition on a designated area in the medical image; Extracting key information from the text displayed on the identified designated area; The key information extraction result is determined as the reply text, and the medical image, the query text for guiding the large language model to perform the key information extraction task, and the reply text are determined as the pre-training samples.

10. The method according to claim 6, wherein the at least one task comprises a table extraction task belonging to a regional level task; The method further comprises: Acquiring a medical image, and performing optical character recognition on a table area in the medical image; Organizing text in the table displayed on the identified table area into structured text having the layout of the table; The structured text is determined as the reply text, and the medical image, the query text for guiding the large language model to perform the table extraction task, and the reply text are determined as the pre-training samples.

11. A training device for a multimodal large model; wherein: The multimodal large model includes a visual encoding layer and a large language model; the visual encoding layer is used to generate image features corresponding to the image; The large language model is used to generate a response text based on the image features and the query text; The device comprises: an alignment module, which adjusts the multimodal large model with the goal of aligning the image feature space of the visual encoding layer with the text representation space of the large language model to obtain an aligned multimodal large model; a pre-training module that obtains pre-training samples constructed by data synthesis and pre-trains the aligned multimodal large model based on the pre-training samples to obtain a pre-trained multimodal large model; wherein the pre-training samples include medical images and query text corresponding to various tasks in at least one task, and response text generated based on the medical images and the query text; A fine-tuning module obtains fine-tuning samples obtained by manual annotation, and fine-tunes the pre-trained multimodal large model based on the fine-tuning samples to obtain a multimodal large model for performing the target task; wherein the fine-tuning samples include medical images and query text corresponding to the target task, and response text annotated for the medical images and the query text.

12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images