A fine alignment method of visual-linguistic features for large multimodal medical models

Abstract

The present invention relates to the field of artificial intelligence technology and provides a method for finely aligning visual language features for a large medical multimodal model. The method comprises: using the large artificial intelligence model to conduct deep learning and semantic understanding of the textual features of medical expert knowledge to generate question-answer pairs; extracting the visual features of medical expert knowledge and converting them into question-answer pairs; training the large medical multimodal model with the question-answer pairs; using the large artificial intelligence model to generate medical image-text question-answer pairs; introducing reinforcement learning to the large medical multimodal model and utilizing the medical image-text question-answer pairs to achieve fine alignment of visual language features. The present invention proposes a method for fully automatically generating high-quality medical multimodal instruction data, enabling the large medical multimodal model to provide accurate and reliable output content in various medical applications. The entire data generation process does not require manual labeling, saving manpower and time costs. At the same time, it reduces model reasoning illusions and improves model performance.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for fine alignment of visual language features for a large multimodal medical model. Background Art

[0002] Large multimodal models are currently attracting widespread attention from researchers due to their flexible ability to process data from various modalities and their strong generalization across a wide range of application scenarios. Medical large multimodal models, in particular, have become a hot topic of research within the vertical application of multimodal models due to their significant potential in the biomedical field. Unlike traditional, single-task expert models, medical large multimodal models can easily process both image and text inputs and flexibly output the desired results. They outperform expert models in a variety of scenarios, including diagnostic assistance and clinical decision support. However, training such models relies on massive amounts of medical data containing specialized knowledge. While the advancement of modern medicine has enriched a wealth of medical knowledge in sources such as medical research papers, medical textbooks, and hospital clinical records, obtaining training samples that accurately align text and visual data remains a significant challenge, hindering the further development of medical large multimodal models.

[0003] Current methods for constructing training data for large multimodal medical models can be broadly categorized into two types. The first involves retrieving medical images and corresponding text descriptions from medical research papers and textbooks. For example, the PMC-OA dataset extracts 1.6 million subimages and their text descriptions from a publicly accessible subset of PubMedCentral, creating rich text-image pairs of medical information. Similarly, Med-Flamingo carefully selects high-quality content from 4,721 medical textbooks, including medical images and associated textual information, and constructs this content as interleaved image-text data. While these methods are capable of generating large training datasets, they suffer from the problem of insufficiently refined mapping between images and text. Such low-quality data inevitably impacts training efficiency. The second category of methods focuses on generating instruction fine-tuning data through external model generation. For example, LLaVA-Med uses GPT-4 to extract key information from image descriptions and generate image-text pair instructions. However, the quality of this kind of image-text instructions is still essentially limited by the capabilities of the external model, and the external models used are often not specifically trained on large amounts of data containing medical knowledge.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for fine alignment of visual language features for large medical multimodal models.

[0006] The present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for fine alignment of visual language features for a large multimodal medical model, comprising:

[0008] In step 201, a publicly available medical concept text corpus is collected, and medical expert knowledge text features are extracted from the medical concept text corpus;

[0009] In step 202, a generative artificial intelligence model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and to generate a large number of high-quality medical text-based question-answer pairs;

[0010] In step 203, a plurality of publicly available medical segmentation datasets are collected, and medical expert knowledge visual features are extracted from the medical segmentation datasets; wherein the medical segmentation datasets include medical images of different modalities, and the medical images of different modalities include one or more of CT, MRI, and X-ray; the medical expert knowledge visual features include a plurality of key visual information, and the key visual information includes one or more of organ contours, lesion areas, and tissue structures;

[0011] In step 204, based on the visual features of the medical expert knowledge acquired in step 203, the features are converted into question-answer pair instructions for the medical term visual localization task;

[0012] In step 205, the question-answer pairs obtained in steps 202 and 204 are integrated and used to train the medical multimodal large model, thereby injecting medical expert knowledge in both textual and visual forms, enabling the medical multimodal large model to understand and generate medical texts and interpret medical images.

[0013] In step 206, publicly available medical image analysis cases are collected and organized, and a medical image graphic feature library is established through screening;

[0014] In step 207, a generative artificial intelligence model is used to deeply analyze and understand the medical image and text feature library obtained in step 206 to generate medical image and text question-answering instructions;

[0015] In step 208, a reinforcement learning phase is introduced for the medical multimodal large model, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, thereby enhancing the generalization ability of the medical multimodal large model when processing novel medical cases, so that it can more accurately understand and analyze complex medical image-text information.

[0016] Preferably, the extracting of medical expert knowledge text features from the medical concept text corpus specifically includes:

[0017] Preliminary identification of potential medical terms using predefined medical dictionaries and regular expression matching rules;

[0018] Using the trained named entity recognition model, we further extract and verify medical terms, and retain those medical terms {c1, c2, ... c N};

[0019] Extract the medical terms {c1, c2, ...c N} One or more text information in the corresponding term explanation, symptom description and treatment plan constitutes the medical expert knowledge text feature {(c1,d1),(c2,d2),…(c N ,d N )}.

[0020] Preferably, the generative artificial intelligence large model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text-based question-answer instructions, specifically including:

[0021] Using the medical expert knowledge text features {(c1, d1), (c2, d2), ... (c N ,d N )}, and with the help of the generative artificial intelligence big model, using predefined instruction prompts, the generative artificial intelligence big model generates textual question-answer instructions for the extracted medical concepts and their related medical expert knowledge. The extracted question-answer instructions are recorded as

[0022] Among them, the generative artificial intelligence large model generates text-based question-and-answer instructions from one or more perspectives of term explanation, symptom description and treatment plan.

[0023] Preferably, the method of using a generative artificial intelligence large model to perform deep learning and semantic understanding on the medical expert knowledge text features acquired in step 201 and generate a large number of high-quality medical text-based question-answer pairs also includes:

[0024] If a single question-answer pair instruction fails to be generated, another instruction prompt will be used. If M instruction prompts all fail to be generated, the corresponding medical concept will be skipped; M is a preset value.

[0025] Preferably, the step of converting the medical expert knowledge visual features obtained in step 203 into question-answer pair instructions for the medical term visual localization task specifically includes:

[0026] The medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into question-answer instructions for the medical terminology visual positioning task through a pre-set question-answer template The pre-set question-and-answer template includes one or more of directly asking about the location, asking about the location after describing the features, and asking about the location after providing relevant information.

[0027] In the pre-set question and answer template, the question is an inquiry about the specific location of the medical term, and the answer contains the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the coordinate of the upper left corner of the minimum rectangular box surrounding the object corresponding to the medical term in the figure, and (x2,y2) is the coordinate of the lower right corner of the minimum rectangular box surrounding the object corresponding to the medical term in the figure.

[0028] Preferably, the collection and organization of publicly available medical imaging analysis cases and the establishment of a medical imaging graphic feature library through screening specifically include:

[0029] Collect publicly available case-level medical image analysis reports to generate a medical image library, which is denoted as {I1, I2, ... I L}, where L is the total number of medical images;

[0030] In each medical image analysis report, a sample of N medical images is selected and stitched into one image; where N is a preset value, N is greater than or equal to 1 and N is less than or equal to 4;

[0031] Extract one or more text information related to image analysis, symptom analysis, and disease diagnosis from the corresponding medical imaging report to form {(I1, d1), (I2, d2), … (I N ,d N )} medical imaging graphic feature library.

[0032] Preferably, the generative artificial intelligence big model is used to deeply analyze and understand the medical image and text feature library obtained in step 206 to generate medical image and text question-answering instructions, specifically including:

[0033] Using the generative artificial intelligence model, the medical image feature library {(I1,d1),(I2,d2),…(I N ,d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I n ,n n )}, from multiple angles, using predefined instruction prompts, to achieve question-answer instruction data generation; wherein, the multiple angles include one or more of symptom description, treatment plan and prognosis evaluation.

[0034] Preferably, the generative artificial intelligence model is used to generate the medical image feature library {(I1, d1), (I2, d2), ... (I N ,d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}, specifically including:

[0035] Generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )};

[0036] And by tampering with the information, negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}; The tampering methods used include one or more of modifying the report observation content, modifying the report analysis, modifying the reasoning process, and modifying the diagnosis conclusion, wherein the probabilities of various tampering methods are equally divided to ensure the diversity of the instruction data.

[0037] Preferably, the medical multimodal large model is introduced into a reinforcement learning phase, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, specifically including:

[0038] The medical multimodal model that contains the medical expert knowledge after training in step 205 is used to perform reinforcement learning on the medical multimodal model; wherein the training data is the positive sample data {(I1, p1), (I2, p2), ... (IN ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}.

[0039] Preferably, in the process of reinforcement learning of the medical multimodal large model, the DPO algorithm is used as the loss function for training the medical multimodal large model.

[0040] In a second aspect, the present invention further provides a device for fine alignment of visual language features for a large medical multimodal model, for implementing the method for fine alignment of visual language features for a large medical multimodal model described in the first aspect, the device comprising:

[0041] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to perform the method for fine alignment of visual language features for medical multimodal large models described in the first aspect.

[0042] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0043] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in a memory to execute the method of the first aspect.

[0044] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method of the first aspect.

[0045] The present invention proposes a method for fully automatically generating high-quality medical multimodal instruction data, helping medical multimodal large models to provide more accurate and reliable output content in various medical applications. The entire process of data generation does not require manual labeling, saving a lot of manpower and time costs. At the same time, it is proposed to introduce a training strategy for the reinforcement learning stage for the medical multimodal large model with the help of the generated instruction data. Through the designed data generation method and the introduced reinforcement learning training strategy, the present invention helps the medical multimodal large model to achieve fine alignment of medical knowledge in visual and linguistic forms, reduce model reasoning hallucinations, and improve model performance. In addition, this method can use real clinical feedback data to quickly update model parameters and improve the downstream generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0047] Figure 1 This is a flowchart of a first method for fine alignment of visual language features for a large multimodal medical model, provided by an embodiment of the present invention;

[0048] Figure 2 This is a flow chart of a second method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0049] Figure 3 This is a flowchart of a third method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0050] Figure 4 4 is a flowchart of a fourth method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0051] Figure 5 Schematic diagram of a method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the architecture of a device for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that the specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0055] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "multiple" means two or more. In addition, for example, the description may also use the method of adding "A" and "B" at the end to describe the same type of nouns as two independent individuals. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the description purposes of the same type of individuals, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0056] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0057] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0058] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0059] Embodiment 1:

[0060] In response to the defects and deficiencies in the current research field, the present invention provides a method for fine alignment of visual language features for large medical multimodal models. Its purpose is to make up for the current situation in which large medical multimodal models lack high-quality image-text question-answering instructions, and to design a set of fully automatic methods for generating medical image-text question-answering instructions by leveraging publicly available medical expertise. And by introducing a reinforcement learning stage, the large medical multimodal model can quickly update model parameters based on real clinical feedback data, thereby improving the generalization ability of the model in downstream application scenarios. Example 1 of the present invention provides a method for fine alignment of visual language features for large medical multimodal models, such as Figure 1 Shown, including:

[0061] In step 201, a publicly available medical concept text corpus is collected, and medical expert knowledge text features are extracted from the medical concept text corpus; wherein the medical concept text corpus includes but is not limited to authoritative sources such as medical textbooks, clinical guidelines, and medical journal articles.

[0062] In step 202, a generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text-based question-answer pairs; these question-answer pairs cover a wide range from basic medical knowledge to complex clinical case analysis.

[0063] In step 203, a plurality of publicly available medical segmentation data sets are collected, and visual features of medical expert knowledge are extracted from the medical segmentation data sets; wherein the medical segmentation data sets include medical images of different modalities, and the medical images of different modalities include one or more of CT, MRI and X-ray; the visual features of medical expert knowledge include a plurality of key visual information, and the key visual information includes one or more of organ contours, lesion areas and tissue structures; that is, a publicly available medical segmentation database is collected, and the medical segmentation database contains images of various common medical organs in modalities such as X-ray, CT, MRI, ultrasound, and the like, as well as pixel-level fine segmentation areas corresponding to the medical organs; thereby, high-quality medical expert knowledge in a visual form can be obtained.

[0064] In step 204 , based on the visual features of the medical expert knowledge acquired in step 203 , they are converted into question-answer pair instructions for the medical term visual localization task.

[0065] In step 205, the question-answer pairs obtained in steps 202 and 204 are integrated, and the integrated question-answer pairs (i.e., including the medical text-based question-answer pairs obtained in step 202 and the question-answer pairs for the medical term visual localization task in step 204) are used to train the medical multimodal large model, thereby injecting medical expert knowledge in both textual and visual forms, enabling the medical multimodal large model to understand and generate high-quality medical text and accurately interpret medical images. That is, the medical concept question-answer pair data generated in both textual and visual forms in steps 202 and 204 is used to train the medical multimodal large model, injecting medical expert knowledge from human professional physicians into the model. This data can serve as a supplement to existing medical multimodal training data, providing more refined medical expert knowledge. The method described in this embodiment does not rely on a specific medical multimodal large model and can be easily applied to any open source model. That is, the medical multimodal large model can be any open source multimodal large model.

[0066] In step 206, publicly available medical image analysis cases are collected and organized, and through screening, a medical image graphic feature library (also called a high-quality medical graphic database) is established; that is, publicly available information on medical image analysis reports is collected. This type of data is mostly in the form of case studies, and professional physicians analyze and diagnose real clinical images. There is a strong correspondence between medical images and reports, which is suitable for training large medical multimodal models.

[0067] In step 207, a generative artificial intelligence big model is used to deeply analyze and understand the medical image and text feature library obtained in step 206, and generate a large number of high-quality medical image and text question-and-answer instructions covering everything from basic image recognition to complex case analysis.

[0068] In step 208, a reinforcement learning phase is introduced for the medical multimodal large model, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, thereby enhancing the generalization ability of the medical multimodal large model when processing novel medical cases, so that it can more accurately understand and analyze complex medical image-text information.

[0069] This example proposes a method for fully automatically generating high-quality medical multimodal instruction data, enabling large medical multimodal models to provide more accurate and reliable output content for various medical applications. The entire data generation process eliminates the need for manual annotation, saving significant manpower and time. Furthermore, a training strategy for the large medical multimodal model, leveraging the generated instruction data, is proposed, introducing a reinforcement learning phase.

[0070] This example aims to help large multimodal medical models achieve precise alignment of visual and verbal medical knowledge through a designed data generation method and an introduced reinforcement learning training strategy, reducing model inference hallucinations and improving model performance. Furthermore, this method can leverage real clinical feedback data to rapidly update model parameters and enhance the model's downstream generalization capabilities.

[0071] In an optional embodiment, the medical expert knowledge text features are extracted from the medical concept text corpus, such as Figure 2 As shown, specifically including:

[0072] In step 301, potential medical terms are initially identified using predefined medical dictionaries and regular expression matching rules.

[0073] In step 302, the trained named entity recognition model is used to further extract and verify medical terms, and the medical terms {c1, c2, ... c N}.

[0074] In step 303, extract the medical terms {c1, c2, ... c N} One or more text information in the corresponding term explanation, symptom description and treatment plan constitutes the medical expert knowledge text feature {(c1,d1),(c2,d2),…(c N ,d N First, we count frequently occurring medical concepts; then, we obtain relevant information about these concepts, such as terminology explanations, symptom descriptions, and treatment plans, from authoritative medical encyclopedia websites, knowledge bases, and other data sources. This allows us to obtain a large number of medical concepts and their related rich medical expert knowledge.

[0075] In actual application scenarios, the generative artificial intelligence model is used to conduct deep learning and semantic understanding of the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text-based question-answer instructions, specifically including:

[0076] Using the medical expert knowledge text features {(c1, d1), (c2, d2), ... (c N ,d N )}, and with the help of the generative artificial intelligence big model, using predefined instruction prompts, the generative artificial intelligence big model generates textual question-answer instructions for the extracted medical concepts and their related medical expert knowledge. The extracted question-answer instructions are recorded as

[0077] Among them, the generative artificial intelligence large model generates text-based question-and-answer instructions from one or more perspectives of term explanation, symptom description and treatment plan.

[0078] In actual use, the generative artificial intelligence model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and to generate a large number of high-quality medical text-based question-answer pairs, and also includes:

[0079] If a single generation of a question-and-answer pair fails, another instruction prompt is used instead. If all M instruction prompts fail to be generated, the corresponding medical concept is skipped. M is a preset value, and M is obtained by those skilled in the art based on empirical analysis. In an optional embodiment, M is 3. That is, with the help of a generative artificial intelligence large model, it uses predefined instruction prompts from the perspectives of term explanation, symptom description, treatment plan, etc. to generate textual question-and-answer pair instructions for the extracted medical concepts and their related rich medical expert knowledge. Under ideal conditions, the large model can generate the selected type of question-and-answer pair instructions, but for some medical concepts, there is a situation where medical knowledge is defaulted. If a single generation fails, another instruction prompt is used instead. If all three instruction prompts fail to be generated, the medical concept is skipped. The predefined instruction prompt is obtained by those skilled in the art based on empirical analysis. In an optional embodiment, a predefined instruction prompt is as follows:

[0080] "Based on the provided medical concepts and corresponding text materials, please generate text question-answer pairs as required. Questions should be considered from the perspectives of term explanation / symptom description / treatment plan. The generated content should maintain the accuracy of medical knowledge and be useful for auxiliary diagnosis. The output should conform to the following format:

[0081] Question: [Text Q&A question about the instruction]

[0082] Answer: [Text Q&A answer to the instruction]

[0083] ”

[0084] In some embodiments, converting the medical expert knowledge visual features obtained in step 203 into question-answer instructions for the medical term visual localization task specifically includes:

[0085] The medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into question-answer instructions for the medical terminology visual positioning task through a pre-set question-answer template Among them, the pre-set question and answer template includes one or more of directly asking about the location, asking about the location after describing the characteristics, and asking about the location after giving relevant information; the pre-set question and answer template is obtained by technical personnel in this field based on experience analysis.

[0086] In the pre-set question and answer template, the question is an inquiry about the specific location of the medical term, and the answer contains the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) are the coordinates of the upper left corner of the smallest rectangular box enclosing the object corresponding to the medical term in the image, and (x2,y2) are the coordinates of the lower right corner of the smallest rectangular box enclosing the object corresponding to the medical term in the image. That is, the medical organ obtained in step 203 and the corresponding pixel-level fine segmentation area are converted to form instruction data in the form of question and answer; the instruction question is to inquire about the coordinate position of the medical organ, and the instruction answer is the coordinates of the upper left and lower right points of the smallest rectangular box containing the medical organ.

[0087] In an optional embodiment, the medical image analysis cases that are publicly available are collected and sorted, and a medical image graphic feature library is established through screening, such as Figure 3 As shown, specifically including:

[0088] In step 401, publicly available case-level medical image analysis reports are collected to generate a medical image library, which is denoted as {I1, I2, ... I L}, where L is the total number of medical images.

[0089] In step 402, in each medical image analysis report, samples containing N medical images are selected and spliced ​​into one image; wherein N is a preset value obtained by those skilled in the art based on empirical analysis, and N is greater than or equal to 1 and N is less than or equal to 4.

[0090] In step 403, one or more text information of corresponding image analysis, symptom analysis, and disease diagnosis is extracted from the corresponding medical image report to form {(I1, d1), (I2, d2), ... (I N ,d N )} medical imaging graphic feature library.

[0091] In actual application scenarios, the generative artificial intelligence model is used to deeply analyze and understand the medical image and text feature library obtained in step 206 to generate medical image and text question-answering instructions, specifically including:

[0092] Using the generative artificial intelligence model, the medical image feature library {(I1,d1),(I2,d2),…(I N ,dN )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}, using predefined instruction prompts from multiple perspectives to achieve question-answer instruction data generation; wherein the multiple perspectives include one or more of symptom description, treatment plan, and prognosis assessment. The predefined instruction prompts are obtained by those skilled in the art based on demand analysis. A predefined instruction prompt for generating positive sample data can be:

[0093] Based on the provided medical images and corresponding analysis reports, please generate text question-and-answer pairs as required. Questions should be considered from multiple perspectives, such as symptom description, treatment plan, and prognosis assessment. The generated content should maintain the accuracy of medical knowledge and be useful for auxiliary diagnosis. The output should conform to the following format:

[0094] Question: [Text Q&A question about the instruction]

[0095] Answer: [Text Q&A answer to the instruction]

[0096] ”

[0097] Wherein, the generative artificial intelligence big model is used to generate the medical image feature library {(I1,d1),(I2,d2),…(I N ,d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I n ,n n )},like Figure 4 As shown, specifically including:

[0098] In step 501, the positive sample data (I1, p1), (I2, p2), ... (I N ,p N )}.

[0099] In step 502, by tampering with the information, negative sample data {(I1,n1),(I2,n2),…(I N ,n NThe tampering methods employed include modifying one or more of the following: report observations, report analysis, reasoning, and diagnostic conclusions. The probability of each tampering method is evenly distributed to ensure the diversity of instruction data. Specifically, a large generative model is used to generate question-and-answer instruction data from real clinical images and analysis reports, using predefined instruction prompts from perspectives such as symptom descriptions and treatment plans. To support the subsequent reinforcement learning phase, this process generates both positive data transformed from the original information and negative data containing erroneous information by tampering with the information. Tampering methods include modifying report observations, report analysis, reasoning, and diagnostic conclusions. Under ideal conditions, the large generative model can generate the selected type of question-and-answer instruction. However, in some clinical image examples, medical knowledge may be missing. If a single generation fails, another instruction prompt is used. If all three instruction prompts fail, the medical concept is skipped.

[0100] In some embodiments, the medical multimodal large model introduces a reinforcement learning phase and utilizes the medical image-text question-answer pair instructions obtained in step 207 to achieve fine alignment of visual language features, specifically including:

[0101] The medical multimodal model that contains the medical expert knowledge after training in step 205 is used to perform reinforcement learning on the medical multimodal model; wherein the training data is the positive sample data {(I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}. In addition, during the reinforcement learning process of the medical multimodal large model, the DPO algorithm is used as the loss function for training the medical multimodal large model. That is, a reinforcement learning stage is introduced into the medical multimodal large model, and the positive and negative sample pairs converted in step 207 are used for training. This process can achieve fine alignment of the visual and linguistic features of medical concepts and enhance the generalization ability of the medical multimodal large model. At the same time, this step can quickly update the model parameters based on real clinical feedback data, thereby improving the generalization ability of the model in downstream scenarios.

[0102] This paper proposes an innovative and efficient medical concept alignment method. This method enhances the accuracy of textual and visual medical concept alignment, thereby helping large-scale medical multimodal models provide more accurate and reliable output in various medical applications, such as disease diagnosis and treatment decision-making. Building on the current training paradigm for large-scale medical multimodal models, this method integrates expert knowledge from textual corpora and visual segmentation databases of medical concepts into the instruction fine-tuning process. This enables the model to effectively learn both textual and visual features of medical concepts. To prepare textual medical concept knowledge, common medical terms are first selected and their relevant information is searched in databases such as the UMLS. To prepare visual medical concept knowledge, the medical segmentation dataset is converted into a bounding box visual localization data format to match the output format of current large-scale medical multimodal models. Despite integrating textual and visual expert knowledge, ensuring accurate alignment between these knowledge forms remains a key challenge. To this end, this paper also designs the use of real clinical medical images and their corresponding diagnostic analyses to construct positive and negative sample training data in the form of medical image-text pairs, and introduces a reinforcement learning phase for the large-scale medical multimodal model. By using the constructed positive and negative samples to train the training data in the reinforcement learning stage, the model can be helped to align medical knowledge in text and image forms, thereby improving the model's generalization ability.

[0103] Example 2:

[0104] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0105] The method for fine alignment of visual language features for a large multimodal medical model provided in this embodiment is as follows: Figure 5 As shown, the following steps are included:

[0106] Step 1: Collect a wide range of publicly available medical concept text corpora, including but not limited to authoritative sources such as medical textbooks, clinical guidelines, and medical journal articles. Then, extract text features of medical expert knowledge from these corpora.

[0107] In step one, a medical concept database for training a large medical multimodal model was collected. These medical concepts were derived from publicly available academic datasets. Medical terms were extracted using natural language processing technology and compared with authoritative medical corpora. Terms that appeared in the corpus were retained to form a set of medical concepts to be processed. Specifically, a strategy combining rule-based methods and machine learning methods was adopted. First, potential medical terms were preliminarily identified using predefined medical dictionaries and regular expression matching rules. Then, a trained named entity recognition (NER) model was used to further extract and verify medical terms. Subsequently, text information such as term explanations, symptom descriptions, and treatment plans corresponding to these medical concepts were extracted from the authoritative medical corpus. Terms whose frequency of appearance in the corpus exceeded a preset threshold were retained to form a medical concept text feature library. This method ensures that the selected medical concepts are sufficiently representative and authoritative. In order to collect medical concepts that are beneficial to model training, the source of medical concepts is publicly available academic datasets. Medical terms are extracted from them through natural language processing technology and compared with authoritative medical corpora. Terms that appear in the corpus are retained. This set is the medical concepts to be processed later and is denoted as {c1, c2, ... c N}, where N is the total number of medical concepts. Then, the text information such as term explanation, symptom description, treatment plan, etc. corresponding to the medical concepts are extracted from the authoritative medical corpus to form {(c1,d1),(c2,d2),…(c N ,d N )} medical concept text feature library (i.e., medical expert knowledge text features).

[0108] This medical concept text feature library not only contains medical concepts that appear frequently in daily medical work, but also includes text information such as term explanations, symptom descriptions, treatment plans, etc. corresponding to these concepts. This provides a powerful supplement to the existing coarse-grained image text description medical multimodal training data.

[0109] Step 2: Leveraging advanced generative AI models, we conduct deep learning and semantic understanding of the textual features of the medical expert knowledge acquired in Step 1, and automatically generate a large number of high-quality medical text-based question-answer pairs. These question-answer pairs cover a wide range of topics, from basic medical knowledge to complex clinical case analysis.

[0110] like Figure 5As shown in the second stage of the , this step constructs medical text-based question-answer pairs. Generating these question-answer pairs requires the medical concept text feature library from the previous step and leverages a generative AI model. This model uses predefined prompts from perspectives such as term explanations, symptom descriptions, and treatment plans to generate text-based question-answer pairs based on extracted medical concepts and their associated rich medical expert knowledge. A sample prompt is as follows:

[0111] "Based on the provided medical concepts and corresponding text materials, please generate text question-answer pairs as required. Questions should be considered from the perspectives of term explanation / symptom description / treatment plan. The generated content should maintain the accuracy of medical knowledge and be useful for auxiliary diagnosis. The output should conform to the following format:

[0112] Question: [Text Q&A question about the instruction]

[0113] Answer: [Text Q&A answer to the instruction]

[0114] ”

[0115] Using prompts similar to the reference instructions above enables the generative AI model to generate high-quality medical text-based question-answer pairs for the provided medical concepts. These generated instructions will be used in subsequent training steps. This approach not only generates a large amount of training data, but also ensures that this data covers a wide range of medical knowledge, providing comprehensive learning material for the model.

[0116] The generation of medical text question-answering instructions requires the previous step {(c1,d1),(c2,d2),…(c N ,d N )}, and with the help of a generative artificial intelligence model, it uses predefined instruction prompts to generate text-based question-answer pairs for the extracted medical concepts and their related rich medical expert knowledge from the perspectives of noun explanation, symptom description, and treatment plan. The extracted question-answer pairs are recorded as

[0117] Step 3: Collect a variety of publicly available medical segmentation datasets, including but not limited to medical images of different modalities such as CT, MRI, and X-ray, and extract medical expert knowledge of key visual information such as organ contours, lesion areas, and tissue structure.

[0118] like Figure 5As shown in the third part of the stage, step three is responsible for collecting medical expert knowledge in visual form. In order to collect comprehensive visual features of medical concepts, the organs, lesion types in the image annotations, and the corresponding pixel-level segmentation region coordinates in the image can be extracted from the publicly available medical segmentation dataset. Including but not limited to different modalities such as CT, MRI, X-ray, PET, and covering various systems and parts of the human body, including but not limited to the cardiovascular system, respiratory system, digestive system, nervous system, etc. The detailed segmentation region coordinates can help the model accurately understand the regional level features in the image, so as to better identify and locate various normal anatomical structures and pathological changes. The collected medical concepts and corresponding segmentation coordinates are marked as {(v1,r1),(v2,r2),…(v M ,r M )}, where M is the total number of medical concepts involved in this step.

[0119] Step 4: Based on the visual features of medical expert knowledge obtained in step 3, convert them into question-answering instructions for the medical term visual localization task.

[0120] like Figure 5 As shown in the fourth stage of the paper, the main function of step 4 is to generate question-answering instructions in the format of medical term visual localization task. In this step, the medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into medical terminology visual positioning instruction data (also called question-answer pair instructions for medical terminology visual positioning tasks) through a pre-set question-answer template. The converted question-answer pair is recorded as To enhance the diversity of instructions, the pre-set templates should be as diverse as possible, including but not limited to asking about the location directly, asking about the location after describing the characteristics, asking about the location after providing relevant information, etc. In the question-and-answer template, the questions are all about asking about the specific location of the medical term, and the answers contain the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the coordinate of the upper left corner of the minimum rectangular box surrounding the object corresponding to the medical term in the figure, and (x2,y2) is the coordinate of the lower right corner of the minimum rectangular box surrounding the object corresponding to the medical term in the figure.

[0121] The generated question-answer pair instructions in the visual localization task format for medical concepts will be used in subsequent training steps to provide the model with accurate visual-language correspondence learning materials.

[0122] Step 5: Integrate the question-answer pairs obtained in steps 2 and 4 for training a large medical multimodal model to effectively inject medical expert knowledge in both textual and visual forms, enabling the model to understand and generate high-quality medical texts and accurately interpret medical images.

[0123] like Figure 5 As shown in the fifth stage of the process, the main function of step five is to use the medical concept question answering instruction data in text form and visual form generated by steps two and four. and The process of training a large multimodal medical model. This process injects medical expert knowledge from human physicians into the model. This data can supplement existing medical multimodal training data, providing more refined medical expert knowledge.

[0124] Step 6: Collect and organize a large number of publicly available medical imaging analysis cases, and establish a high-quality medical image and text database through screening.

[0125] like Figure 5 As shown in the sixth stage of the process, the main purpose of step six is ​​to collect medical knowledge data with aligned text and image features. In this step, it is necessary to collect publicly available case-level medical image analysis reports. The medical image library is denoted as {I1,I2,…I L}, where L is the total number of medical images. In the available case-level medical image analysis reports, the number of medical images can often be as many as ten or more. To facilitate the processing of the current medical multimodal large model, in this step, only samples containing one to four medical images are selected and spliced ​​into a single image. Subsequently, the corresponding text information such as image analysis, symptom analysis, and disease diagnosis is extracted from the corresponding medical report to form {(I1, d1), (I2, d2), … (I N ,d N )} Medical image text feature library. This type of database contains one or more medical images, along with comprehensive textual information such as image analysis, symptom analysis, disease diagnosis, treatment plan, and prognosis assessment. This medical image text feature library achieves precise alignment of text and image features of medical knowledge, significantly improving the ability of large multimodal medical models to read and understand textual information.

[0126] Step 7: Use the generative artificial intelligence model to generate the medical image and text database {(I1,d1),(I2,d2),…(I N ,d N )} conducts in-depth analysis and understanding, and automatically generates a large number of high-quality medical image-text question-and-answer instructions covering everything from basic image recognition to complex case analysis.

[0127] like Figure 5 As shown in the seventh stage of the process, in step seven, the generative artificial intelligence model is used to analyze and process the medical image feature library obtained in the previous step. From the perspectives of symptom description, treatment plan prognosis evaluation, etc., predefined instruction prompts are used to generate question-answer instruction data. To support the subsequent reinforcement learning stage, this process generates positive sample data converted from the original information, denoted as {(I1, p1), (I2, p2), ... (I N ,p N )}; and by tampering with the information, negative sample data containing wrong information is generated, which is recorded as {(I1,n1),(I2,n2),…(I N ,n N The tampering methods used include modifying the report observation content, modifying the report analysis, modifying the reasoning process, and modifying the diagnostic conclusion. The probability of each tampering method is evenly distributed to ensure the diversity of the instruction data. A reference instruction for generating training positive samples is as follows:

[0128] Based on the provided medical images and corresponding analysis reports, please generate text question-and-answer pairs as required. Questions should be considered from multiple perspectives, such as symptom description, treatment plan, and prognosis assessment. The generated content should maintain the accuracy of medical knowledge and be useful for auxiliary diagnosis. The output should conform to the following format:

[0129] Question: [Text Q&A question about the instruction]

[0130] Answer: [Text Q&A answer to the instruction]

[0131] ”

[0132] Using instruction prompts similar to the above-mentioned reference instructions can enable the generative artificial intelligence model to generate high-quality image-text question-answer instruction data for the medical imaging image-text feature library. The generated instructions will be used in the training of subsequent steps.

[0133] To generate negative training samples, as previously described, we modify some information in the medical image feature library and then use the aforementioned reference instructions to generate negative training samples for the image-text question-answering data. When generating negative samples, we must carefully control the difficulty and plausibility of the error information to ensure that the model learns meaningful error patterns.

[0134] Step 8: Introducing a reinforcement learning phase into the large medical multimodal model and leveraging the medical image-text question-answering instructions obtained in Step 7 to achieve fine alignment of visual language features, enhances the model's generalization ability when processing novel medical cases, and enables it to more accurately understand and analyze complex medical image-text information.

[0135] like Figure 5As shown in the eighth stage of the paper, the main function of this step is to introduce the reinforcement learning training stage for the medical multimodal large model. This stage uses the model trained in step five, which contains a lot of medical expert knowledge. The training data is the positive and negative sample pair instruction data extracted in step seven, namely {(I1, p1), (I2, p2), ... (I N ,p N )} and {(I1,n1),(I2,n2),…(I N ,n N )}. During this training process, the model will prefer to output the high-quality medical knowledge contained in the training positive samples {(I1,p1),(I2,p2),…(I1,p1)}, thereby helping to achieve fine alignment of the visual and linguistic features of medical concepts and enhancing the generalization ability of large medical multimodal models. In order to implement the constraints on the model's preference output, the DPO algorithm is used as the loss function for model training. At the same time, the training data for this step can be derived from real clinical feedback data. With the help of this training strategy, the model parameters can be quickly updated, thereby getting rid of the tedious process of using large-scale medical data to train the model from scratch when updating the model training data. This training paradigm significantly improves the generalization ability and application value of the model in downstream scenarios.

[0136] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the invention to be protected. For example, the model can be specialized trained according to the needs of different medical specialties; more data enhancement technologies can be introduced to improve the robustness of the model; and the method can be explored in combination with other advanced artificial intelligence technologies, such as meta-learning, multi-agent reinforcement learning technology, etc., to further enhance the performance and adaptability of the model. The scope of protection claimed in the present invention is defined by the attached claims and their equivalents.

[0137] Example 3:

[0138] like Figure 6 , is a schematic diagram of the architecture of a device for fine alignment of visual language features for a large medical multimodal model according to an embodiment of the present invention. The device for fine alignment of visual language features for a large medical multimodal model according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0139] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0140] Memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for fine alignment of visual language features for a large medical multimodal model in Example 1. Processor 21 executes the method for fine alignment of visual language features for a large medical multimodal model by running the non-volatile software program and instructions stored in memory 22.

[0141] The memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and such remote memory may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The program instructions / modules are stored in the memory 22 and, when executed by the one or more processors 21 , perform the visual language feature fine alignment method for the medical multimodal large model in the above-mentioned embodiment 1.

[0143] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific content can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk or an optical disk, etc.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for fine alignment of visual language features for large multimodal medical models, characterized by: include: In step 201, a publicly available medical concept text corpus is collected, and medical expert knowledge text features are extracted from the medical concept text corpus; In step 202, a generative artificial intelligence model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and to generate a large number of high-quality medical text-based question-answer pairs; In step 203, a plurality of publicly available medical segmentation datasets are collected, and medical expert knowledge visual features are extracted from the medical segmentation datasets; wherein the medical segmentation datasets include medical images of different modalities, and the medical images of different modalities include one or more of CT, MRI, and X-ray; the medical expert knowledge visual features include a plurality of key visual information, and the key visual information includes one or more of organ contours, lesion areas, and tissue structures; In step 204, based on the visual features of the medical expert knowledge acquired in step 203, the features are converted into question-answer pair instructions for the medical term visual localization task; In step 205, the question-answer pairs obtained in steps 202 and 204 are integrated and used to train the medical multimodal large model, thereby injecting medical expert knowledge in both textual and visual forms, enabling the medical multimodal large model to understand and generate medical texts and interpret medical images. In step 206, publicly available medical image analysis cases are collected and organized, and a medical image graphic feature library is established through screening; In step 207, a generative artificial intelligence model is used to deeply analyze and understand the medical image and text feature library obtained in step 206 to generate medical image and text question-answering instructions; In step 208, a reinforcement learning phase is introduced for the medical multimodal large model, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, thereby enhancing the generalization ability of the medical multimodal large model when processing novel medical cases, so that it can more accurately understand and analyze complex medical image-text information.

2. The method for fine alignment of visual language features for a large multimodal medical model according to claim 1 is characterized in that: The extraction of medical expert knowledge text features from the medical concept text corpus specifically includes: Preliminary identification of potential medical terms using predefined medical dictionaries and regular expression matching rules; Using the trained named entity recognition model, we further extract and verify medical terms, and retain those medical terms {c1, c2, ... c N }; Extract medical terms {c1, c2, ...c N } One or more text information in the corresponding term explanation, symptom description and treatment plan constitutes the medical expert knowledge text feature {(c1,d1),(c2,d2),…(c N ,d N )}.

3. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1 is characterized in that: The generative artificial intelligence model is used to conduct deep learning and semantic understanding of the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text-based question-answer instructions, specifically including: Using the medical expert knowledge text features {(c1, d1), (c2, d2), ... (c N ,d N )}, and with the help of the generative artificial intelligence big model, using predefined instruction prompts, the generative artificial intelligence big model generates textual question-answer instructions for the extracted medical concepts and their related medical expert knowledge. The extracted question-answer instructions are recorded as Among them, the generative artificial intelligence large model generates text-based question-and-answer instructions from one or more perspectives of term explanation, symptom description and treatment plan.

4. The method for fine alignment of visual language features for a large multimodal medical model according to claim 2 is characterized in that: The method of using a generative artificial intelligence large model to perform deep learning and semantic understanding on the medical expert knowledge text features acquired in step 201 and generate a large number of high-quality medical text-based question-answer pairs also includes: If a single question-answer pair instruction fails to be generated, another instruction prompt will be used. If M instruction prompts all fail to be generated, the corresponding medical concept will be skipped; M is a preset value.

5. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1 is characterized in that: The step of converting the medical expert knowledge visual features obtained in step 203 into question-answer instructions for the medical term visual localization task specifically includes: The medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into question-answer instructions for the medical terminology visual positioning task through a pre-set question-answer template The pre-set question-and-answer template includes one or more of directly asking about the location, asking about the location after describing the features, and asking about the location after providing relevant information. In the pre-set question and answer template, the question is an inquiry about the specific location of the medical term, and the answer contains the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the coordinate of the upper left corner of the minimum rectangular box surrounding the object corresponding to the medical term in the figure, and (x2,y2) is the coordinate of the lower right corner of the minimum rectangular box surrounding the object corresponding to the medical term in the figure.

6. The method for fine alignment of visual language features for a large multimodal medical model according to claim 1 is characterized in that: The collection and organization of publicly available medical imaging analysis cases, through screening, to establish a medical imaging graphic feature library, specifically includes: Collect publicly available case-level medical image analysis reports to generate a medical image library, which is denoted as {I1, I2, ... I L }, where L is the total number of medical images; In each medical image analysis report, a sample of N medical images is selected and stitched into one image; where N is a preset value; Extract one or more text information related to image analysis, symptom analysis, and disease diagnosis from the corresponding medical imaging report to form {(I1, d1), (I2, d2), … (I N ,d N )} medical imaging graphic feature library.

7. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1 is characterized in that: The generative artificial intelligence model is used to deeply analyze and understand the medical image and text feature library obtained in step 206 to generate medical image and text question-answering instructions, specifically including: Using the generative artificial intelligence model, the medical image feature library {(I1, d1), (I2, d2), ... (I N , d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N , p N )} and negative sample data {(I1, n1), (I2, n2), ... (I N , n N )}, from multiple angles, using predefined instruction prompts, to achieve question-answer instruction data generation; wherein, the multiple angles include one or more of symptom description, treatment plan and prognosis evaluation.

8. The method for fine alignment of visual language features for a large medical multimodal model according to claim 7 is characterized in that: The generative artificial intelligence model is used to generate the medical image feature library {(I1, d1), (I2, d2), ... (I N , d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N , p N )} and negative sample data {(I1, n1), (I2, n2), ... (I N , n N )}, specifically including: Generate positive sample data (I1, p1), (I2, p2), ... (I N , p N )}; And by tampering with the information, we generate negative sample data containing wrong information {(I1, n1), (I2, n2), ... (I N , n N )}; The tampering methods used include one or more of modifying the report observation content, modifying the report analysis, modifying the reasoning process, and modifying the diagnosis conclusion, wherein the probabilities of various tampering methods are equally divided to ensure the diversity of the instruction data.

9. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1 is characterized in that: The aforementioned process introduces a reinforcement learning phase into the medical multimodal large model and utilizes the medical image-text question-answer pair instructions obtained in step 207 to achieve fine alignment of visual language features, specifically including: The medical multimodal model that contains the medical expert knowledge after training in step 205 is used to perform reinforcement learning on the medical multimodal model; wherein the training data is the positive sample data {(I1, p1), (I2, p2), ... (I N , p N )} and negative sample data {(I1, n1), (I2, n2), ... (I N , n N )}.

10. The method for fine alignment of visual language features for a large medical multimodal model according to claim 9, characterized in that: In the process of reinforcement learning of the medical multimodal large model, the DP0 algorithm is used as the loss function for training the medical multimodal large model.

Images