Visual language feature fine alignment method for medical multi-mode large model

Abstract

The invention relates to the technical field of artificial intelligence, and provides a visual language feature fine alignment method for a medical multi-modal large model, and the method comprises the steps: carrying out the deep learning and semantic understanding of medical expert knowledge text features through an artificial intelligence large model, and generating a question and answer pair instruction; extracting medical expert knowledge visual features, and converting the features into question-answer pair instructions; training the medical multi-modal large model by using a question and answer pair instruction; using the artificial intelligence large model to generate a medical image-text question-answer pair instruction; reinforcement learning is introduced into the medical multi-modal large model, and fine alignment of visual language features is realized by using a medical image-text question and answer pair instruction. The invention provides a method for fully automatically generating high-quality medical multi-mode instruction data, so that the medical multi-mode large model provides accurate and reliable output content in various medical applications. The whole process of data generation does not need manual annotation, and manpower and time cost is saved. Meanwhile, model reasoning illusion is reduced, and model performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for fine alignment of visual language features for a large medical multimodal model. Background Art

[0002] At present, multimodal large models have attracted extensive attention from researchers because they can flexibly process data of various modalities and show strong generalization ability in a large number of application scenarios. Medical multimodal large models have become a research hotspot in the vertical application field of multimodal large models due to their great potential in the biomedical field. Unlike traditional expert models for a single task, medical multimodal large models can easily process image and text inputs and flexibly output the required results. They perform better than expert models in a variety of scenarios such as diagnostic assistance and clinical decision support. However, the training of such models relies on massive medical data with professional knowledge. Although with the development of modern medicine, there is a wealth of medical knowledge in sources such as medical research papers, medical textbooks, and hospital clinical records, it is still a major challenge to obtain training samples that accurately align text and visual forms of data, which hinders the further development of medical multimodal large models.

[0003] In order to construct training data for medical multimodal large models, current methods can be roughly divided into two categories. The first category involves retrieving medical images and corresponding text descriptions from medical research papers and medical textbooks. For example, the PMC-OA dataset extracts 1.6 million sub-images and their text descriptions from the publicly accessible subset of PubMedCentral, creating rich medical information text-image pairs. Similarly, Med-Flamingo carefully screened high-quality content from 4721 medical textbooks, including medical images and the text information involved, and constructed these contents as interleaved image-text data. Although these methods are able to generate a large number of training data sets, there is a problem that the mapping between images and texts is not fine enough. Such low-quality data inevitably affects the training efficiency. The second category of methods focuses on generating instruction fine-tuning data through external models. For example, LLaVA-Med uses GPT-4 to extract key information from image descriptions and generate image-text pair instructions. However, the quality of such image-text instructions is still essentially limited by the capabilities of external models, and the external models used are often not specifically trained on large amounts of data containing medical knowledge.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for fine alignment of visual language features for large medical multimodal models.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for fine alignment of visual language features for a large multimodal medical model, comprising:

[0008] In step 201, a publicly available medical concept text corpus is collected, and medical expert knowledge text features are extracted from the medical concept text corpus;

[0009] In step 202, a generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pair instructions;

[0010] In step 203, a plurality of publicly available medical segmentation data sets are collected, and medical expert knowledge visual features are extracted from the medical segmentation data sets; wherein the medical segmentation data sets include medical images of different modalities, and the medical images of different modalities include one or more of CT, MRI and X-ray; the medical expert knowledge visual features include a plurality of key visual information, and the key visual information includes one or more of organ contours, lesion areas and tissue structures;

[0011] In step 204, based on the visual features of the medical expert knowledge acquired in step 203, the features are converted into question-answer pair instructions for the medical term visual localization task;

[0012] In step 205, the question-answer pair instructions obtained in step 202 and step 204 are integrated, and the integrated question-answer pair instructions are used to train the medical multimodal big model, so as to inject medical expert knowledge in text form and visual form, so that the medical multimodal big model can understand and generate medical texts and interpret medical images;

[0013] In step 206, publicly available medical image analysis cases are collected and sorted, and a medical image graphic feature library is established through screening;

[0014] In step 207, the generative artificial intelligence big model is used to deeply analyze and understand the medical image text feature library obtained in step 206 to generate medical image text question and answer instructions;

[0015] In step 208, a reinforcement learning phase is introduced for the medical multimodal big model, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, thereby enhancing the generalization ability of the medical multimodal big model when processing novel medical cases, so that it can more accurately understand and analyze complex medical image-text information.

[0016] Preferably, the extracting of medical expert knowledge text features from the medical concept text corpus specifically includes:

[0017] Preliminary identification of potential medical terms using predefined medical dictionaries and regular expression matching rules;

[0018] Using the trained named entity recognition model, we further extract and verify medical terms and retain the medical terms {c1, c2, …c N};

[0019] Extract medical terms {c1, c2, …c N} One or more text information in the corresponding term explanation, symptom description and treatment plan constitutes the medical expert knowledge text feature {(c1,d1),(c2,d2),…(c N ,d N )}.

[0020] Preferably, the generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pair instructions, specifically including:

[0021] Using the medical expert knowledge text features {(c1, d1), (c2, d2), … (c N ,d N )}, and with the help of the generative artificial intelligence big model, using the predefined instruction prompts, the generative artificial intelligence big model generates text-based question-answer pair instructions for the extracted medical concepts and their related medical expert knowledge. The extracted question-answer pair instructions are recorded as

[0022] Among them, the generative artificial intelligence big model generates text-based question-and-answer instructions from one or more perspectives of term explanation, symptom description and treatment plan.

[0023] Preferably, the method of using a generative artificial intelligence large model to perform deep learning and semantic understanding on the medical expert knowledge text features acquired in step 201 and generate a large number of high-quality medical text form question-answer pair instructions also includes:

[0024] If a single generation of a question-answer pair fails, another instruction prompt will be used. If M instruction prompts all fail to be generated, the corresponding medical concept will be skipped; M is a preset value.

[0025] Preferably, the step of converting the visual features of the medical expert knowledge acquired in step 203 into question-answer pair instructions for the medical terminology visual localization task specifically includes:

[0026] The medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into question-answer instructions for the medical terminology visual localization task through a pre-set question-answer template. The pre-set question-and-answer template includes one or more of directly asking about the location, asking about the location after describing the features, and asking about the location after providing relevant information;

[0027] In the pre-set question and answer template, the question is an inquiry about the specific location of the medical term, and the answer contains the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the upper left corner coordinate of the smallest rectangular box enclosing the object corresponding to the medical term in the figure, and (x2,y2) is the lower right corner coordinate of the smallest rectangular box enclosing the object corresponding to the medical term in the figure.

[0028] Preferably, the collection and organization of publicly available medical image analysis cases and the establishment of a medical image graphic feature library through screening specifically include:

[0029] Collect publicly available case-level medical image analysis reports to generate a medical image library, which is denoted as {I1, I2, …I L}, where L is the total number of medical images;

[0030] In each medical image analysis report, a sample of N medical images is selected and stitched into one image; where N is a preset value, N is greater than or equal to 1 and N is less than or equal to 4;

[0031] Extract one or more text information of corresponding image analysis, symptom analysis, and disease diagnosis from the corresponding medical image report to form {(I1,d1),(I2,d2),…(I N ,d N )} medical imaging graphic feature library.

[0032] Preferably, the generative artificial intelligence big model is used to deeply analyze and understand the medical image text feature library obtained in step 206 to generate medical image text question and answer instructions, specifically including:

[0033] Using the generative artificial intelligence model, the medical image feature library {(I1,d1),(I2,d2),…(I N ,d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I n ,n n )}, from multiple angles, using predefined instruction prompts, to achieve question-answer instruction data generation; wherein the multiple angles include one or more of symptom description, treatment plan and prognosis evaluation.

[0034] Preferably, the generative artificial intelligence model is used to generate the medical image feature library {(I1, d1), (I2, d2), ... (I N ,d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}, including:

[0035] Generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )};

[0036] And by tampering with the information, negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}; The tampering methods used include one or more of modifying the report observation content, modifying the report analysis, modifying the reasoning process and modifying the diagnosis conclusion, wherein the probability of various tampering methods is equally divided to ensure the diversity of instruction data.

[0037] Preferably, the medical multimodal large model is introduced into a reinforcement learning phase, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, specifically including:

[0038] The medical multimodal large model that contains the medical expert knowledge after training in step 205 is used to perform reinforcement learning on the medical multimodal large model; wherein the training data is the positive sample data {(I1, p1), (I2, p2), … (IN ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}.

[0039] Preferably, in the process of reinforcement learning of the medical multimodal large model, the DPO algorithm is used as the loss function for training the medical multimodal large model.

[0040] In a second aspect, the present invention further provides a device for fine alignment of visual language features for a large medical multimodal model, which is used to implement the method for fine alignment of visual language features for a large medical multimodal model described in the first aspect, and the device comprises:

[0041] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for fine alignment of visual language features for a large medical multimodal model as described in the first aspect.

[0042] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0043] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory to execute the method of the first aspect.

[0044] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0045] The present invention proposes a method for how to fully automatically generate high-quality medical multimodal instruction data, helping the medical multimodal large model to provide more accurate and reliable output content in various medical applications. The entire process of data generation does not require manual annotation, saving a lot of manpower and time costs. At the same time, it is proposed to introduce a training strategy for the reinforcement learning stage for the medical multimodal large model with the help of the generated instruction data. The present invention helps the medical multimodal large model to achieve fine alignment of medical knowledge in visual and linguistic forms, reduce model reasoning hallucinations, and improve model performance through the designed data generation method and the introduced reinforcement learning training strategy. In addition, the method can use real clinical feedback data to quickly update model parameters and improve the downstream generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flowchart of a first method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0048] Figure 2 It is a flow chart of a second method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0049] Figure 3 It is a flowchart of a third method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0050] Figure 4 It is a flowchart of a fourth method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a method for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention;

[0052] Figure 6 It is a schematic diagram of the architecture of a device for fine alignment of visual language features for a large medical multimodal model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0055] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0056] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0057] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0058] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0059] Embodiment 1:

[0060] In view of the defects and shortcomings in the current research field, the present invention provides a method for fine alignment of visual language features for large medical multimodal models. Its purpose is to make up for the current situation that large medical multimodal models lack high-quality picture-text question-answer instructions, and to design a set of fully automatic methods for generating medical picture-text question-answer instructions with the help of publicly available medical expertise. And by introducing a reinforcement learning stage, the large medical multimodal model can quickly update the model parameters according to real clinical feedback data, thereby improving the generalization ability of the model in downstream application scenarios. A method for fine alignment of visual language features for large medical multimodal models provided in Example 1 of the present invention, such as Figure 1 As shown, including:

[0061] In step 201, publicly available medical concept text corpora are collected, and medical expert knowledge text features are extracted from the medical concept text corpora; wherein the medical concept text corpus includes but is not limited to authoritative sources such as medical textbooks, clinical guidelines, and medical journal articles.

[0062] In step 202, a generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pairs; these question-answer pairs cover a wide range from basic medical knowledge to complex clinical case analysis.

[0063] In step 203, a plurality of publicly available medical segmentation data sets are collected, and medical expert knowledge visual features are extracted from the medical segmentation data sets; wherein the medical segmentation data sets include medical images of different modalities, and the medical images of different modalities include one or more of CT, MRI and X-ray; the medical expert knowledge visual features include a plurality of key visual information, and the key visual information includes one or more of organ contours, lesion areas and tissue structures; that is, a publicly available medical segmentation database is collected, and the medical segmentation database contains images of various common medical organs in modalities such as X-ray, CT, MRI, ultrasound, and the like, as well as pixel-level fine segmentation areas corresponding to the medical organs; thereby, high-quality medical expert knowledge in visual form can be obtained.

[0064] In step 204, based on the visual features of the medical expert knowledge acquired in step 203, they are converted into question-answer pair instructions for the medical terminology visual localization task.

[0065] In step 205, the question-answer pair instructions obtained in step 202 and step 204 are integrated, and the integrated question-answer pair instructions (i.e., including the question-answer pair instructions in the form of medical text obtained in step 202 and the question-answer pair instructions of the medical term visual positioning task in step 204) are used to train the medical multimodal large model, so as to inject medical expert knowledge in text form and visual form, so that the medical multimodal large model can understand and generate high-quality medical texts and accurately interpret medical images; that is, the medical concept question-answer pair instruction data in text form and visual form generated by step 202 and step 204 are used to train the medical multimodal large model, and the medical expert knowledge from human professional physicians is injected into the model. This part of the data can be used as a supplement to the existing medical multimodal training data to provide more refined medical expert knowledge. The method described in this embodiment does not rely on a specific medical multimodal large model and can be conveniently used on any open source model. That is, the medical multimodal large model can be any open source multimodal large model.

[0066] In step 206, publicly available medical image analysis cases are collected and organized, and a medical image graphic feature library (also called a high-quality medical graphic database) is established through screening; that is, publicly available information on medical image analysis reports is collected. This type of data is mostly in the form of case studies, and professional physicians analyze and diagnose real clinical images. There is a strong correspondence between medical images and reports, which is suitable for training large medical multimodal models.

[0067] In step 207, a generative artificial intelligence big model is used to deeply analyze and understand the medical image text feature library obtained in step 206, and generate a large number of high-quality medical image text question and answer instructions covering everything from basic image recognition to complex case analysis.

[0068] In step 208, a reinforcement learning phase is introduced for the medical multimodal big model, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, thereby enhancing the generalization ability of the medical multimodal big model when processing novel medical cases, so that it can more accurately understand and analyze complex medical image-text information.

[0069] This embodiment proposes a method for fully automatically generating high-quality medical multimodal instruction data, helping the medical multimodal large model to provide more accurate and reliable output content in various medical applications. The entire process of data generation does not require manual annotation, saving a lot of manpower and time costs. At the same time, it is proposed to introduce a training strategy for the reinforcement learning stage for the medical multimodal large model with the help of the generated instruction data.

[0070] This embodiment aims to help the medical multimodal large model achieve fine alignment of medical knowledge in visual and linguistic forms, reduce model reasoning hallucinations, and improve model performance through the designed data generation method and the introduced reinforcement learning training strategy. In addition, this method can use real clinical feedback data to quickly update model parameters and improve the model's downstream generalization ability.

[0071] In an optional embodiment, the medical expert knowledge text features are extracted from the medical concept text corpus, such as Figure 2 As shown, specifically including:

[0072] In step 301, potential medical terms are initially identified using predefined medical dictionaries and regular expression matching rules.

[0073] In step 302, the trained named entity recognition model is used to further extract and verify medical terms, and the medical terms {c1, c2, ...c N}.

[0074] In step 303, extract the medical terms {c1, c2, ...c N} One or more text information in the corresponding term explanation, symptom description and treatment plan constitutes the medical expert knowledge text feature {(c1,d1),(c2,d2),…(c N ,d N )}. That is, firstly, we count the medical concepts that appear frequently; then, we obtain the corresponding information of medical concepts, such as term explanations, symptom descriptions, treatment plans, etc., from authoritative medical knowledge encyclopedia websites, knowledge bases and other data sources; thus, we can obtain a large number of medical concepts and their related rich medical expert knowledge.

[0075] In actual application scenarios, the generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pair instructions, specifically including:

[0076] Using the medical expert knowledge text features {(c1, d1), (c2, d2), … (c N ,d N )}, and with the help of the generative artificial intelligence big model, using the predefined instruction prompts, the generative artificial intelligence big model generates text-based question-answer pair instructions for the extracted medical concepts and their related medical expert knowledge. The extracted question-answer pair instructions are recorded as

[0077] Among them, the generative artificial intelligence big model generates text-based question-and-answer instructions from one or more perspectives of term explanation, symptom description and treatment plan.

[0078] In actual use, the generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pair instructions, and also includes:

[0079] If a single generation of a question-and-answer pair fails, another instruction prompt is used. If all M instruction prompts fail to generate, the corresponding medical concept is skipped. M is a preset value, and M is obtained by technical personnel in this field based on empirical analysis. In an optional embodiment, M is 3. That is, with the help of a generative artificial intelligence large model, it uses predefined instruction prompts from the perspectives of term explanation, symptom description, treatment plan, etc. to generate textual question-and-answer pair instructions for the extracted medical concepts and their related rich medical expert knowledge. Under ideal conditions, the large model can generate the selected type of question-and-answer pair instructions, but for some medical concepts, there is a situation where medical knowledge is defaulted. If a single generation fails, another instruction prompt is used. If all three instruction prompts fail to generate, the medical concept is skipped. The predefined instruction prompt is obtained by technical personnel in this field based on empirical analysis. In an optional embodiment, a predefined instruction prompt is as follows:

[0080] "Based on the provided medical concepts and corresponding text materials, please generate text question-answering instructions as required. Questions should be considered from the perspective of term explanation / symptom description / treatment plan. The generated content should pay attention to maintaining the accuracy of medical knowledge and its usefulness for auxiliary diagnosis. The output should conform to the following format:

[0081] Question: [Text Q&A question about instruction]

[0082] Answer: [Text Question Answer to Instructions]

[0083] ”

[0084] In some embodiments, the step of converting the medical expert knowledge visual features obtained in step 203 into question-answer pair instructions for the medical term visual localization task specifically includes:

[0085] The medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into question-answer instructions for the medical terminology visual localization task through a pre-set question-answer template. Among them, the pre-set question and answer template includes one or more of directly asking about the location, asking about the location after describing the characteristics, and asking about the location after giving relevant information; the pre-set question and answer template is obtained by technical personnel in this field based on empirical analysis.

[0086] In the pre-set question and answer template, the question is an inquiry about the specific location of the medical term, and the answer contains the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the coordinate of the upper left corner of the smallest rectangular box surrounding the object corresponding to the medical term in the figure, and (x2,y2) is the coordinate of the lower right corner of the smallest rectangular box surrounding the object corresponding to the medical term in the figure. That is, the medical organ obtained in step 203 and the corresponding pixel-level fine segmentation area are converted to form instruction data in the form of question and answer; wherein the question of the instruction is to inquire about the coordinate position of the medical organ, and the answer of the instruction is the coordinates of the upper left and lower right points of the smallest rectangular box containing the medical organ.

[0087] In an optional embodiment, the medical image analysis cases that are publicly available are collected and sorted, and a medical image graphic feature library is established through screening, such as Figure 3 As shown, specifically including:

[0088] In step 401, publicly available case-level medical image analysis reports are collected to generate a medical image library, which is denoted as {I1, I2, ... I L}, where L is the total number of medical images.

[0089] In step 402, in each medical image analysis report, a sample of N medical images is selected and stitched into one image; wherein N is a preset value obtained by technical personnel in this field based on empirical analysis, and N is greater than or equal to 1 and N is less than or equal to 4.

[0090] In step 403, one or more text information of corresponding image analysis, symptom analysis, and disease diagnosis is extracted from the corresponding medical image report to form {(I1, d1), (I2, d2), ... (I N ,d N )} medical imaging graphic feature library.

[0091] In actual application scenarios, the generative artificial intelligence big model is used to deeply analyze and understand the medical image text feature library obtained in step 206 to generate medical image text question and answer instructions, specifically including:

[0092] Using the generative artificial intelligence model, the medical image feature library {(I1,d1),(I2,d2),…(I N ,dN )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}, from multiple angles, using predefined instruction prompts, to achieve the generation of question-answer instruction data; wherein the multiple angles include one or more of symptom description, treatment plan and prognosis evaluation. The predefined instruction prompts are obtained by technicians in this field according to demand analysis. A predefined instruction prompt for generating positive sample data may be:

[0093] "Based on the provided medical images and corresponding analysis reports, please generate text question-answering instructions as required. Questions should be considered from multiple perspectives such as symptom description / treatment plan / prognosis assessment. The generated content should pay attention to maintaining the accuracy of medical knowledge and its usefulness for auxiliary diagnosis. The output should conform to the following format:

[0094] Question: [Text Q&A question about instruction]

[0095] Answer: [Text Question Answer to Instructions]

[0096] ”

[0097] Wherein, the generative artificial intelligence big model is used to generate the medical image feature library {(I1,d1),(I2,d2),…(I N ,d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I n ,n n )},like Figure 4 As shown, specifically including:

[0098] In step 501, positive sample data (I1, p1), (I2, p2), ... (I N ,p N )}.

[0099] In step 502, by tampering with the information, negative sample data {(I1,n1),(I2,n2),…(I N ,n N)}; The tampering methods used include modifying the report observation content, modifying the report analysis, modifying the reasoning process and modifying the diagnosis conclusion. Among them, the probability of various tampering methods is equally divided to ensure the diversity of instruction data. That is, with the help of the generation of a large model, it uses predefined instruction prompts from the perspectives of symptom description, treatment plan, etc. to generate question-answer instruction data for the collected real clinical images and analysis reports. To support the subsequent reinforcement learning stage, this process not only generates positive sample data converted from the original information, but also generates negative sample data containing erroneous information by tampering with the information. The tampering methods include modifying the report observation content, modifying the report analysis, modifying the reasoning process and modifying the diagnosis conclusion. Under ideal conditions, the generation of a large model can generate the selected type of question-answer instruction, but in some clinical image samples, there is a lack of medical knowledge. If a single generation fails, another instruction prompt is used. If all three instruction prompts fail to generate, the medical concept is skipped.

[0100] In some embodiments, the medical multimodal large model introduces a reinforcement learning phase, and uses the medical image-text question-answer pair instructions obtained in step 207 to achieve fine alignment of visual language features, specifically including:

[0101] The medical multimodal large model that contains the medical expert knowledge after training in step 205 is used to perform reinforcement learning on the medical multimodal large model; wherein the training data is the positive sample data {(I1, p1), (I2, p2), … (I N ,p N )} and negative sample data {(I1,n1),(I2,n2),…(I N ,n N )}. In addition, in the process of reinforcement learning of the medical multimodal large model, the DPO algorithm is used as the loss function for training the medical multimodal large model. That is, the reinforcement learning stage is introduced into the medical multimodal large model, and the positive and negative sample pairs converted in step 207 are used for training. This process can achieve fine alignment of the visual and language features of medical concepts and enhance the generalization ability of the medical multimodal large model. At the same time, this step can quickly update the model parameters according to the real clinical feedback data and improve the generalization ability of the model in downstream scenarios.

[0102] The present invention proposes an innovative and efficient medical concept alignment method. The present invention enhances the accuracy of alignment of medical concepts in text and visual forms, thereby helping medical multimodal large models to provide more accurate and reliable output content in various medical applications, such as disease diagnosis and treatment plan decision-making. Based on the current popular training paradigm of medical multimodal large models, the present invention integrates the expert knowledge of the text corpus and visual segmentation database of medical concepts into the instruction fine-tuning process, which enables the model to effectively learn the text and visual features of medical concepts. In order to prepare the medical concept knowledge in text form, common medical terms are first selected and their relevant information is searched in databases such as UMLS. In order to prepare the medical concept knowledge in visual form, the medical segmentation dataset is converted into a visual positioning data format in the form of a bounding box to match the output format of the current medical multimodal large model. Despite the integration of expert knowledge in text and visual forms, ensuring that these knowledge forms can be accurately aligned is still a key issue. To this end, the present invention also designs the use of real clinical medical images and their corresponding diagnostic analysis to form positive and negative sample pair training data in the form of medical image-text pairs, and introduces a reinforcement learning phase for the medical multimodal large model. By using the constructed positive and negative samples to train the training data in the reinforcement learning stage, the model can be helped to align medical knowledge in text and image forms and improve the generalization ability of the model.

[0103] Embodiment 2:

[0104] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0105] The method for fine alignment of visual language features for a large medical multimodal model provided in this embodiment is as follows: Figure 5 As shown, the following steps are included:

[0106] Step 1: Collect a wide range of publicly available medical concept text corpora, including but not limited to authoritative sources such as medical textbooks, clinical guidelines, and medical journal articles. Then, extract text features of medical expert knowledge from these corpora.

[0107] In step 1, a medical concept database for training a large medical multimodal model was collected. These medical concepts are derived from publicly available academic data sets. Through natural language processing technology, medical terms are extracted and compared with authoritative medical corpora. The terms that appear in the corpus are retained to form a set of medical concepts to be processed. Specifically, a strategy combining rule-based methods and machine learning methods is adopted. First, potential medical terms are preliminarily identified using predefined medical dictionaries and regular expression matching rules. Then, the trained named entity recognition (NER) model is used to further extract and verify medical terms. Subsequently, the text information such as term explanations, symptom descriptions, and treatment plans corresponding to these medical concepts are extracted from the authoritative medical corpus, and the terms whose frequency of occurrence in the corpus exceeds the preset threshold are retained to form a medical concept text feature library. This method ensures that the selected medical concepts are sufficiently representative and authoritative. In order to collect medical concepts that are beneficial to model training, the source of medical concepts is publicly available academic datasets. Medical terms are extracted through natural language processing technology and compared with authoritative medical corpora. The terms that appear in the corpus are retained. This set is the medical concepts to be processed later, denoted as {c1, c2, …c N}, where N is the total number of medical concepts. Then, the corresponding term explanations, symptom descriptions, treatment plans and other text information of medical concepts are extracted from the authoritative medical corpus to form {(c1, d1), (c2, d2), … (c N ,d N )} medical concept text feature library (i.e., medical expert knowledge text features).

[0108] This medical concept text feature library not only contains medical concepts that appear frequently in daily medical work, but also includes text information such as term explanations, symptom descriptions, treatment plans, etc. corresponding to these concepts, which provides a powerful supplement to the existing coarse-grained image text description medical multimodal training data.

[0109] Step 2: Using advanced generative AI models, we conduct deep learning and semantic understanding of the medical expert knowledge text features obtained in step 1, and automatically generate a large number of high-quality medical text question-answer pairs. These question-answer pairs cover a wide range from basic medical knowledge to complex clinical case analysis.

[0110] like Figure 5As shown in the second part of the stage, this step constructs medical text form question-answer pairs. The generation of these question-answer pair instruction data requires the medical concept text feature library of the previous step, and with the help of the generative artificial intelligence big model, the model uses predefined instruction prompts from the perspectives of term explanation, symptom description, treatment plan, etc. to generate text form question-answer pairs for the extracted medical concepts and their related rich medical expert knowledge. A reference instruction prompt is as follows:

[0111] "Based on the provided medical concepts and corresponding text materials, please generate text question-answering instructions as required. Questions should be considered from the perspective of term explanation / symptom description / treatment plan. The generated content should pay attention to maintaining the accuracy of medical knowledge and its usefulness for auxiliary diagnosis. The output should conform to the following format:

[0112] Question: [Text Q&A question about instruction]

[0113] Answer: [Text Question Answer to Instructions]

[0114] ”

[0115] Using instruction prompts similar to the reference instructions above, the generative AI model can generate high-quality medical text-based question-answer pairs for the provided medical concepts, which will be used in the training of subsequent steps. This method can not only generate a large amount of training data, but also ensure that the data covers a wide range of medical knowledge, thereby providing comprehensive learning materials for the model.

[0116] The generation of medical text form question answering instructions requires the previous step {(c1,d1),(c2,d2),…(c N ,d N )}, and with the help of a generative artificial intelligence model, it uses predefined instruction prompts to generate text-based question-answer pairs for the extracted medical concepts and their related rich medical expert knowledge from the perspectives of term explanation, symptom description, and treatment plan. The extracted question-answer pairs are recorded as

[0117] Step 3: Collect a variety of publicly available medical segmentation datasets, including but not limited to medical images of different modalities such as CT, MRI, and X-ray, and extract medical expert knowledge of key visual information such as organ contours, lesion areas, and tissue structures.

[0118] like Figure 5As shown in the third part of the stage, step three is responsible for collecting medical expert knowledge in visual form. In order to collect comprehensive visual features of medical concepts, the organs, lesion types in the image annotations, and the corresponding pixel-level segmentation region coordinates in the image can be extracted from publicly available medical segmentation datasets. Including but not limited to different modalities such as CT, MRI, X-ray, PET, and covering various systems and parts of the human body, including but not limited to the cardiovascular system, respiratory system, digestive system, nervous system, etc. Detailed segmentation region coordinates can help the model accurately understand the region-level features in the image, so as to better identify and locate various normal anatomical structures and pathological changes. The collected medical concepts and corresponding segmentation coordinates are labeled as {(v1,r1),(v2,r2),…(v M ,r M )}, where M is the total number of medical concepts involved in this step.

[0119] Step 4: Based on the visual features of medical expert knowledge obtained in step 3, convert them into question-answer instructions for the medical terminology visual localization task.

[0120] like Figure 5 As shown in the fourth stage of the figure, the main function of step 4 is to generate question-answer pairs in the format of medical terminology visual localization task. In this step, the medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into medical terminology visual positioning instruction data (also called question-answer pair instructions for medical terminology visual positioning tasks) through a pre-set question-answer template. The converted question-answer pair is recorded as To enhance the diversity of instructions, the pre-set templates should be as diverse as possible, including but not limited to asking about the location directly, asking about the location after describing the characteristics, asking about the location after giving relevant information, etc. In the question-and-answer template, the questions are all about asking about the specific location of the medical term, and the answers contain the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the upper left corner coordinate of the smallest rectangular box enclosing the object corresponding to the medical term in the figure, and (x2,y2) is the lower right corner coordinate of the smallest rectangular box enclosing the object corresponding to the medical term in the figure.

[0121] The generated question-answer pair instructions in the format of visual localization tasks for medical concepts will be used in the training of subsequent steps to provide the model with accurate visual-language correspondence learning materials.

[0122] Step 5: Integrate the question-answer pair instructions obtained in steps 2 and 4 for the training of the medical multimodal large model, and effectively inject medical expert knowledge in text and visual forms, so that the model can understand and generate high-quality medical texts and accurately interpret medical images.

[0123] like Figure 5 As shown in the fifth part of the stage, the main function of step five is to use the medical concept question-answering instruction data in text and visual form generated by steps two and four and The process of training a large medical multimodal model. This process injects medical expert knowledge from human professional physicians into the model. This part of the data can be used as a supplement to the existing medical multimodal training data to provide more refined medical expert knowledge.

[0124] Step 6: Collect and organize a large number of publicly available medical imaging analysis cases, and establish a high-quality medical image and text database through screening.

[0125] like Figure 5 As shown in the sixth part of the stage, the main purpose of step six is ​​to collect medical knowledge data with aligned text and image features. In this step, it is necessary to collect publicly available case-level medical image analysis reports. The medical image library is denoted as {I1,I2,…I L}, where L is the total number of medical images. In the available case-level medical image analysis reports, the number of medical images can often be as many as ten. To facilitate the processing of the current medical multimodal large model, in this step, only samples containing one to four medical images are selected and spliced ​​into one image. Subsequently, the corresponding image analysis, symptom analysis, disease diagnosis and other text information are extracted from the corresponding medical reports to form {(I1, d1), (I2, d2), … (I N ,d N )}. This type of database contains one or more medical images, as well as comprehensive text information such as corresponding image analysis, symptom analysis, disease diagnosis, treatment plan and prognosis evaluation. Such a medical image text feature library achieves fine alignment of text and image features of medical knowledge, which is of great help in improving the ability of medical multimodal large models to read and understand text information.

[0126] Step 7: Use the generative artificial intelligence model to generate the medical image and text database {(I1,d1),(I2,d2),…(I N ,d N )} conducts in-depth analysis and understanding, and automatically generates a large number of high-quality medical image-text question-and-answer instructions covering everything from basic image recognition to complex case analysis.

[0127] like Figure 5 As shown in the seventh stage of the paper, in step seven, the generative artificial intelligence model is used to analyze and process the medical image text feature library obtained in the previous step, and the predefined instruction prompts are used to generate question-answer instruction data from the perspectives of symptom description, treatment plan prognosis evaluation, etc. To support the subsequent reinforcement learning stage, this process generates positive sample data converted from the original information, denoted as {(I1,p1),(I2,p2),…(I N ,p N )}; and by tampering with the information, negative sample data containing wrong information is generated, recorded as {(I1,n1),(I2,n2),…(I N ,n N )}. The tampering methods used include modifying the report observation content, modifying the report analysis, modifying the reasoning process, and modifying the diagnosis conclusion. The probability of each tampering method is equally divided to ensure the diversity of instruction data. A reference instruction prompt for generating a training positive sample is as follows:

[0128] "Based on the provided medical images and corresponding analysis reports, please generate text question-answering instructions as required. Questions should be considered from multiple perspectives such as symptom description / treatment plan / prognosis assessment. The generated content should pay attention to maintaining the accuracy of medical knowledge and its usefulness for auxiliary diagnosis. The output should conform to the following format:

[0129] Question: [Text Q&A question about instruction]

[0130] Answer: [Text Question Answer to Instructions]

[0131] ”

[0132] Using instruction prompts similar to the above-mentioned reference instructions can enable the generative artificial intelligence large model to generate high-quality image-text question-answer instruction data for the medical imaging image-text feature library, and the generated instructions will be used in the training of subsequent steps.

[0133] The process of generating training negative samples requires, as mentioned above, modifying some information in the medical image text feature library, and then using the above reference instruction prompts to generate negative samples for training the text-image question-answer instruction data. When generating negative samples, attention should be paid to controlling the difficulty and rationality of the error information to ensure that the model can learn meaningful error patterns.

[0134] Step 8: Introduce a reinforcement learning phase for the medical multimodal large model, and use the medical image-text question-answer instructions obtained in step 7 to achieve fine alignment of visual language features, thereby enhancing the model's generalization ability when processing novel medical cases, enabling it to more accurately understand and analyze complex medical image-text information.

[0135] like Figure 5As shown in the eighth part of the figure, the main function of this step is to introduce the reinforcement learning training stage for the medical multimodal large model. This stage uses the model that contains a lot of medical expert knowledge after training in step five. The training data is the positive and negative sample pair instruction data extracted in step seven, that is, {(I1, p1), (I2, p2), … (I N ,p N )} and {(I1,n1),(I2,n2),…(I N ,n N )}. During the training process, the model will prefer to output the high-quality medical knowledge contained in the training positive samples {(I1,p1),(I2,p2),…(I1,p1)}, thereby helping to achieve fine alignment of the visual and linguistic features of medical concepts and enhancing the generalization ability of large medical multimodal models. In order to implement the constraints on the model's preference output, the DPO algorithm is used as the loss function for model training. At the same time, the training data for this step can be derived from real clinical feedback data. With the help of this training strategy, the model parameters can be quickly updated, thereby getting rid of the cumbersome process of using large-scale medical data to train the model from scratch when updating the model training data. This training paradigm significantly improves the generalization ability and application value of the model in downstream scenarios.

[0136] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which are all within the scope of the present invention to be protected. For example, the model can be trained specifically according to the needs of different medical specialties; more data enhancement techniques can be introduced to improve the robustness of the model; and the method can be combined with other advanced artificial intelligence technologies, such as meta-learning, multi-agent reinforcement learning technology, etc., to further enhance the performance and adaptability of the model. The scope of protection claimed in the present invention is defined by the attached claims and their equivalents.

[0137] Embodiment 3:

[0138] like Figure 6 , is a schematic diagram of the architecture of a device for fine alignment of visual language features for a large medical multimodal model according to an embodiment of the present invention. The device for fine alignment of visual language features for a large medical multimodal model according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0139] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0140] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the fine alignment method of visual language features for medical multimodal large models in Example 1. The processor 21 executes the fine alignment method of visual language features for medical multimodal large models by running the non-volatile software programs and instructions stored in the memory 22.

[0141] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the method for fine alignment of visual language features for a large medical multimodal model in the above-mentioned embodiment 1 is executed.

[0143] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0144] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fine alignment of visual language features for large medical multimodal models, characterized in that: include: In step 201, a publicly available medical concept text corpus is collected, and medical expert knowledge text features are extracted from the medical concept text corpus; In step 202, a generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pair instructions; In step 203, a plurality of publicly available medical segmentation data sets are collected, and medical expert knowledge visual features are extracted from the medical segmentation data sets; wherein the medical segmentation data sets include medical images of different modalities, and the medical images of different modalities include one or more of CT, MRI and X-ray; the medical expert knowledge visual features include a plurality of key visual information, and the key visual information includes one or more of organ contours, lesion areas and tissue structures; In step 204, based on the visual features of the medical expert knowledge acquired in step 203, the features are converted into question-answer pair instructions for the medical term visual localization task; In step 205, the question-answer pair instructions obtained in step 202 and step 204 are integrated, and the integrated question-answer pair instructions are used to train the medical multimodal big model, so as to inject medical expert knowledge in text form and visual form, so that the medical multimodal big model can understand and generate medical texts and interpret medical images; In step 206, publicly available medical image analysis cases are collected and sorted, and a medical image graphic feature library is established through screening; In step 207, the generative artificial intelligence big model is used to deeply analyze and understand the medical image text feature library obtained in step 206 to generate medical image text question and answer instructions; In step 208, a reinforcement learning phase is introduced for the medical multimodal big model, and the medical image-text question-answer pair instructions obtained in step 207 are used to achieve fine alignment of visual language features, thereby enhancing the generalization ability of the medical multimodal big model when processing novel medical cases, so that it can more accurately understand and analyze complex medical image-text information.

2. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1, characterized in that: The extracting of medical expert knowledge text features from the medical concept text corpus specifically includes: Preliminary identification of potential medical terms using predefined medical dictionaries and regular expression matching rules; Using the trained named entity recognition model, we further extract and verify medical terms and retain the medical terms {c1, c2, …c N }; Extract medical terms {c1,c2,…c N } One or more text information in the corresponding term explanation, symptom description and treatment plan constitutes the medical expert knowledge text feature {(c1,d1),(c2,d2),…(c N ,d N )}.

3. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1, characterized in that: The generative artificial intelligence big model is used to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201, and generate a large number of high-quality medical text form question-answer pair instructions, specifically including: Using the medical expert knowledge text features {(c1, d1), (c2, d2), … (c N ,d N )}, and with the help of the generative artificial intelligence big model, using the predefined instruction prompts, the generative artificial intelligence big model generates text-based question-answer pair instructions for the extracted medical concepts and their related medical expert knowledge. The extracted question-answer pair instructions are recorded as Among them, the generative artificial intelligence big model generates text-based question-and-answer instructions from one or more perspectives of term explanation, symptom description and treatment plan.

4. The method for fine alignment of visual language features for a large medical multimodal model according to claim 2, characterized in that: The method of using the generative artificial intelligence big model to perform deep learning and semantic understanding on the medical expert knowledge text features obtained in step 201 and generate a large number of high-quality medical text form question-answer pair instructions also includes: If a single generation of a question-answer pair fails, another instruction prompt will be used. If M instruction prompts all fail to be generated, the corresponding medical concept will be skipped; M is a preset value.

5. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1, characterized in that: The step of converting the visual features of the medical expert knowledge obtained in step 203 into question-answer pair instructions for the medical term visual localization task specifically includes: The medical expert knowledge visual features {(v1,r1),(v2,r2),…(v M ,r M )} is parsed and processed, and converted into question-answer instructions for the medical terminology visual localization task through a pre-set question-answer template. The pre-set question-and-answer template includes one or more of directly asking about the location, asking about the location after describing the features, and asking about the location after providing relevant information; In the pre-set question and answer template, the question is an inquiry about the specific location of the medical term, and the answer contains the specific location information of the medical term. The format provided is <x1,y1,x x ,y2>, where (x1,y1) is the upper left corner coordinate of the smallest rectangular box enclosing the object corresponding to the medical term in the figure, and (x2,y2) is the lower right corner coordinate of the smallest rectangular box enclosing the object corresponding to the medical term in the figure.

6. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1, characterized in that: The collection and organization of publicly available medical image analysis cases, and the establishment of a medical image graphic feature library through screening, specifically include: Collect publicly available case-level medical image analysis reports to generate a medical image library, which is denoted as {I1, I2, …I L }, where L is the total number of medical images; In each medical image analysis report, a sample of N medical images is selected and stitched into one image; where N is a preset value; Extract one or more text information of corresponding image analysis, symptom analysis, and disease diagnosis from the corresponding medical image report to form {(I1,d1),(I2,d2),…(I N ,d N )} medical imaging graphic feature library.

7. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1, characterized in that: The generative artificial intelligence big model is used to deeply analyze and understand the medical image text feature library obtained in step 206 to generate medical image text question and answer instructions, specifically including: Using the generative artificial intelligence model, the medical image feature library {(I1, d1), (I2, d2), ... (I N , d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N , p N )} and negative sample data {(I1, n1), (I2, n2), ... (I N , n N )}, from multiple angles, using predefined instruction prompts, to achieve question-answer instruction data generation; wherein the multiple angles include one or more of symptom description, treatment plan and prognosis evaluation.

8. The method for fine alignment of visual language features for a large medical multimodal model according to claim 7, characterized in that: The generative artificial intelligence model is used to generate the medical image feature library {(I1, d1), (I2, d2), ... (I N , d N )} to analyze and process, and generate positive sample data (I1, p1), (I2, p2), ... (I N , p N )} and negative sample data {(I1, n1), (I2, n2), ... (I N , n N )}, including: Generate positive sample data (I1, p1), (I2, p2), ... (I N , p N )}; And by tampering with the information, negative sample data containing wrong information is generated {(I1, n1), (I2, n2), ... (I N , n N )}; The tampering methods used include one or more of modifying the report observation content, modifying the report analysis, modifying the reasoning process and modifying the diagnosis conclusion, wherein the probability of various tampering methods is equally divided to ensure the diversity of instruction data.

9. The method for fine alignment of visual language features for a large medical multimodal model according to claim 1, characterized in that: The aforementioned method introduces a reinforcement learning phase into the medical multimodal large model, and uses the medical picture-text question-answer pair instructions obtained in step 207 to achieve fine alignment of visual language features, specifically including: The medical multimodal large model containing the medical expert knowledge after training in step 205 is used to perform reinforcement learning on the medical multimodal large model; wherein the training data is the positive sample data {(I1, p1), (I2, p2), ... (I N , p N )} and negative sample data {(I1, n1), (I2, n2), ... (I N , n N )}.

10. The method for fine alignment of visual language features for a large medical multimodal model according to claim 9, characterized in that: In the process of reinforcement learning of the medical multimodal large model, the DP0 algorithm is used as the loss function for training the medical multimodal large model.

Images