Method and device for training debiased multi-modal large language model for medical care

Abstract

The invention provides a method and device for training a debiased multi-modal large language model for medical care, and the method comprises the steps: obtaining disease data related to medical care, and recognizing a causal variable for the prediction of the multi-modal large language model from the disease data; constructing a causal relationship graph corresponding to the disease data based on the causal variables; performing causal reasoning processing on the causal relationship graph pair to obtain debiased training data corresponding to the disease data, and generating the debiased training data through a generative adversarial network to obtain synthetic multi-modal data; and weighting causal variables in the synthesized multi-modal data, and training a multi-modal large language model through the weighted synthesized multi-modal data. Through the method and the device, the defect that prediction deviation still exists in diagnosis and analysis of diseases due to prejudice when a medical multi-mode large language model is used for predicting medical care disease data in the prior art is overcome.

Description

Technical Field

[0001] The present invention relates to the field of medical care technology, and in particular to a training method and device for a debiased multimodal large language model for medical care. Background Art

[0002] Large Language Models (LLMs) have acquired emerging capabilities, especially in understanding, reasoning, and planning, by pre-training on massive datasets. The corpus and data used during pre-training and instruction tuning significantly affect the capabilities of LLMs. In the field of healthcare, most studies utilize publicly available medical literature as disease data. This selection of disease data may introduce certain biases in medical LLMs (or multimodal large language models MLLMs). For example, medical academic papers often focus on discovering new diseases and exploring rare diseases, causing LLMs trained on these disease data to show excessive bias towards diagnosing and analyzing rare diseases, while there are predictive biases for the diagnosis and analysis of common diseases, which is not consistent with real-world healthcare needs.

[0003] In the process of developing medical MLLM, it is crucial to select appropriate disease data, as this will directly affect the performance and application effect of MLLM. Existing MLLMs usually rely on public medical literature libraries, and the disease data in these libraries focus on the latest medical discoveries and research on rare diseases. Although this provides MLLM with rich expertise and data, possible prediction biases in the diagnosis and analysis of diseases will cause problems in the application of MLLM in actual medical settings, because real-world medical needs are often more extensive, including not only the diagnosis of rare diseases, but also the management and treatment of common diseases. Therefore, the existing MLLM still needs to be improved in the diagnosis and analysis of diseases. Summary of the invention

[0004] The present invention provides a method and device for training a debiased multimodal large language model for medical care, so as to solve the defect that the traditional Chinese medicine multimodal large language model in the prior art is biased when predicting disease data of medical care, resulting in the diagnosis and analysis of the disease still having prediction bias.

[0005] The present invention provides a method for training a debiased multimodal large language model for medical care, comprising the following steps: Obtaining disease data about medical care and identifying causal variables from the disease data for prediction by a multimodal large language model; Constructing a causal relationship graph corresponding to the disease data based on the causal variables; Performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generating the debiased training data through a generative adversarial network to obtain synthetic multimodal data; The causal variables in the synthetic multimodal data are weighted, and the multimodal large language model is trained by using the weighted synthetic multimodal data.

[0006] In some embodiments, constructing a causal relationship graph corresponding to the disease data based on the causal variables includes: Define the causal relationship between each causal variable based on prior knowledge of health care; The causal variables are taken as vertices, the causal relationships are taken as edges, and a causal relationship graph corresponding to the disease data is constructed based on the vertices and the edges.

[0007] In some embodiments, performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data includes: Calling a structural causal model to model the causal relationship graph; Through backdoor adjustment analysis, the existing causal relationships to be intervened are eliminated from the modeled causal relationship graph to obtain debiased training data.

[0008] In some embodiments, the calling structure causal model to model the causal relationship graph includes: Determine each target variable in the causal relationship diagram; Inputting the target variables into the structural causal model to perform causal relationship reasoning to obtain the causal relationship between each target variable; An observed variable and a result variable are determined from the target variable according to a causal relationship, wherein the result variable is a variable predicted by a multimodal large language model.

[0009] In some embodiments, the method further comprises: Generate counterfactual causal relationships for the factual causal relationships between observed variables and outcome variables; According to the counterfactual causal relationship between the observed variable and the outcome variable, the causal relationship graph is adjusted to obtain corresponding counterfactual data, wherein the counterfactual data is used to train a multimodal large language model.

[0010] In some embodiments, the method further comprises: Calling a causal discovery algorithm to identify the causal relationship graph corresponding to the debiased training data, and obtaining the causal relationship with hidden bias in the causal relationship graph; Correct causal relationships for hidden biases.

[0011] The present invention also provides a training device for a debiased multimodal large language model for medical care, the device comprising the following modules: An acquisition module, for acquiring disease data of medical care, and identifying causal variables for prediction by a multimodal large language model from the disease data; A construction module, used for constructing a causal relationship graph corresponding to the disease data based on the causal variables; An inference module, configured to perform causal inference processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generate the debiased training data through a generative adversarial network to obtain synthetic multimodal data; A training module is used to weight the causal variables in the synthetic multimodal data and train the multimodal large language model through the weighted synthetic multimodal data.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method of a debiased multimodal large language model for health care as described in any one of the above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for training a debiased multimodal large language model for health care as described in any one of the above.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the training method of a debiased multimodal large language model for medical care as described in any one of the above.

[0015] The training method and device of the debiased multimodal large language model for medical care provided by the present invention, by identifying the causal variables predicted by the multimodal large language model from the disease data before model training, thereby constructing a causal relationship graph corresponding to the disease data. Then, the causal relationship graph is causally inferred to obtain debiased training data for training the multimodal large language model. In this way, the bias caused by the multimodal large language model when predicting disease data can be eliminated, so that the multimodal large language model can treat various types of disease data fairly, and improve the processing ability and prediction effectiveness of disease data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a training method for a debiased multimodal large language model for medical care provided by the present invention.

[0018] Figure 2 It is a schematic diagram of constructing a cause-effect relationship diagram provided by the present invention.

[0019] Figure 3 It is a schematic diagram of mining causal relationships provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of a training device for a debiased multimodal large language model for medical care provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 2 The present invention describes a method for training a debiased multimodal large language model for healthcare. Figure 1 is a flow chart of a method for training a debiased multimodal large language model for medical care provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0024] Step 101: Obtain disease data related to medical care, and identify causal variables for prediction by a multimodal large language model from the disease data.

[0025] First, disease data about medical care is obtained, and then causal variables for prediction by the multimodal large language model are identified from the disease data. These causal variables can affect the reasoning and prediction of the multimodal large language model, such as disease prevalence, patient demographics, and clinical characteristics. Disease prevalence will affect patient population, and clinical characteristics will also affect disease prevalence. There is a causal relationship between them, so they become causal variables. It should be noted that the multimodal large language model in the embodiment of the present invention refers to a large language model that can process image modality and text modality at the same time, that is, a large language model that can perform relevant predictions and reasoning on image data and text data.

[0026] Step 102: construct a causal relationship graph corresponding to the disease data based on causal variables.

[0027] Next, a causal relationship diagram corresponding to the disease data is constructed based on the causal variables. Here, the causal relationship between the causal variables needs to be analyzed. Specifically, the causal relationship between each causal variable is defined based on the prior knowledge of medical care. Prior knowledge of medical care can be domain knowledge in the medical field or statistical analysis based on a large number of medical databases. For example, in the medical field, the higher the disease prevalence, the more the patient population will increase, which means that the disease prevalence will affect the patient population, and the more obvious the clinical characteristics, the more it will restrict the flow of patients and reduce the disease prevalence. From this, it can be determined that the disease prevalence is the cause and the patient population is the result, or that the clinical characteristics are the cause and the disease prevalence is the result. In this way, the causal relationship between the disease prevalence, the patient population, and the clinical characteristics can be determined.

[0028] When constructing a causal relationship graph, the causal variables are used as vertices and the causal relationships are used as edges. The causal relationship graph corresponding to the disease data is constructed based on the vertices and edges. If there is a causal relationship between two causal variables, they are connected by an edge. This edge is generally a directed edge, with the direction from cause to effect. There is no edge connecting causal variables that do not have a causal relationship.

[0029] For example Figure 2As shown in the figure, in the causal relationship graph, the multimodal large language model predicts cancer based on infection (Pollution) and smoking (Smoking). Then infection and smoking are the cause, and cancer is the effect. If there is a causal relationship between the two, they are connected by a directed edge. If there is no causal relationship between infection and smoking, there is no directed edge. Similarly, cancer (Cancer) may cause dyspnea (Dyspnoea), and cancer (Cancer) can be treated by X-rays (X-rays). Therefore, the multimodal large language model predicts dyspnea (Dyspnoea) and X-rays (X-rays) based on cancer (Cancer), indicating that cancer is the cause, and dyspnea and X-rays are the effects. If there is a causal relationship between the two, they are connected by a directed edge, and if there is no causal relationship between dyspnea and X-rays, there is no directed edge.

[0030] Step 103: Perform causal reasoning on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generate the debiased training data through a generative adversarial network to obtain synthetic multimodal data.

[0031] according to Figure 2 In the actual environment, infection and smoking do not necessarily lead to cancer. If such causal relationships are used to train a multimodal large language model, the multimodal large language model will be biased when diagnosing diseases such as infection and smoking, resulting in poor prediction results. Therefore, in the causal relationship graph, the directed edges between infection and smoking and cancer may belong to biased paths, and the biased paths need to be intervened to prevent the multimodal large language model from making incorrect disease diagnoses.

[0032] Therefore, the embodiment of the present invention performs causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data. The process of causal reasoning processing is to remove or correct these biased paths, and at the same time, it is necessary to explore other hidden causal mechanisms therein, so as to reduce the impact of bias. The process of causal reasoning processing is described below.

[0033] First, the structural causal model (SCM) is called to model the causal relationship graph. First, each target variable in the causal relationship graph is determined, and then the target variable is input into the structural causal model for causal reasoning to obtain the causal relationship between each target variable. The resulting causal model is expressed as follows: Formula (1): (1)

[0034] In the above formula (1), Y can represent the outcome variable, that is, the variable that the multimodal large language model needs to predict, such as patient demographics; X represents the observed variable, that is, the dependent variable, such as disease prevalence; The function represents the result of causal reasoning on the observed variable X; represents unobserved confounders that exist in the disease data and may also affect the prediction of the multimodal large language model on patient demographics Y based on disease prevalence X.

[0035] The target variables in the causal relationship diagram are input into the structural causal model for causal reasoning to obtain the causal relationship between each target variable. Therefore, the corresponding observed variables and result variables can be determined according to formula (1). Here, for any target variable, it is then input into the structural causal model as an observed variable, and the result variable is the variable predicted by the multimodal large language model, which is also input into the structural causal model. This can determine the causal relationship between the observed variable and the result variable and the corresponding confounding factors. This can help the multimodal large language model understand the impact of various factors on the prediction results.

[0036] After modeling through the causal relationship graph, the backdoor adjustment analysis is used to eliminate the existing causal relationships to be intervened from the modeled causal relationship graph to obtain debiased training data. The purpose of the backdoor adjustment analysis is to make interventions, because the modeling of the structural causal model may lead to the emergence of confounding factors, which may produce erroneous causal relationships, and this erroneous causal relationship is also the causal relationship to be intervened.

[0037] For example, a structural causal model is built using cancer, dyspnea, and X-rays. In the model, cancer can cause dyspnea, which is a causal relationship. Cancer can be treated with X-rays, which is also a causal relationship. Therefore, cancer becomes a common cause of dyspnea and X-rays. This common cause may lead to a pseudo-correlation between dyspnea and X-rays that has no causal relationship. This pseudo-correlation may cause the multimodal large language model to mistakenly believe that dyspnea can also be treated with X-rays when predicting.

[0038] Therefore, through backdoor adjustment analysis, these erroneous causal relationships are intervened in the modeled causal relationship diagram, and the causal relationships to be intervened caused by these confounding factors are removed, thereby updating the causal relationship diagram. The backdoor adjustment analysis method can be expressed as the following formula (2): (2)

[0039] In the above formula (2), Y represents the outcome variable, X represents the treatment variable, Z represents the set of confounding variables, and do() represents the intervention. represents the probability of predicting Y under the condition of intervention X, It represents the probability of predicting Y under the conditions of the treatment variable and the confounding variable set. Represents the predicted impact probability of a confounding variable in the set of confounding variables.

[0040] Through backdoor adjustment analysis, we can control the confounding variables in the modeled causal relationship diagram, remove the erroneous causal relationships, and block the causal path between dyspnea and X-rays, thereby reducing the probability of using X-rays for treatment based on the symptoms of dyspnea predicted by the multimodal large language model, and ensuring that the multimodal large language model will not deviate from the causal relationship prediction between dyspnea and X-rays.

[0041] Although the modeled causal relationship graph can eliminate the causal relationships to be intervened in it through backdoor adjustment analysis, there are still some causal relationships in the causal relationship graph that are contrary to the actual environment. Therefore, in the process of modeling through the structural causal model, the embodiment of the present invention can also perform counterfactual reasoning on the modeled causal relationship graph to train the multimodal large language model by generating counterfactual data under different conditions.

[0042] Specifically, for the factual causal relationship between the observed variable and the outcome variable, a counterfactual causal relationship is generated. For example, the lower the disease prevalence, the smaller the patient population. The more obvious the clinical characteristics, the more restricted the flow of patients and the lower the disease prevalence. There is a factual causal relationship between the three. In the current situation, a counterfactual causal relationship is generated, that is, keeping the clinical characteristics unchanged and doubling the disease prevalence. This is impossible, so it is a counterfactual. Now it is necessary to break this impossible counterfactual under counterfactual conditions to determine whether the patient population will also increase. Through verification, it was found that the number of patients did increase, which means that the clinical characteristics have a direct or indirect impact on the patient population, indicating that there is a counterfactual causal relationship between the observed variable clinical characteristics and the structural variable patient population.

[0043] Next, according to the counterfactual causal relationship between the observed variable and the outcome variable, the causal relationship graph is adjusted to obtain the corresponding counterfactual data. According to the above example, there is a counterfactual causal relationship between the observed variable clinical characteristics and the structural variable patient population. In the modeled causal relationship graph, a directed edge can be added between the clinical characteristics and the patient population to prove that there is a certain causal relationship between the two, thereby generating the corresponding counterfactual data. The counterfactual data is then used to train the multimodal large language model, so that the multimodal large language model predicts an increase in the patient population based on the increase in clinical characteristics, achieving an unbiased prediction. Therefore, training the multimodal large language model with counterfactual data can improve the generalization ability of the multimodal large language model under various conditions.

[0044] Through backdoor adjustment analysis, the existing causal relationships to be intervened are eliminated from the modeled causal relationship graph, and after obtaining the debiased training data, the debiased training data is generated through a generative adversarial network to obtain synthetic multimodal data. The generation process here belongs to data enhancement. In order to target different diseases and population characteristics, synthetic data is generated to balance the representativeness in the data set, especially the representation of rare diseases. For example, debiased training data can be generated through a generative adversarial network. To synthesize the corresponding medical images and the corresponding medical report texts as synthetic multimodal data to increase the diversity of the data. Among them, the medical images generated by the generative adversarial network are image data as image modality, and the medical report text is text data as text modality, so it is called synthetic multimodal data, which is used as a training data set for training multimodal large language models.

[0045] In some embodiments, after obtaining the debiased training data, a causal discovery algorithm may be called to identify the causal relationship graph corresponding to the debiased training data to obtain the causal relationship with hidden bias in the causal relationship graph.

[0046] like Figure 3 As shown, the causal relationship between each causal variable can be fully mined through the causal discovery algorithm, such as infection can also cause breathing difficulties, smoking can also cause lung infection, etc., and then determine the causal relationship with hidden bias, such as quitting smoking through X-rays. This causal relationship has hidden biases and will lead to inaccurate predictions of multimodal large language models. Among them, the causal discovery algorithm can be a PC (Peter-Clark algorithm) algorithm, such as the acyclic hypothesis, the causal sufficiency hypothesis, the Markov condition, the causal fidelity hypothesis, etc., which are not limited here.

[0047] After the causal relationship of hidden bias is mined, the causal relationship with hidden bias can be corrected in a targeted manner. The correction process can be to eliminate the directed edges in the causal relationship graph that do not actually have causal relationships, or to exchange the causal relationships between variables. For example, "If you are infected, you will smoke" is corrected to "Smoking can cause lung infection." In this way, the deviations caused by the multimodal large language model in predicting these diseases can be corrected, so that correct predictions can be made.

[0048] Step 104: weight the causal variables in the synthetic multimodal data, and train the multimodal large language model using the weighted synthetic multimodal data.

[0049] After synthetic multimodal data is generated by generative adversarial networks, the causal variables in the synthetic multimodal data can be weighted. The causal variables here can be some diseases, including not only rare diseases but also common diseases. The only difference between the two is the different prevalence in the real environment. For example, there will be bias when the multimodal large language model predicts common diseases, so a larger weight is given to the causal variables corresponding to common diseases. For rare diseases, the prediction of the multimodal large language model will not be biased and the prediction effect is better, so a relatively small weight is given to the causal variables corresponding to rare diseases. In this way, the importance of rare diseases and common diseases can be balanced, that is, the prevalence of rare diseases and common diseases can be balanced, ensuring that the multimodal large language model can be trained on a balanced data set.

[0050] Finally, the multimodal large language model is trained by weighted synthetic multimodal data, which is used as a balanced corpus to train the multimodal large language model. During the pre-training process, domain-specific data and real-world clinical data can be used to fine-tune the pre-trained model to ensure that the predictions of the multimodal large language model meet actual medical needs and scenarios.

[0051] The evaluation phase of the training process uses fairness metrics and difference scores to measure the performance of multimodal large language models. In addition, clinical trials can be conducted to verify the effectiveness of the model in real-world settings. Going further, a framework for continuous monitoring and updating of the model is established, constantly incorporating new data to join the balanced corpus and causal insights to maintain relevance and accuracy.

[0052] The training method and device of the debiased multimodal large language model for medical care provided by the present invention, by identifying the causal variables predicted by the multimodal large language model from the disease data before model training, thereby constructing a causal relationship graph corresponding to the disease data. Then, the causal relationship graph is causally inferred to obtain debiased training data for training the multimodal large language model. In this way, the bias caused by the multimodal large language model when predicting disease data can be eliminated. In addition, causal relationship mining and correction are also performed during the training process, which can further eliminate the bias of the multimodal large language model when predicting causal relationships, so that the multimodal large language model can treat various types of disease data fairly and improve the processing ability and prediction effectiveness of disease data.

[0053] The following is a description of the training device for the debiased multimodal large language model for medical care provided by the present invention. The training device for the debiased multimodal large language model for medical care described below and the training method for the debiased multimodal large language model for medical care described above can be referenced to each other.

[0054] See also Figure 4 , Figure 4 Schematic diagram of the structure of the training device for the medical care-oriented debiased multimodal large language model provided by the present invention. Figure 4 As shown, the training device of the debiased multimodal large language model for medical care includes: an acquisition module 401, a construction module 402, an inference module 403, and a training module 404. Specifically, the acquisition module 401 is used to acquire disease data of medical care, and identify causal variables for prediction by the multimodal large language model from the disease data; the construction module 402 is used to construct a causal relationship graph corresponding to the disease data based on the causal variables; the inference module 403 is used to perform causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generate the debiased training data through a generative adversarial network to obtain synthetic multimodal data; the training module 404 is used to weight the causal variables in the synthetic multimodal data, and train the multimodal large language model through the weighted synthetic multimodal data.

[0055] It should be noted that the beneficial effects of the training device for the debiased multimodal large language model for medical care here and the training method for the debiased multimodal large language model for medical care mentioned above can correspond to each other, so the beneficial effects of the training device for the debiased multimodal large language model for medical care will not be repeated here.

[0056] Figure 5 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a training method for a debiased multimodal large language model for medical care, the method comprising: obtaining disease data about medical care, and identifying causal variables for prediction by a multimodal large language model from the disease data; constructing a causal relationship graph corresponding to the disease data based on the causal variables; performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generating the debiased training data through a generative adversarial network to obtain synthetic multimodal data; weighting the causal variables in the synthetic multimodal data, and training the multimodal large language model through the weighted synthetic multimodal data.

[0057] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0058] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of a debiased multimodal large language model for medical care provided by the above-mentioned methods, the method including: obtaining disease data about medical care, and identifying causal variables for prediction by a multimodal large language model from the disease data; constructing a causal relationship graph corresponding to the disease data based on the causal variables; performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generating the debiased training data through a generative adversarial network to obtain synthetic multimodal data; weighting the causal variables in the synthetic multimodal data, and training the multimodal large language model through the weighted synthetic multimodal data.

[0059] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of a debiased multimodal large language model for medical care provided by the above-mentioned methods, the method comprising: obtaining disease data about medical care, and identifying causal variables for prediction by a multimodal large language model from the disease data; constructing a causal relationship graph corresponding to the disease data based on the causal variables; performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generating the debiased training data through a generative adversarial network to obtain synthetic multimodal data; weighting the causal variables in the synthetic multimodal data, and training the multimodal large language model through the weighted synthetic multimodal data.

[0060] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a debiased multimodal large language model for healthcare, characterized in that: The method comprises: Obtaining disease data about medical care and identifying causal variables from the disease data for prediction by a multimodal large language model; Constructing a causal relationship graph corresponding to the disease data based on the causal variables; Performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generating the debiased training data through a generative adversarial network to obtain synthetic multimodal data; The causal variables in the synthetic multimodal data are weighted, and the multimodal large language model is trained by using the weighted synthetic multimodal data.

2. The method for training a debiased multimodal large language model for healthcare according to claim 1, characterized in that: The constructing a causal relationship graph corresponding to the disease data based on the causal variables includes: Define the causal relationship between each causal variable based on prior knowledge of health care; The causal variables are taken as vertices, the causal relationships are taken as edges, and a causal relationship graph corresponding to the disease data is constructed based on the vertices and the edges.

3. The method for training a debiased multimodal large language model for medical care according to claim 1, characterized in that: The performing causal reasoning processing on the causal relationship graph to obtain debiased training data corresponding to the disease data includes: Calling a structural causal model to model the causal relationship graph; Through backdoor adjustment analysis, the existing causal relationships to be intervened are eliminated from the modeled causal relationship graph to obtain debiased training data.

4. The method for training a debiased multimodal large language model for medical care according to claim 3, characterized in that: The calling structure causal model models the causal relationship graph, including: Determine each target variable in the causal relationship diagram; Inputting the target variables into the structural causal model to perform causal relationship reasoning to obtain the causal relationship between each target variable; An observed variable and a result variable are determined from the target variable according to a causal relationship, wherein the result variable is a variable predicted by a multimodal large language model.

5. The method for training a debiased multimodal large language model for medical care according to claim 4, characterized in that: The method further comprises: Generate counterfactual causal relationships for the factual causal relationships between observed variables and outcome variables; According to the counterfactual causal relationship between the observed variable and the outcome variable, the causal relationship graph is adjusted to obtain corresponding counterfactual data, wherein the counterfactual data is used to train a multimodal large language model.

6. The method for training a debiased multimodal large language model for medical care according to claim 1, characterized in that: The method further comprises: Calling a causal discovery algorithm to identify the causal relationship graph corresponding to the debiased training data, and obtaining the causal relationship with hidden bias in the causal relationship graph; Correct causal relationships for hidden biases.

7. A training device for a debiased multimodal large language model for medical care, characterized in that: The device comprises: An acquisition module, for acquiring disease data of medical care, and identifying causal variables for prediction by a multimodal large language model from the disease data; A construction module, used for constructing a causal relationship graph corresponding to the disease data based on the causal variables; An inference module, configured to perform causal inference processing on the causal relationship graph to obtain debiased training data corresponding to the disease data, and generate the debiased training data through a generative adversarial network to obtain synthetic multimodal data; A training module is used to weight the causal variables in the synthetic multimodal data and train the multimodal large language model through the weighted synthetic multimodal data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for training a debiased multimodal large language model for healthcare is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a debiased multimodal large language model for healthcare is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a debiased multimodal large language model for healthcare is implemented as described in any one of claims 1 to 6.

Images