Medical visual question and answer method and device based on causal reasoning and large language model

Abstract

The embodiment of the invention provides a medical visual question answering method and device based on causal reasoning and a large language model, and is applied to the field related to computer vision, and the method comprises the following steps: S1, respectively extracting initial visual features and initial text features from a medical image and a question text; s2, performing feature mapping operation on the initial visual features; s3, performing feature fusion on the local visual features obtained in the step S2 and the initial text features; s4, performing feature fusion on the fusion feature, the global visual feature and the local visual feature to obtain a visual intermediary variable; carrying out feature fusion on the fusion feature and the initial text feature to obtain a text intermediary variable; s5, performing causal intervention operation; s6, generating corresponding question and answer pairs according to the training samples; and S7, generating a medical question and answer result. In this way, the technical problem that a current medical visual question answering method has some limitations such as lack of relevance or accuracy of question answering is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, natural language processing and intelligent recognition, and in particular to a medical visual question-answering method and device based on causal reasoning and a large language model. Background Art

[0002] In the field of medical image processing and visual question answering, with the development and popularization of medical imaging technology, more and more medical image data are widely collected and applied. However, traditional medical image processing methods are often limited to solving specific tasks and lack understanding of the overall context and causal relationships. At the same time, although large language models have achieved remarkable results in natural language processing tasks, they face many challenges in medical visual question answering tasks, such as insufficient understanding and interpretation of medical images, and high requirements for the accuracy and richness of medical knowledge. Therefore, current medical visual question answering methods have some limitations, including but not limited to: lack of comprehensive understanding of the context of medical images, resulting in answers to questions that may lack relevance or accuracy, limited expression and reasoning capabilities of medical knowledge, inability to make good use of image and text information for causal reasoning and comprehensive analysis, lack of integration and utilization capabilities of multimodal data, and inability to fully mine the correlation information between medical images and text. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a medical visual question answering method and device based on causal reasoning and a large language model for the current medical visual question answering task, so as to solve some limitations of the current medical visual question answering method, including but not limited to: lack of comprehensive understanding of the context of medical images, resulting in the lack of relevance or accuracy of answers to questions, limited ability to express and reason about medical knowledge, inability to make good use of image and text information for causal reasoning and comprehensive analysis, lack of ability to integrate and utilize multimodal data, inability to fully mine the correlation information between medical images and texts, and other technical problems.

[0004] According to a first aspect of the present invention, a medical visual question answering method based on causal reasoning and a large language model is provided, comprising the following steps:

[0005] S1. Extract initial visual features f from medical images and question texts respectively i and initial text features f q ;

[0006] S2, the initial visual feature f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features f il ;

[0007] S3, the local visual feature f il With the initial text feature f q Perform feature fusion to obtain the fusion feature f iq ;

[0008] S4, the fusion feature f iq With the global visual feature f ig and local visual features f il Perform feature fusion to obtain the visual mediator variable m i ; And the fusion feature f iq With the initial text feature f q Perform feature fusion to obtain the text mediating variable m q ;

[0009] S5, respectively, the initial visual features f i With the visual mediator variable m i , the initial text feature f q With the text mediating variable m q Perform causal intervention operations to obtain visual causal features f i 'With text causal feature f' q ;

[0010] S6. Use the pre-trained large language model to generate corresponding question-answer pairs based on the training samples as prompts for the question-answering process, providing rich context and reference for the question-answering process;

[0011] S7, the visual causal feature f i ', text causal feature f' q The text features of the prompt content f p Input into the large language model to generate medical question-answering results.

[0012] Furthermore, in step S1, it includes:

[0013] Extract features from medical images into feature vectors of w×h×c, where w represents the width of the feature vector, h represents the height of the feature vector, and c represents the number of channels of the feature vector;

[0014] The question text is characterized as a v×d feature vector, where v represents the size of the vocabulary and d represents the dimension of word embedding.

[0015] Furthermore, in step S2, the global visual feature f ig Provides information about the shape, position and overall state of the organ; the local visual features f il Provide detailed information, including lesions and abnormalities.

[0016] Furthermore, in step S3, the feature fusion formula is as follows:

[0017] f iq =MLP(MHA(f il ,f q ,f q ))

[0018] Furthermore, in step S4, the following formula for extracting mediating variables is used for feature fusion:

[0019] m i =MLP(MHA(f il ,f ig ,f ig ),MHA(f il ,f iq ,f iq ))

[0020] m q =MLP(MHA(f q ,f q ,f q ),MHA(f iq ,f q ,f q ))

[0021] Furthermore, in step S5, the causal intervention formula is as follows:

[0022]

[0023] Among them, P(A|do(I,Q)) represents the total causal effect of the treatment variables, i.e., image I and question Q, on the outcome variable, i.e., answer A; the do algorithm represents the intervention operation; m is the mediating variable; i, q represent the feature vectors of the image and question text, respectively; P(m|I,Q) represents the conditional probability distribution of the mediating variable m given the image I and question Q; P(A|m,i,q) represents the conditional probability of generating the answer A given the mediating variable m, image I and question Q; P(i,q) represents the joint probability distribution of image I and question Q.

[0024] Furthermore, in step S5, the causal intervention operation adopts front door adjustment, and the formula of the front door adjustment is as follows:

[0025] f i '=FDA(f i ,m i ),f' q =FDA(f q ,m q )

[0026] Here, FDA stands for Front Door Adjustment Process.

[0027] Further, in step S6, the generating corresponding question-answer pairs according to the training samples using the pre-trained large language model includes: generating question-answer pairs according to the training samples and template types input by the model;

[0028] The template type is determined by the type of question, which can be divided into open-ended questions and closed-ended questions.

[0029] In step S7, different cross entropy losses are selected for closed questions and open questions, which are defined as follows:

[0030]

[0031] Among them, Lc is the cross entropy loss of the closed problem, Lo is the cross entropy loss of the closed problem, and a t represents the true answer, T represents the training data, F() represents the overall question-answering framework, N represents the length of the answer, and a 1:n-1 Indicates partial output, p t Represents the prompt text, θ represents the model parameters; at the same time, in order to ensure the consistency between the model's prediction of the original features and the prediction of the causal features, the following loss function is defined:

[0032] L cau =KL(F(f i ',f' q ),F(f i ,f q ))

[0033] The joint loss function is defined as:

[0034] L closed =L c +L cau ,L open =L o +L cau

[0035] According to a second aspect of the present invention, a medical visual question-answering device based on causal reasoning and a large language model is also provided, comprising:

[0036] The base network is used to extract initial visual features f from medical images and question texts respectively. i and initial text features f q ; Wherein, the basic network includes a pre-trained image encoder and a text encoder;

[0037] Feature selector, used to select the initial visual features f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features fil ;

[0038] An information integration module is used to integrate the local visual features f il With the initial text feature f q Through multi-head attention, feature fusion is performed to obtain the fused feature f iq ;

[0039] The mediating variable extraction module is used to extract the fusion feature f iq With the global visual feature f ig and local visual features f il Through multi-head attention feature fusion, we get the visual mediating variable m i ; And the fusion feature f iq With the initial text feature f q Through multi-head attention feature fusion, we get the text mediating variable m q ;

[0040] A causal intervention operation module is used to respectively perform the initial visual feature f i With the visual mediator variable m i , the initial text feature f q With the text mediating variable m q Perform causal intervention operations to obtain visual causal features f i 'With text causal feature f' q ;

[0041] The prompt module is composed of pre-trained visual and language backbones. It uses the pre-trained large language model to generate corresponding question-answer pairs according to the training samples and template structure as prompt content in the question-answering process, providing rich context and reference for the question-answering process;

[0042] The visual question answering generation module is used to transform the visual causal features f i ', text causal feature f' q The text features of the prompt content f p Input into the large language model to generate medical question-answering results.

[0043] Further, wherein the causal intervention operation module includes two front-door adjustment units, and the front-door adjustment unit is composed of two attention fusion modules;

[0044] The causal intervention operation module respectively converts the initial visual features f i With the visual mediator variable m i , the initial text feature f q With the text mediating variable m q Input into the two front-door adjustment units for causal intervention operation to obtain the visual causal feature fi 'With text causal feature f' q .

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention proposes a medical visual question answering method that combines causal reasoning with a large language model to comprehensively and accurately analyze and interpret the causal relationship between medical images and related text information, thereby improving the performance and accuracy of the medical visual question answering system.

[0047] 2. The present invention can make full use of the multimodal features of medical images and text information, enabling the question-answering system to understand the questions more comprehensively and thus provide answers with more depth and credibility.

[0048] 3. The present invention can effectively remove confounding factors in the medical visual question-answering process by emphasizing causal reasoning, thereby improving the system's ability to accurately understand and analyze medical problems.

[0049] 4. The present invention can provide effective question guidance and information prompts for the medical visual question-answering system, thereby improving the system's comprehensive understanding of medical questions and its ability to accurately answer them.

[0050] 5. The present invention can combine causal reasoning with a large language model, thereby improving the ability of the medical visual question-answering system to parse and answer complex medical questions, making the question-answering results more accurate and in-depth.

[0051] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0053] Figure 1 A schematic flow chart of a medical visual question answering method based on causal reasoning and a large language model according to an embodiment of the present invention is shown;

[0054] Figure 2 A schematic diagram of the visual question answering network structure of the present invention is shown;

[0055] Figure 3 The causal reasoning diagram and the front door adjustment causal diagram of the present invention are shown;

[0056] Figure 4A de-confusion module in the causal reasoning module of the present invention is provided;

[0057] Figure 5 The accuracy comparison of the method of the present invention and other methods on VQA-RAD, SLAKE, and PathVQA datasets is presented;

[0058] Figure 6 Comparison of question answering results of the method of the present invention and other methods on the SLAKE dataset;

[0059] Figure 7 A module schematic diagram of a medical visual question-answering device based on causal reasoning and a large language model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 A schematic flow chart of a medical visual question answering method based on causal reasoning and a large language model according to an embodiment of the present invention is shown; Figure 2 FIG. 1 shows a schematic diagram of a visual question answering network structure according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a medical visual question answering method based on causal reasoning and a large language model includes the following steps:

[0063] S1. Extract initial visual features f from medical images and question texts respectively i and initial text features f q .

[0064] In step S1, the basic network module is used to extract initial visual features f from the medical image and the question text. i and initial text features f q These features can include local and global features of the image and semantic information of the question. The basic network module is composed of pre-trained image encoder and text encoder, such as Figure 1 shown.

[0065] In some embodiments, the image encoder extracts features from the medical image into a feature vector of w×h×c, where w represents the width of the feature vector, h represents the height of the feature vector, and c represents the number of channels of the feature vector. The text encoder extracts features from the input text into a feature vector of v×d, where v represents the size of the vocabulary and d represents the dimension of the word embedding.

[0066] S2, the initial visual feature f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features f il .

[0067] In step S2, a feature selector is used to select the initial visual feature f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features f il ,like Figure 3 As shown. Among them, f ig Provides information about the shape, position, and overall condition of an organ. il Provide more detailed information such as lesions, abnormalities, or other important details.

[0068] S3, the local visual feature f il With the initial text feature f q Perform feature fusion to obtain the fusion feature f iq .

[0069] Preferably, in step S3, the local visual feature f il With the initial text feature f q Through multi-head attention, feature fusion is performed to obtain the fused feature f iq ,Through this feature fusion, the relationship between text and local visual information can be made clearer and closer.

[0070] Specifically, in some embodiments, the feature fusion formula is as follows:

[0071] f iq =MLP(MHA(f il ,f q ,f q )).

[0072] S4, the fusion feature f iq With the global visual feature f ig and local visual features f il Perform feature fusion to obtain the visual mediator variable m i ; And the fusion feature f iq With the initial text feature f qPerform feature fusion to obtain the text intermediary variable m q .

[0073] In step S4, according to the needs of causal inference theory, mediating variables are first constructed, which fully integrate the characteristic information of different levels and sources. i , m q It provides a richer and more comprehensive input for subsequent causal reasoning operations. Then, causal intervention is performed to eliminate the influence of confounding factors and achieve causal effects. After the causal intervention operation, the true causal characteristics can be obtained. The causal intervention formula is as follows:

[0074]

[0075] Among them, P(A|do(I,Q)) represents the total causal effect of the treatment variables, i.e., image I and question Q, on the outcome variable, i.e., answer A; the do algorithm represents the intervention operation; m is the mediating variable; i, q represent the feature vectors of the image and question text, respectively; P(m|I,Q) represents the conditional probability distribution of the mediating variable m given the image I and question Q; the mediating variables represent the hidden layer features, which affect the answer A by extracting and combining the information of I and Q; P(A|m,i,q) represents the conditional probability of generating the answer A given the mediating variable m as well as the image I and question Q, which partly expresses how the image and question jointly affect the final answer under the action of the mediating variables; P(i,q) represents the joint probability distribution of the image I and question Q, reflecting the natural distribution of the respective features of the image and question in the absence of intervention.

[0076] Preferably, in some embodiments, in step S4, the fusion feature f iq With the global visual feature f ig and local visual features f il Through multi-head attention feature fusion, we get the visual mediating variable m i ; And the fusion feature f iq With the initial text feature f q Through multi-head attention feature fusion, we get the text mediating variable m q Specifically, the following formula for extracting mediating variables is used for feature fusion:

[0077] m i =MLP(MHA(f il ,f ig ,f ig ),MHA(f il ,f iq ,f iq ))

[0078] m q =MLP(MHA(f q ,f q ,f q ),MHA(f iq ,f q ,f q ))

[0079] S5, respectively, the initial visual features f i With the visual mediator variable m i , the initial text feature f q With the text mediating variable m q Perform causal intervention operations to obtain visual causal features f i 'With text causal feature f' q .

[0080] Preferably, in some embodiments, the causal intervention operation in step S5 adopts front-door adjustment, and the original features and mediating variables are intervened by front-door adjustment, so as to realize the decontamination process of causal reasoning. The formula of front-door adjustment is as follows:

[0081] f i '=FDA(f i ,m i ),f' q =FDA(f q ,m q )

[0082] Here, FDA stands for Front Door Adjustment Process.

[0083] S6. Use the pre-trained large language model to generate corresponding question-answer pairs based on the training samples as prompts for the question-answering process, providing rich context and reference for the question-answering process.

[0084] Specifically, in step S6, a question-answer pair is generated based on the training sample and template type input by the model; wherein the template type is determined by the type of question, which is divided into open questions and closed questions. Prompt content is provided based on the input sample to assist the causal reasoning process, thereby improving the effectiveness of generating answers.

[0085] S7, the visual causal feature f i ', text causal feature f' q The text features of the prompt content f p Input into the large language model to generate medical question-answering results.

[0086] In step S7, the visual causal feature f i ', text causal feature f' qThe text features of the prompt content f p The information is input into a large language model for the medical visual question answering process. By combining causal features with rich contextual information, the causal effect is enhanced, the accuracy of question understanding and the quality of answers are improved, thereby improving the question answering performance.

[0087] Preferably, in some embodiments, in step S7, different cross entropy losses are selected for closed questions and open questions, and are defined as follows:

[0088]

[0089] Among them, Lc is the cross entropy loss of the closed problem, Lo is the cross entropy loss of the closed problem, and a t represents the true answer, T represents the training data, F() represents the overall question-answering framework, N represents the length of the answer, and a 1:n-1 Indicates partial output, p t Represents the prompt text, θ represents the model parameters; at the same time, in order to ensure the consistency between the model's prediction of the original features and the prediction of the causal features, the following loss function is defined:

[0090] L cau =KL(F(f i ',f' q ),F(f i ,f q ))

[0091] The joint loss function is defined as:

[0092] L closed =L c +L cau ,L open =L o +L cau

[0093] According to the above embodiments, the present invention proposes a medical visual question-answering method that combines causal reasoning with a large language model to comprehensively and accurately analyze and interpret the causal relationship between medical images and related text information, thereby improving the performance and accuracy of the medical visual question-answering system. By making full use of the multimodal features of medical images and text information, the question-answering system can understand the problem more comprehensively, thereby providing more in-depth and credible answers. By emphasizing causal reasoning, confounding factors in the medical visual question-answering process are effectively removed, thereby improving the system's ability to accurately understand and analyze medical problems. By combining causal reasoning with a large language model, the medical visual question-answering system's ability to parse and answer complex medical questions is improved, making the question-answering results more accurate and in-depth.

[0094] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0095] Figure 7 FIG. 1 shows a schematic diagram of a module of a medical visual question-answering device based on causal reasoning and a large language model according to an embodiment of the present invention. Figure 7 As shown, a medical visual question answering device based on causal reasoning and a large language model includes:

[0096] The basic network module 11 is used to extract initial visual features f from medical images and question texts respectively. i and initial text features f q ; Among them, Figure 2 As shown, the basic network module 11 includes a pre-trained image encoder and a text encoder.

[0097] Feature selector 12, used to select the initial visual feature f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features f il ,like Figure 4 shown.

[0098] The information integration module 13 is used to integrate the local visual features f il With the initial text feature f q Perform feature fusion to obtain the fusion feature f iq .

[0099] The mediating variable extraction module 14 is used to extract the fusion feature f iq With the global visual feature f ig and local visual features f il Perform feature fusion to obtain the visual mediator variable m i ; and fuse the features f iq With the initial text feature f q Perform feature fusion to obtain the text intermediary variable m q ,like Figure 3 and Figure 4 shown.

[0100] The causal intervention operation module 15 is used to respectively i With the visual mediator variable m i , the initial text feature f q With the text mediating variable m q Perform causal intervention operations to obtain visual causal features f i 'With text causal feature f' q .

[0101] Specifically, in some embodiments, Figures 2 to 4 As shown, the causal intervention operation module 15 includes two front door adjustment units, and the front door adjustment unit is composed of two attention fusion modules; the causal intervention operation module 15 respectively converts the initial visual features f i With the visual mediator variable m i , the initial text feature f q With the text mediating variable m q Input into the two front-door adjustment units for causal intervention operation to obtain the visual causal feature f i 'With text causal feature f' q .

[0102] The prompt module 16 is composed of a pre-trained visual and language backbone, and uses a pre-trained large language model to generate corresponding question-answer pairs according to the training samples and the template structure, which serves as the prompt content of the question-answering process, providing rich context and reference for the question-answering process; wherein, the prompt module 16 provides prompt content according to the input sample, assists the causal reasoning process, and thereby improves the effectiveness of generating answers.

[0103] The visual question answering generation module 17 is used to transform the visual causal feature f i ', text causal feature f' q The text features of the prompt content f p Input into the large language model to generate medical question-answering results.

[0104] The causal reasoning module (mediating variable extraction module 14 and causal intervention operation module 15) and prompt module 16 provided in this example can be added to the existing medical visual network framework, thereby effectively improving the accuracy of the model question answering without increasing excessive computational costs.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0106] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0107] In order to verify the effect of the present invention, the following test was conducted. Test configuration: The hardware environment of the experiment in this paper is AMD Ryzen 9 5900k+NVID IA GeForce RTX 3090 24GB, and the software environment is Wi ndows 10x64+CUDA 12.2+CuDNN 7.1+Pytorch 2.1.1+Python3.8.

[0108] Datasets: The datasets used in the experiments of the present invention are VQA-RAD, SLAKE and PathVQA datasets, which contain abundant medical images and corresponding question-answer pairs. Table 1 gives the basic information of the datasets selected by the present invention.

[0109] Table 1 Basic information of the dataset selected in this invention

[0110]

[0111] The effects produced by the present invention are as follows Figure 6 As shown in the figure, it is shown that by adding the causal reasoning module and the prompt module, the interference of confounding factors is effectively eliminated, and the question-answering effect of the model is better. Figure 5 As shown, we can see that compared with other methods, the medical visual question answering method that combines causal reasoning with a large language model proposed in the present invention effectively removes the interference of confounding factors in the medical visual question answering process, and has achieved advanced results in three large public datasets. It can also be applied to other medical image processing and natural language processing tasks.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical visual question answering method based on causal reasoning and large language model, characterized in that: The following steps are involved: S1. Extract initial visual features f from medical images and question texts respectively i and initial text features f q ; S2, the initial visual feature f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features f il ; S3, the local visual feature f il With the initial text feature f q Perform feature fusion to obtain the fusion feature f iq ; S4, the fusion feature f iq With the global visual feature f ig and local visual features f il Perform feature fusion to obtain the visual mediator variable m i ; And the fusion feature f iq With the initial text feature f q Perform feature fusion to obtain the text mediating variable m q ; S5, respectively, the initial visual features f i With the visual mediator variable m i , the initial text feature f q With the text mediator variable m q Perform causal intervention operations to obtain visual causal features f i 'With the text causal feature f q '; S6. Use the pre-trained large language model to generate corresponding question-answer pairs based on the training samples as prompts for the question-answering process, providing rich context and reference for the question-answering process; S7, the visual causal feature f i ', text causal feature f q 'With the text features of the prompt content f p Input into the large language model to generate medical question-answering results.

2. The method according to claim 1, characterized in that: In step S1, it includes: Extract features from medical images into feature vectors of w×h×c, where w represents the width of the feature vector, h represents the height of the feature vector, and c represents the number of channels of the feature vector; The question text is characterized as a v×d feature vector, where v represents the size of the vocabulary and d represents the dimension of word embedding.

3. The method according to claim 1, characterized in that: In step S2, the global visual feature f ig Provides information about the shape, position and overall state of the organ; the local visual features f il Provide detailed information, including lesions and abnormalities.

4. The method according to claim 1, characterized in that: In step S3, the feature fusion formula is as follows: f iq =MLP(MHA(f il ,f q ,f q )) 5. The method according to claim 1, characterized in that: In step S4, feature fusion is performed using the following formula for extracting mediating variables: m i =MLP(MHA(f il ,f ig ,f ig ),MHA(f il ,f iq ,f iq )) m q =MLP(MHA(f q ,f q ,f q ),MHA(f iq ,f q ,f q )) 6. The method according to claim 1 or 5, characterized in that: In step S5, the causal intervention formula is as follows: Among them, P(A|do(I,Q)) represents the total causal effect of the treatment variables, i.e., image I and question Q, on the outcome variable, i.e., answer A; the do algorithm represents the intervention operation; m is the mediating variable; i, q represent the feature vectors of the image and question text, respectively; P(m|I,Q) represents the conditional probability distribution of the mediating variable m given the image I and question Q; P(A|m,i,q) represents the conditional probability of generating the answer A given the mediating variable m, image I and question Q; P(i,q) represents the joint probability distribution of image I and question Q.

7. The method according to claim 6, characterized in that: In step S5, the causal intervention operation adopts front door adjustment, and the formula of the front door adjustment is as follows: f i '=FDA(f i ,m i ),f q '=FDA(f q ,m q ) Here, FDA stands for Front Door Adjustment Process.

8. The method according to claim 1, characterized in that: in, In step S6, the generating corresponding question-answer pairs according to the training samples using the pre-trained large language model includes: generating question-answer pairs according to the training samples and template types input by the model; The template type is determined by the type of question, which can be divided into open-ended questions and closed-ended questions. In step S7, different cross entropy losses are selected for closed questions and open questions, which are defined as follows: Among them, Lc is the cross entropy loss of the closed problem, Lo is the cross entropy loss of the closed problem, and a t represents the true answer, T represents the training data, F() represents the overall question-answering framework, N represents the length of the answer, and a 1:n-1 Indicates partial output, p t Represents the prompt text, θ represents the model parameters; at the same time, in order to ensure the consistency between the model's prediction of the original features and the prediction of the causal features, the following loss function is defined: L cau =KL(F(f i ',f q '),F(f i ,f q )) The joint loss function is defined as: L closed =L c +L cau ,L open =L o +L cau 9. A medical visual question-answering device based on causal reasoning and large language model, characterized in that: include: The base network is used to extract initial visual features f from medical images and question texts respectively. i and initial text features f q ; Wherein, the basic network includes a pre-trained image encoder and a text encoder; Feature selector, used to select the initial visual features f i Perform feature mapping operation to obtain the global visual feature f ig and local visual features f il ; An information integration module is used to integrate the local visual features f il With the initial text feature f q Perform feature fusion to obtain the fusion feature f iq ; The mediating variable extraction module is used to extract the fusion feature f iq With the global visual feature f ig and local visual features f il Perform feature fusion to obtain the visual mediator variable m i ; And the fusion feature f iq With the initial text feature f q Perform feature fusion to obtain the text mediating variable m q ; A causal intervention operation module is used to respectively perform the initial visual feature f i With the visual mediator variable m i , the initial text feature f q With the text mediator variable m q Perform causal intervention operations to obtain visual causal features f i 'With the text causal feature f q '; The prompt module is composed of pre-trained visual and language backbones. It uses the pre-trained large language model to generate corresponding question-answer pairs according to the training samples and template structure as prompt content in the question-answering process, providing rich context and reference for the question-answering process; The visual question answering generation module is used to transform the visual causal features f i ', text causal feature f q 'With the text features of the prompt content f p Input into the large language model to generate medical question-answering results.

10. The device according to claim 9, characterized in that: in, The causal intervention operation module includes two front-door adjustment units, and the front-door adjustment unit is composed of two attention fusion modules; The causal intervention operation module respectively converts the initial visual features f i With the visual mediator variable m i , the initial text feature f q With the text mediator variable m q Input into the two front-door adjustment units for causal intervention operation to obtain the visual causal feature f i 'With the text causal feature f q '.

Images