A medical visual question answering implementation method and device and storage medium

Abstract

This invention discloses a method, apparatus, and storage medium for implementing medical visual question answering. The method includes: acquiring input information; inputting the input information into a trained medical visual question answering model; and outputting answer information. The medical visual question answering model is obtained through the following methods: acquiring a medical image-question-answer text pair; acquiring an image-encoded feature vector Fv; acquiring a text-encoded feature vector Fq; concatenating the image-encoded feature vector Fv and the text-encoded feature vector Fq to obtain an image-text-encoded feature vector Fc; and obtaining an image-text fusion feature vector Fcm with attention enhancement representation; acquiring the predicted answer and uncertainty estimation result; and training the model to obtain a medical visual question answering model with uncertainty estimation. This invention incorporates a Bayesian uncertainty estimation method into the medical question answering model to estimate the uncertainty of the model's output, obtaining a more reliable answer, and can be widely applied in the biomedical field.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, biomedicine, imaging, and intelligent systems, and in particular to a method, apparatus, and storage medium for implementing medical visual question answering. Background Technology

[0002] Medical visual question answering is a multimodal and challenging task that has been extensively studied in the fields of computer vision and natural language processing. It is also an important research direction in the fields of biomedicine and artificial intelligence.

[0003] With the rapid development of artificial intelligence (AI) technology in recent years, many AI models have caught up with or even surpassed human performance in most tasks in computer vision and natural language processing. Visual question answering is a highly challenging multimodal task that combines these two fields, aiming to enable AI models to understand the information contained in images and provide correct answers. Compared to visual question answering in natural scenes, visual question answering in medical scenarios can improve the efficiency of medical image and video analysis, enhance the accuracy of medical diagnosis, facilitate medical education and training, and increase the accessibility of healthcare consultations, offering rich and practical applications. However, due to factors such as scarce samples, high image noise, difficult and costly annotation, low reliability, and a high probability of misdiagnosis, its development has been very slow. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, device and storage medium for implementing medical visual question answering.

[0005] The technical solution adopted in this invention is:

[0006] A method for implementing medical visual question answering includes the following steps:

[0007] Obtain input information, input the input information into the trained medical vision question answering model, and output answer information with uncertain estimation;

[0008] The medical visual question-answering model is obtained through the following methods:

[0009] Obtain medical image-question-answer text pairs, and then obtain the training set based on the medical image-question-answer text pairs;

[0010] Build an image encoder to encode image data and obtain the image encoding feature vector Fv;

[0011] Build a text encoder to encode text data and obtain the text encoding feature vector Fq;

[0012] A cross-modal self-attention module is constructed, which concatenates the image-encoded feature vector Fv and the text-encoded feature vector Fq to obtain the image-text encoded feature vector Fc, and then passes it to the cross-modal self-attention module to finally obtain the image-text fusion feature vector Fcm with attention enhancement representation;

[0013] Build a Bayesian classifier, pass the image-text fusion feature vector Fcm to the Bayesian classifier, and output the predicted answer and uncertainty estimation result;

[0014] A medical visual question answering model was built based on an image encoder, a text encoder, a cross-modal self-attention module, and a Bayesian classifier. The model was trained using a training set, and finally a medical visual question answering model with uncertainty estimation was obtained.

[0015] Furthermore, the image encoder includes three sub-encoders, each with a different function, and the encoding results are concatenated to obtain the final image encoding vector Fv;

[0016] The three sub-encoders are a denoising autoencoder, a heuristic learning encoder, and a contrastive learning-based pre-trained encoder, which respectively perform image denoising, reduce sample requirements, and obtain more generalized image representations.

[0017] Furthermore, the text encoder is a pre-trained encoder based on contrastive learning. Contrastive learning has the ability to acquire cross-modal semantics, which can expand the text representation space and has better text feature extraction capabilities.

[0018] Furthermore, the cross-modal self-attention module includes a self-attention layer, a pooling layer, and a convolutional layer connected in sequence, and the self-attention layer uses residual connections inside.

[0019] Further, the step of concatenating the image-encoded feature vector Fv and the text-encoded feature vector Fq to obtain the image-text encoded feature vector Fc, and then passing it to the cross-modal self-attention module to finally obtain the image-text fusion feature vector Fcm with attention enhancement representation, includes:

[0020] The image-encoded feature vector Fv and the text-encoded feature vector Fq are fused by concatenating their feature dimensions to obtain the image-text fused encoded feature vector Fc.

[0021] After convolution, the image-text fusion encoded feature vector Fc is linearly transformed to obtain feature matrices Q, K, and V. The attention feature matrix A is then calculated using the feature matrices Q and K. Finally, the attention feature matrix A and feature matrix V are processed again to obtain the image-text fusion feature vector Fcm with attention enhancement representation.

[0022]

[0023] Fcm=A×V

[0024] Here, softmax is the exponential normalization function, and d is a constant.

[0025] Furthermore, the Bayesian classifier is constructed from two layers of Bayesian neural networks (BNNs);

[0026] The step of passing the image-text fusion feature vector Fcm to a Bayesian classifier and outputting the predicted answer and uncertainty estimation results includes:

[0027] The Bayesian neural network receives the image-text fusion feature vector Fcm with attention enhancement representation, and performs multiple samplings according to a preset prior distribution. When the input training sample is D, the input x is sampled T times by Monte Carlo and then classified by the Bayesian classifier to obtain the predicted distribution P(y|x,D) of the result y.

[0028] The mean and variance are calculated based on the prediction distribution P(y|x,D). The mean represents the prediction answer, and the variance represents the uncertainty estimate.

[0029] Furthermore, the expression for the predicted answer (Answer) is:

[0030]

[0031] The expression for the uncertainty estimation result, Uncertainty, is as follows:

[0032]

[0033] Where T represents the number of Monte Carlo samplings of the input x by the Bayesian classifier, which is set manually; P i The prediction result of the Bayesian classifier for the i-th sample in these T Monte Carlo samplings is output by the network.

[0034] Furthermore, during the training process of the medical visual question answering model, an error loss (Loss) is used to train the model. The error loss (Loss) includes Bayesian estimation loss (Lcls) and autoencoder loss (Lrec).

[0035] The error loss is calculated using the Bayesian backpropagation algorithm to determine the gradient of the mean and the gradient of the standard deviation, thereby updating the variational parameters in the model and making the variational posterior distribution of the weights in the model continuously approximate the true posterior.

[0036] Another technical solution adopted in this invention is:

[0037] A device for implementing medical visual question answering, comprising:

[0038] At least one processor;

[0039] At least one memory for storing at least one program;

[0040] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0041] Another technical solution adopted in this invention is:

[0042] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0043] The beneficial effects of this invention are: by incorporating uncertainty into the medical question-and-answer model to estimate the uncertainty of its own output results, this invention enables users to obtain safer and more reliable answers. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the steps involved in building and training a medical visual question-answering model in an embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating a method for implementing medical visual question answering in an embodiment of the present invention;

[0047] Figure 3 This is a structural diagram of the medical visual question-answering model in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of Bayesian network sampling in an embodiment of the present invention;

[0049] Figure 5 This is a graph showing the prediction performance of the medical visual question answering model in the Med-RAD dataset in this embodiment of the invention;

[0050] Figure 6 This is a diagram showing the prediction performance of the medical visual question answering model on the SLAKE dataset in this embodiment of the invention;

[0051] Figure 7 This is a prediction demo image of the medical visual question-answering model in this embodiment of the invention;

[0052] Figure 8 This is a schematic diagram of an uncertainty prediction example of the medical visual question answering model when T=10 in an embodiment of the present invention. Detailed Implementation

[0053] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0055] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0056] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0057] To address the issues of low reliability and susceptibility to misdiagnosis in existing medical question-answering models, this embodiment provides a method for implementing medical visual question answering, including the following steps:

[0058] The system acquires input information, feeds it into the trained medical visual question-answering model, and outputs a response with uncertain estimates. The input information includes medical image information and question text information.

[0059] Among them, such as Figure 1As shown, the medical visual question-answering model is obtained through the following steps:

[0060] S1. Obtain medical image-question-answer text pairs, and obtain the training set based on the medical image-question-answer text pairs.

[0061] like Figure 2 As shown, medical image-question-answer text pairs are used as model input and as model training data D. In this embodiment, the publicly available medical question-answering dataset Med-RAD SLAKE is used as the source of training data D.

[0062] S2. Build an image encoder to encode the image data and obtain the image encoding feature vector Fv.

[0063] See Figure 3 In this embodiment, step S2 specifically includes steps S21-S22:

[0064] S21. The image encoder is specifically composed of 3 sub-encoders, which are connected in parallel at the same level. This makes the encoding of each sub-encoder independent and does not interfere with each other, thus improving the purity of the encoded information.

[0065] S22. The three image sub-encoders are the Denoising Autoencoder (DAE), the Heuristic Learning Encoder (MAML), and the Contrastive Learning Pre-trained Encoder (PubMedCLIP). The DAE uses compression reconstruction to reduce image noise. The MAML, through similar task training and parameter transfer, can obtain better initial weights, reducing sample pressure and training overhead. The PubMedCLIP can fully leverage cross-domain knowledge, possessing extremely strong zero-shot learning capabilities and enhanced cross-modal representation capabilities.

[0066] S3. Build a text encoder to encode the text data and obtain the text encoding feature vector Fq.

[0067] In this embodiment, step S3 specifically includes steps S31-S32:

[0068] S31. The text encoder used is the contrastive learning pre-trained encoder PubMedCLIP. Contrastive learning has the ability to acquire cross-modal semantics and can also expand the text representation space. After pre-training, it has a better text feature extraction capability.

[0069] S32. The text encoder, after passing through PubMedCLIP, is connected to a Long Short-Term Recurrent Neural Network (LSTM) to learn the correlation between text features and output a text encoding feature vector Fq with global information.

[0070] S4. Construct a cross-modal self-attention module, concatenate the image-encoded feature vector Fv and the text-encoded feature vector Fq to obtain the image-text encoded feature vector Fc, and pass it to the cross-modal self-attention module to finally obtain the image-text fusion feature vector Fcm with attention enhancement representation.

[0071] See Figure 3 In this embodiment, step S4 specifically includes steps S41-S43:

[0072] S41. The image-encoded feature vector Fv and the text-encoded feature vector Fq are directly concatenated for feature fusion. This method can obtain a unified representation that combines information from different modalities: Image-Text Fusion Encoded Feature Fc.

[0073] Fc = [Fv; Fq]

[0074] S42. The cross-modal self-attention model CMSA is composed of a self-attention layer, a pooling layer and a convolutional layer connected one after another, in the order of self-attention layer-pooling layer-convolutional layer. The self-attention layer uses residual connections. The residuals can maintain the global flow of information (feature vectors) during the transmission process, which helps the model to better handle the correlation between vectors.

[0075] S43. The image-text fusion encoded feature vector Fc is convolved and then linearly transformed to obtain feature matrices Q, K, and V. The attention feature matrix A is calculated using the feature matrices Q and K. The attention feature matrix A and the feature matrix V are then processed again to obtain the image-text fusion feature vector Fcm with attention enhancement representation.

[0076] A = softmax(Q × K) T )

[0077] Fcm=A×V

[0078] S5. Build a Bayesian classifier, pass the image-text fusion feature vector Fcm to the Bayesian classifier, and output the predicted answer and uncertainty estimation result.

[0079] See Figure 3 In this embodiment, step S5 specifically includes steps S51-S53:

[0080] S51. The Bayesian classifier is constructed from a Bayesian neural network (BNN), the structure of which is shown in the attached figure. Figure 4 As shown, the prediction distribution of BNN for feature input x given training data D is as follows:

[0081] P(y|x, D)=∫P(y|x, w)P(w|D)dw

[0082] =∫P(y|x,w)q(w|θ)dw

[0083] Where P(y|x, w) is the output of the neural network with weights given as w, P(w|D) is the posterior distribution of the weights given training data D, and q(w|θ) is the variational posterior distribution of the weights w. This integral is equivalent to averaging the predictions of countless neural models, but since integrals in the weight space w are often difficult to compute, this integral is usually approximated using the Monte Carlo sampling method.

[0084] S52. The Bayesian classifier performs T Monte Carlo samplings on the image-text fusion feature vector Fcm with attention enhancement representation according to a spherical Gaussian distribution. At this time, the probability distribution prediction P(y|x, D) of the output by the BNN can be approximated as:

[0085]

[0086] Where w t The weights are sampled from its variational posterior distribution q(w|θ), where θ is the variational parameter.

[0087] S53. The mean of the model prediction results [P1, P2, ..., PT] after T Monte Carlo samplings can be regarded as the answer, and the difference (variance) can be regarded as the prediction uncertainty. It can be seen that BNN prediction is equivalent to the ensemble prediction of multiple models. A prediction demo is attached. Figure 7 As shown:

[0088]

[0089]

[0090] S6. Build a medical visual question answering model based on the image encoder, text encoder, cross-modal self-attention module and Bayesian classifier. Train the model using the training set to finally obtain a medical visual question answering model with uncertainty estimation.

[0091] See Figure 2 and Figure 3 The various models were sequentially connected to form a system. The error loss between the system output and the label was calculated, and the model was trained. Finally, a medical visual question-answering model with uncertainty estimation was obtained. The question-answering performance of this model on the Med-RAD and SLAKE datasets are shown in the appendix. Figure 5 , attached Figure 6As shown in Table 1, the performance has reached or exceeded that of some mainstream medical visual question answering models that lack uncertainty prediction capabilities.

[0092] Table 1 Uncertainty prediction performance under multiple sampling

[0093]

[0094] Where U / A represents the proportion of the number of uncertain issues U in the prediction to the total number of issues A in the dataset.

[0095] Uncertainty prediction sample of the model when T=10 Figure 8 As shown.

[0096] In this embodiment, step S6 specifically includes steps S61-S62:

[0097] S61, see also Figure 3 The error loss Loss(Lvqa) is obtained by adding two sub-items, including the Bayesian estimation loss Lcls and the autoencoder loss Lrec:

[0098] Loss=αLcls+βLrec

[0099] Here, α and β represent the weighting relationship between the losses, and the hyperparameters of the model are manually input constants.

[0100] S62. The error loss Loss(Lvqa) is calculated by Bayesian backpropagation algorithm to calculate the gradient of the mean and the gradient of the standard deviation, thereby updating the variational parameters in the network, so that the variational posterior distribution of the weights in the network continuously approaches the true posterior.

[0101] In summary, compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0102] (1) Based on the traditional medical visual question answering model, this invention integrates the idea of ​​multi-encoder integration and cross-modal self-attention mechanism, which reduces the training sample requirements while achieving better model performance.

[0103] (2) This invention also designs and builds a Bayesian classifier based on the Bayesian uncertainty estimation method. Compared with existing classifiers, the Bayesian classifier can provide both a comprehensive prediction answer and its prediction accuracy. This significantly improves the reliability and misdiagnosis prevention capabilities of existing medical question-answering models, while also enhancing the interpretability and robustness of the models.

[0104] This embodiment also provides a medical visual question-answering implementation device, including:

[0105] At least one processor;

[0106] At least one memory for storing at least one program;

[0107] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0108] This embodiment provides a medical visual question-answering implementation device that can execute a medical visual question-answering implementation method provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0109] This embodiment also provides a storage medium storing instructions or programs that can execute the medical visual question answering implementation method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0110] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0111] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0114] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0117] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0118] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method of implementing medical visual question answering, characterized by, Includes the following steps: Obtain input information, input the input information into the trained medical vision question answering model, and output answer information with uncertain estimation; The medical visual question-answering model is obtained through the following methods: Obtain medical image-question-answer text pairs, and then obtain the training set based on the medical image-question-answer text pairs; Build an image encoder to encode image data and obtain the image encoding feature vector Fv; Build a text encoder to encode text data and obtain the text encoding feature vector Fq; A cross-modal self-attention module is constructed, which concatenates the image-encoded feature vector Fv and the text-encoded feature vector Fq to obtain the image-text encoded feature vector Fc, and then passes it to the cross-modal self-attention module to finally obtain the image-text fusion feature vector Fcm with attention enhancement representation; Build a Bayesian classifier, pass the image-text fusion feature vector Fcm to the Bayesian classifier, and output the predicted answer and uncertainty estimation result; A medical visual question answering model was built based on an image encoder, a text encoder, a cross-modal self-attention module, and a Bayesian classifier. The model was trained using a training set, and finally a medical visual question answering model with uncertainty estimation was obtained. The image encoder includes three sub-encoders, each with a different function. The encoding results are then concatenated to obtain the final image encoding vector Fv. The three sub-encoders are a denoising autoencoder, a heuristic learning encoder, and a contrastive learning-based pre-trained encoder, which respectively function to reduce image denoising, reduce sample requirements, and obtain more generalized image representations. The text encoder is a pre-trained encoder based on contrastive learning.

2. The method for implementing medical visual question answering according to claim 1, characterized in that, The cross-modal self-attention module includes a self-attention layer, a pooling layer, and a convolutional layer connected in sequence, and the self-attention layer uses residual connections inside.

3. The method for implementing medical visual question answering according to claim 2, characterized in that, The step of concatenating the image-encoded feature vector Fv and the text-encoded feature vector Fq to obtain the image-text encoded feature vector Fc, and then passing it to the cross-modal self-attention module, ultimately obtaining the image-text fusion feature vector Fcm with attention enhancement representation, includes: The image-encoded feature vector Fv and the text-encoded feature vector Fq are fused by concatenating their feature dimensions to obtain the image-text fused encoded feature vector Fc. After convolution, the image-text fusion encoded feature vector Fc is linearly transformed to obtain feature matrices Q, K, and V. The attention feature matrix A is then calculated using the feature matrices Q and K. Finally, the attention feature matrix A and feature matrix V are processed again to obtain the image-text fusion feature vector Fcm with attention enhancement representation. in, It is an exponentially normalized function. It is a constant.

4. The method for implementing medical visual question answering according to claim 1, characterized in that, The Bayesian classifier is constructed from two layers of Bayesian neural networks; The step of passing the image-text fusion feature vector Fcm to a Bayesian classifier and outputting the predicted answer and uncertainty estimation results includes: The Bayesian neural network receives the image-text fusion feature vector Fcm with attention enhancement representation, and performs multiple samplings according to a preset prior distribution. When the input training sample is D, the input x is sampled T times using Monte Carlo methods and then classified by the Bayesian classifier to obtain the predicted distribution of the result y. ; According to the predicted distribution Calculate the mean and variance. Use the mean to represent the predicted answer and the variance to represent the uncertainty estimate.

5. The method for implementing medical visual question answering according to claim 4, characterized in that, The expression for the predicted answer is: The expression for the uncertainty estimation result, Uncertainty, is as follows: in, This represents the number of Monte Carlo samplings performed by the Bayesian classifier on the input x. For the T Monte Carlo sampling, the Bayesian classifier is used to evaluate the first... i Prediction results from the second sampling.

6. The method for implementing medical visual question answering according to claim 1, characterized in that, In the training process of the medical visual question answering model, an error loss (Loss) is used to train the model. This error loss includes Bayesian estimation loss. And autoencoder loss Lrec; The error loss is calculated using the Bayesian backpropagation algorithm to determine the gradient of the mean and the gradient of the standard deviation, thereby updating the variational parameters in the model and making the variational posterior distribution of the weights in the model continuously approximate the true posterior.

7. A device for implementing medical visual question answering, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1-6.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-6.

Images