Medical visual question-answering method and device based on efficient fine tuning of parameters, and medium

Abstract

The invention provides a medical visual question answering method and device based on efficient parameter fine tuning and a medium, and the method comprises the steps: obtaining a medical image and a question text, and carrying out the preprocessing of the medical image and the question text, and obtaining an initial visual embedding matrix and an initial text embedding matrix; the initial vision embedding matrix and the initial text embedding matrix are input into a trained medical question and answer model, answer texts for the medical images and the question texts are obtained, and the medical question and answer model comprises a vision encoder, a text encoder, a feature fusion module and an answer generator; when the medical question and answer model is trained, only parameters of adapters in the visual encoder, the text encoder and the feature fusion module are updated. The pre-trained visual model and language model are migrated into the medical question and answer model, and then the adapter is accessed to realize fine tuning of the medical question and answer model, so that time and computing resources required by training can be remarkably reduced, and meanwhile, relatively high accuracy is ensured.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a medical visual question-answering method, device, and medium based on efficient parameter fine-tuning. Background Technology

[0002] Medical visual question answering aims to combine medical images (such as CT, MRI, and X-rays) with clinical text information (such as case descriptions, laboratory reports, and doctor's questions) to automatically generate accurate and interpretable diagnostic or interpretive answers. It is an important research direction for intelligent assisted diagnosis and clinical decision support.

[0003] Based on medical visual question answering, many applications can be realized. For example: (1) Clinical auxiliary diagnosis platform: In scenarios such as tumor staging, vascular lesions, and lung infections, doctors expect the system to answer difficult questions such as "Is there any abnormality in this area of ​​the image?" and "Location and nature of foreign objects" in real time; (2) Medical education and residency training system: In order to improve the image interpretation ability of resident physicians and radiology interns, the platform needs to support multimodal question answering exercises and provide automatic comparison, difference analysis and feedback with expert standard answers; (3) Multicenter scientific research data analysis: For large-scale clinical images and electronic medical record data, researchers need high-throughput automatic question answering tools to quickly extract statistical features such as lesion distribution and complication association; (4) Telemedicine and multidisciplinary collaboration: In resource-scarce areas, doctors upload images and questions through mobile devices. The system needs to ensure multimodal semantic consistency and realize interactive diagnostic support across regions and professions.

[0004] To cope with various complex medical images, existing medical vision question answering models usually have a large number of parameters and complex structures. This not only consumes a lot of computing resources, but also has low inference efficiency, making it difficult to meet the real-time application needs in actual clinical practice, especially on resource-constrained devices. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides a medical visual question answering method, device and medium based on efficient parameter fine-tuning, so as to solve the technical problem of low training and inference efficiency caused by complex models and large number of parameters.

[0006] To achieve the above and other related objectives, this invention provides a medical visual question answering method based on efficient parameter fine-tuning, comprising: acquiring medical images and question text, and preprocessing them to obtain an initial visual embedding matrix and an initial text embedding matrix; inputting the initial visual embedding matrix and the initial text embedding matrix into a trained medical question answering model to obtain answer text for the medical images and the question text, wherein the medical question answering model comprises: a visual encoder, including a pre-trained visual model and a first adapter inserted therein, for extracting visual feature sequences from the initial visual embedding matrix; a text encoder, including a pre-trained language model and a second adapter inserted therein, for extracting text feature sequences from the initial text embedding matrix; a feature fusion module, including a third adapter inserted therein, for performing cross-modal interaction on the visual feature sequences and the text feature sequences to obtain a fused feature sequence; and an answer generator, for generating the answer text based on the fused feature sequence; the medical question answering model is trained by updating only the parameters of the first adapter, the second adapter, and the third adapter.

[0007] In one embodiment of the present invention, the visual model is a Swing Transformer model, which is pre-trained on a large-scale medical image dataset; the language model is a PubMedBERT model, which is initialized using weights pre-trained on a large-scale medical text dataset.

[0008] In one embodiment of the present invention, cross-modal interaction is performed on the visual feature sequence and the text feature sequence to obtain a fused feature sequence, including: using a self-attention mechanism to calculate the query, key, and value of the visual feature sequence and the query, key, and value of the text feature sequence respectively; using a cross-modal attention mechanism to interactively fuse the query, key, and value of the visual feature sequence and the query, key, and value of the text feature sequence to obtain text-guided visual features and visually enhanced text features respectively; using a feedforward network to process the text-guided visual features and the visually enhanced text features, and concatenating the processing results to obtain the fused feature sequence.

[0009] In one embodiment of the present invention, a cross-modal attention mechanism is used to interactively fuse the queries, keys, and values ​​of the visual feature sequence and the queries, keys, and values ​​of the text feature sequence to obtain text-guided visual features and visually enhanced text features, respectively. This includes: performing attention calculations on the queries of the visual feature sequence and the keys and values ​​of the text feature sequence to obtain the text-guided visual features; and performing attention calculations on the queries of the text feature sequence and the keys and values ​​of the visual feature sequence to obtain the visually enhanced text features.

[0010] In one embodiment of the present invention, each adapter is located after the feedforward network sublayer in each Transformer layer of its respective module.

[0011] In one embodiment of the present invention, the first adapter, the second adapter, and the third adapter have the same structure, and all adapters process the input feature x to obtain the output feature y in the following manner: y = x + GeLU(xW) down W up In the formula, W down ∈R Dmodel×d Let W be the downward projection weight matrix. up ∈R d×Dmodel Let d be the weight matrix for upward projection. Dmodel is the bottleneck dimension of the adapter, and GeLU is the activation function; during training, the medical question-answering model only updates W. down and W up The parameters.

[0012] In one embodiment of the present invention, the answer generator is a pre-trained autoregressive language model decoder. The pre-trained autoregressive language model decoder uses the fused feature sequence as a condition to generate natural language answers word by word, so as to generate open-ended answer text.

[0013] In one embodiment of the present invention, the answer generator is a classification head, which consists of one or more fully connected layers and a Softmax activation function, and directly generates the probability distribution of the category corresponding to the answer text.

[0014] To achieve the above and other related objectives, the present invention also provides an electronic device, including a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method provided in any of the above embodiments.

[0015] To achieve the above and other related objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being used to cause a computer to perform the method provided in any of the above embodiments.

[0016] The beneficial effects of the present invention are as follows: The present invention proposes a medical visual question answering method, device and medium based on parameter efficient fine-tuning. The method transfers the pre-trained visual model and language model to the medical question answering model, and then uses an adapter to fine-tune the medical question answering model. This can significantly reduce the time and computing resources required for training, while ensuring high accuracy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings are incorporated in and constitute a part of this specification, illustrating embodiments consistent with this application, and are used together with the description to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a medical visual question-answering method provided in an embodiment of the present invention; Figure 2 An architectural diagram of a medical question-answering model provided in an embodiment of the present invention; Figure 3 This is a flowchart of the feature fusion module provided in an embodiment of the present invention; Figure 4 The diagram shows the architecture of an adapter provided in one embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 501, processor; 502, memory. Detailed Implementation

[0020] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other. In addition to the specific methods, equipment, and materials used in the embodiments, based on the knowledge of the prior art and the description of the present invention by those skilled in the art, any prior art methods, equipment, and materials similar to or equivalent to the methods, equipment, and materials in the embodiments of the present invention can be used to implement the present invention.

[0021] It should be understood that the terminology used in the embodiments of this invention is for describing specific particular implementations and not for limiting the scope of protection of this invention. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In some embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0023] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations that may be implemented in the methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0024] Please see Figure 1 , Figure 1 An embodiment of the present invention provides a medical visual question answering method based on efficient parameter fine-tuning, comprising steps S101 and S102.

[0025] Step S101: Obtain medical images and problem text, and perform preprocessing to obtain the initial visual embedding matrix and the initial text embedding matrix.

[0026] In a specific embodiment of the present invention, medical images are acquired and preprocessed to obtain an initial visual embedding matrix, including: (1) standardizing the medical images, for example, normalizing the pixel values ​​to the range of [0,1] or [-1,1]; (2) denoising by Gaussian filtering or median filtering and converting them to a preset size, such as a resolution of 224×224; (3) dividing the resized medical images into N image blocks of size P×P; (4) flattening each image block and linearly projecting it onto a D-dimensional embedding space to obtain the initial visual embedding matrix. This preprocessing step ensures input consistency by standardizing the image size and transforms high-resolution images into structured sequence data. Image segmentation and linear projection efficiently encode local visual information (such as texture and edges) into dense vectors, laying a solid foundation for the subsequent visual encoder to capture global dependencies and greatly improving the model's ability to extract key lesion features in medical images.

[0027] In a specific embodiment of the present invention, the question text is acquired and preprocessed to obtain an initial text embedding matrix, including: (1) using a tokenizer to segment the question text into M tokens; (2) mapping each token to a D-dimensional embedding space to obtain the initial text embedding matrix. This preprocessing step deconstructs the medical question into semantic units through tokenization, and then transforms it into a numerical representation rich in semantic information through embedding mapping. This process not only preserves the integrity of professional terminology, but also transforms the abstract medical question into a continuous vector space that the model can process, providing a high-quality text feature foundation for subsequent cross-modal alignment and ensuring the model's accurate understanding of complex medical questions.

[0028] Step S102: Input the initial visual embedding matrix and the initial text embedding matrix into the trained medical question answering model to obtain the answer text for the medical image and the question text.

[0029] In this step, the training of the medical question-answering model is consistent with that of the conventional model, and it can be roughly divided into the following steps: (1) constructing a dataset; (2) constructing a medical question-answering model; (3) using the dataset to train and validate the medical question-answering model. In this way, a well-trained medical question-answering model can be obtained.

[0030] Please see Figure 2 In this step, the specific structure of the medical question-answering model is a significant innovation of this invention. It includes a visual encoder, a text encoder, a feature fusion module, and an answer generator. The visual encoder comprises a pre-trained visual model and a first adapter inserted therein, used to extract visual feature sequences from an initial visual embedding matrix. The text encoder comprises a pre-trained language model and a second adapter inserted therein, used to extract text feature sequences from an initial text embedding matrix. The feature fusion module includes a third adapter inserted therein, used to perform cross-modal interaction between the visual and text feature sequences to obtain a fused feature sequence. The answer generator is used to generate answer text based on the fused feature sequence. During training, the medical question-answering model only updates the parameters of the first, second, and third adapters.

[0031] In a specific embodiment of the present invention, the visual model is a Swing Transformer model, pre-trained on a large-scale medical image dataset. The initial text embedding matrix is ​​sequentially passed through multiple Swing Transformer layers, each layer containing Windowed Multi-Head Self-Attention (W-MSA) and Shifted Window MSA (SW-MSA) mechanisms. As vectors flow through these layers, the model can capture cross-scale contextual information from local texture to global organ structure. Almost all parameters in the visual model (including the weight matrices in the Transformer layers) are set to a non-trainable state (requires_grad=False), which preserves the model's general visual representation capabilities learned from large-scale data. However, the parameters in the first adapter inserted into the visual model are trainable and used to fine-tune the visual encoder. The visual encoder ultimately outputs a sequence of feature maps containing rich spatial and semantic information, i.e., a visual feature sequence, typically with a dimension of N×D.

[0032] In one specific embodiment of the invention, the language model is a PubMedBERT model, initialized using weights pre-trained on a large-scale medical text dataset. The initial text embedding matrix is ​​fed into a multi-layer Transformer encoder. In each layer, the vectors flow through a multi-head self-attention mechanism and a feed-forward network, enabling it to fully understand the grammatical structure, word meaning, and contextual dependencies in the question text. Similarly, most parameters in the language model are frozen to preserve its powerful medical language understanding capabilities. However, the parameters of the second adapter inserted into the language model are trainable and used to fine-tune the text encoder. The text encoder ultimately outputs a sequence of context vectors representing the entire question semantics, i.e., a text feature sequence, typically with a dimension of M×D.

[0033] Please see Figure 3 In a specific embodiment of the present invention, cross-modal interaction is performed on visual feature sequences and text feature sequences to obtain fused feature sequences, including steps S301 to S303.

[0034] Step S301: Using a self-attention mechanism, calculate the query, key, and value of the visual feature sequence and the text feature sequence, respectively. In this step, the visual feature sequence and the text feature sequence each pass through a self-attention layer. During this process, visual terms aggregate global image context information, while text terms further deepen the understanding of the overall problem. The query Q calculated based on the visual feature sequence is denoted as Query Q.V Key K V Value V V The result calculated based on the text feature sequence is denoted as query Q. T Key K T Value V T .

[0035] Step S302: Using a cross-modal attention mechanism, the query, key, and value of the visual feature sequence and the query, key, and value of the text feature sequence are interactively fused to obtain text-guided visual features and visually enhanced text features, respectively.

[0036] In a specific embodiment of the present invention, step S302 includes: (1) querying Q of the visual feature sequence. V and the key K of the text feature sequence T Value V T (2) Perform attention calculation to obtain the visual features of the text guidance; (3) Query Q of the text feature sequence. T and the key K of the visual feature sequence V Value V V Attention calculations are performed to obtain visually enhanced text features.

[0037] Step S303: Use a feedforward network to process the visual features of the text guidance and the text features of the visual enhancement, and then concatenate the processing results to obtain a fused feature sequence.

[0038] By setting up a feature fusion module, the model can learn the correspondence between a textual concept (e.g., "lung nodule") and a specific highlighted area in a medical image. The main parameters of the feature fusion module (such as the attention weight matrix and feedforward network weights) are also kept frozen. The self-attention mechanism and cross-modal self-attention mechanism in the feature fusion module are similar in structure to those in models such as BERT and ViT. Therefore, the main parameters in the feature fusion module can be initialized using the weights of the corresponding layers in these pre-trained models, providing a good starting point.

[0039] In one specific embodiment of the present invention, each adapter is located after the feedforward network sublayer in each Transformer layer of its respective module. This design introduces a lightweight "tuning node" at the end of each core processing unit, enabling fine-tuning of feature representations and task-oriented adjustments without compromising the core knowledge of the pre-trained model. This approach strictly limits the trainable parameters to within the adapter, achieving extreme parameter efficiency, significantly reducing computational overhead and the risk of overfitting, while ensuring that the model can quickly and stably adapt to complex medical visual question-answering tasks.

[0040] In a specific embodiment of the present invention, the first adapter, the second adapter, and the third adapter have the same structure, and all adapters process the input feature x to obtain the output feature y in the following manner: y = x + GeLU(xW) down W up In the formula, W down ∈R Dmodel×d Let W be the downward projection weight matrix. up ∈R d×Dmodel Let d be the weight matrix for upward projection. Dmodel is the bottleneck dimension of the adapter, and GeLU is the activation function; during training, the medical question-answering model only updates W. down and W up The parameters.

[0041] Please see Figure 4 In this embodiment, the adapter is a small bottleneck structure neural network module, whose structure mainly consists of: (1) Down-projection: a method that projects the input feature vector from a high-dimensional D... model Linear projection onto a low-dimensional d-feedforward network layer (d D model (2) Nonlinear activation: A nonlinear activation function, such as GeLU or ReLU. (3) Up-projection: A function that restores the feature vector from the low-dimensional d to the original high-dimensional D. model The feedforward network layer. (4) Residual Connection: The output of the adapter is added to its input element by element. This ensures that the original information flow can pass smoothly even when the adapter has not been fully learned in the early stage of training, thus stabilizing the training process.

[0042] In this embodiment, one dimension is D model After the eigenvector x enters the adapter, it undergoes downprojection W down Its dimension becomes d, which can be expressed by the formula: h = xW down Then, it undergoes processing with a non-linear activation function: h' = GeLU(h). After further upprojection by Wup, its dimension is restored to D. model This can be expressed by the formula: h”=h'W up Finally, the residual connection is added to the original input to obtain the final output: y = x + h.

[0043] Throughout the training process of the model, only the parameters in all inserted adapter modules (i.e., W) are updated. down and W up Meanwhile, the main parameters of the pre-trained model and the feature fusion module remain frozen. This is because d is much smaller than D. modelEach adapter introduces very few trainable parameters (typically less than 1% of the original model parameters), which greatly reduces computational resource requirements and memory usage, and effectively avoids catastrophic forgetting.

[0044] For answer generators, different structures can be selected depending on the type of the output answer text. The types can be mainly divided into two categories: one is open-ended answer text; the other is purely closed-ended answer text (such as closed-ended questions like "yes / no" or "exists / does not exist").

[0045] In a specific embodiment of the present invention, the answer generator is a pre-trained autoregressive language model decoder. The pre-trained autoregressive language model decoder uses the fused feature sequence as a condition to generate natural language answers word by word, so as to generate open-ended answer text.

[0046] For generating open-ended answer text, the answer text can be generated as follows: (1) Initialize a sequence of lexical characters containing only one start character. The start character can be, for example, [CLS] or <start>(2) For the current decoding time step, perform the following steps: a) Input the currently generated word sequence into the Transformer decoder, and use the enhanced fused feature sequence as the key and value to calculate the encoder-decoder attention and obtain the decoder hidden state that incorporates multimodal context information; b) Calculate the probability distribution on the vocabulary based on the decoder hidden state; c) Select the next word according to the probability distribution and append it to the generated word sequence. (3) When the generated word is the end symbol, stop the loop and convert the generated word sequence into the answer text. Understandably, when performing step (2), each generated word is judged to determine whether it is the end symbol: if not, step (2) needs to be repeated to generate the next word; if it is, step (3) is performed, the loop stops, and the final answer text is output.

[0047] The aforementioned answer generator employs an autoregressive Transformer decoder to achieve end-to-end natural language answer generation. Its core advantage lies in dynamically referencing the enhanced multimodal sequence (as key-value pairs) at each generation step, ensuring that the answer is always based on the most relevant visual and textual context. This design not only supports generating closed-ended answers such as "yes / no" and categorization, but also outputs complex open-ended answers such as lesion descriptions and diagnostic criteria, significantly enhancing the system's practicality and flexibility in real-world clinical scenarios. Its progressive generation mechanism also provides inherent support for the interpretability of the answers.

[0048] In one specific embodiment of the present invention, the answer generator is a classification head, which consists of one or more fully connected layers and a Softmax activation function, directly generating the probability distribution of the corresponding category of the answer text. In this embodiment, the complex natural language generation problem is simplified into a closed-loop classification task. The fused features are nonlinearly transformed and dimensionality reduced through fully connected layers, and finally, the Softmax function outputs the probability distribution for the preset answer category. This approach significantly reduces model complexity and computational overhead, making it particularly suitable for clinical question-and-answer scenarios with clear options, such as "yes / no" and "disease staging," ensuring both efficient reasoning and standardized answers.

[0049] In one specific embodiment of the present invention, to help doctors understand the decision-making basis of the model, attention maps can be generated using attention weight scores in cross-modal attention mechanisms. These maps can highlight the image regions that the model focuses on when answering specific questions. For example, when answering a question about "lung nodules," the region corresponding to a lung nodule on the attention map will be highlighted. This visualization result will be presented to the user along with the generated answer, greatly enhancing the transparency and credibility of the model's decision-making.

[0050] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0051] Please see Figure 5 , Figure 5 An electronic device provided in one embodiment of the present invention includes a processor 501, a memory 502, and a communication bus; the communication bus is used to connect the processor 501 and the memory 502; the processor 501 is used to execute a computer program stored in the memory 502 to implement the above-mentioned medical visual question answering method.

[0052] The aforementioned electronic device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0053] The aforementioned electronic devices can be any electronic product capable of human-computer interaction with users, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0054] The aforementioned electronic devices may also include network devices and / or user devices. Among them, network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0055] The networks in which the aforementioned electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0056] The processors mentioned above can be, for example, general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components; the memory mentioned above may include random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0057] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being used to cause a computer to execute the above-described medical visual question-and-answer method.

[0058] In summary, compared with existing medical visual question answering technologies, this invention innovatively proposes a parameter-efficient fine-tuning scheme that combines a pre-trained model with fine-tuning using a small amount of data, significantly improving inference efficiency and generalization ability. By introducing an adapter, only key layers of the model are fine-tuned, greatly reducing training time and computational resource consumption. Unlike traditional full-parameter fine-tuning methods, this invention freezes most of the pre-trained parameters, focusing on optimizing the visual encoder, language encoder, and feature fusion module, thus improving the ability to understand medical images and text. Furthermore, by combining a multimodal fusion mechanism and an adapter fine-tuning strategy, this invention enhances the model's transfer learning capability and provides interpretable output, supporting clinical decision-making and demonstrating significant technological advancement and industrial application value.

[0059] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.< / start>

Claims

1. A medical visual question-answering method based on efficient parameter fine-tuning, characterized in that, include: Acquire medical images and question text, and perform preprocessing to obtain initial visual embedding matrix and initial text embedding matrix; The initial visual embedding matrix and the initial text embedding matrix are input into the trained medical question-answering model to obtain the answer text for the medical image and the question text, wherein the medical question-answering model includes: A visual encoder, comprising a pre-trained visual model and a first adapter inserted therein, for extracting a sequence of visual features from the initial visual embedding matrix; A text encoder, comprising a pre-trained language model and a second adapter inserted therein, is used to extract a sequence of text features from the initial text embedding matrix; A feature fusion module, including a third adapter inserted therein, for performing cross-modal interaction on the visual feature sequence and the text feature sequence to obtain a fused feature sequence; and An answer generator is used to generate the answer text based on the fused feature sequence; The medical question-answering model is trained by updating only the parameters of the first adapter, the second adapter, and the third adapter.

2. The medical visual question answering method based on efficient parameter fine-tuning according to claim 1, characterized in that, The visual model is a Swing Transformer model, pre-trained on a large-scale medical image dataset; the language model is a PubMedBERT model, initialized using weights pre-trained on a large-scale medical text dataset.

3. The medical visual question answering method based on efficient parameter fine-tuning according to claim 2, characterized in that, Cross-modal interaction is performed on the visual feature sequence and the text feature sequence to obtain a fused feature sequence, including: Using a self-attention mechanism, the query, key, and value of the visual feature sequence and the query, key, and value of the text feature sequence are calculated respectively. By utilizing a cross-modal attention mechanism, the queries, keys, and values ​​of the visual feature sequence and the queries, keys, and values ​​of the text feature sequence are interactively fused to obtain text-guided visual features and visually enhanced text features, respectively. The visual features of the text guidance and the text features of the visual enhancement are processed using a feedforward network, and the processing results are concatenated to obtain the fused feature sequence.

4. The medical visual question answering method based on efficient parameter fine-tuning according to claim 3, characterized in that, By utilizing a cross-modal attention mechanism, the queries, keys, and values ​​of the visual feature sequence and the queries, keys, and values ​​of the text feature sequence are interactively fused to obtain text-guided visual features and visually enhanced text features, including: Attention is calculated on the query of the visual feature sequence and the key and value of the text feature sequence to obtain the visual features guided by the text; Attention is calculated on the query of the text feature sequence and the keys and values ​​of the visual feature sequence to obtain the visually enhanced text features.

5. The medical visual question answering method based on efficient parameter fine-tuning according to claim 3, characterized in that, Each adapter is located after the feedforward network sublayer in each Transformer layer of its respective module.

6. The medical visual question answering method based on efficient parameter fine-tuning according to claim 1, characterized in that, The first adapter, the second adapter, and the third adapter have the same structure, and all adapters process the input feature x to obtain the output feature y in the following manner: y =x+GeLU(xW down )W up , In the formula, W down ∈R Dmodel×d Let W be the downward projection weight matrix. up ∈R d×Dmodel Let d be the weight matrix for upward projection. Dmodel is the bottleneck dimension of the adapter, and GeLU is the activation function; during training, the medical question-answering model only updates W. down and W up The parameters.

7. The medical visual question answering method based on efficient parameter fine-tuning according to claim 1, characterized in that, The answer generator is a pre-trained autoregressive language model decoder. The pre-trained autoregressive language model decoder uses the fused feature sequence as a condition to generate natural language answers word by word, so as to generate open-ended answer text.

8. The medical visual question answering method based on efficient parameter fine-tuning according to claim 1, characterized in that, The answer generator is a classification head, which consists of one or more fully connected layers and a Softmax activation function, and directly generates the probability distribution of the category corresponding to the answer text.

9. An electronic device, characterized in that, It includes a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, the computer program being used to cause the computer to perform the method as described in any one of claims 1 to 8.

Images