Medical question and answer method based on dynamic prompt and decoding knowledge editing and related device

Abstract

This application discloses a medical question-answering method and related apparatus based on dynamic prompts and decoded knowledge editing, belonging to the field of artificial intelligence technology. The method first acquires a medical visual question-answering dataset containing medical images and question-answer pairs. Then, it pre-trains a medical visual language model using a contrastive learning framework, combining local contrast loss, global contrast loss, image-text matching loss, and masked language modeling loss to enhance the fine-grained and overall modal interaction between vision and text. Subsequently, dynamic prompts are added to the medical image and text features, eliminating the need for manually designed prompt words and adapting to multimodal feature differences. Next, the features with dynamic prompts are input into the pre-trained model to obtain initial predicted answers. Finally, through decoded knowledge editing, the output is corrected using a retrieval and verification mechanism from an external knowledge base, enhancing robustness in knowledge-intensive medical scenarios. Experimental results demonstrate that this method effectively improves the accuracy of medical visual question answering and performs excellently on various medical datasets.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a medical question-answering method and related apparatus based on dynamic prompts and decoded knowledge editing. Background Technology

[0002] Medical visual question answering aims to enable models to generate accurate answers based on medical images (such as radiological images and pathological slides) and natural language questions, aligning with medical knowledge. It is a core task connecting medical image analysis and natural language understanding, and has significant application value in clinical auxiliary diagnosis and medical education. Currently, pre-trained medical visual language models are the mainstream approach for solving medical visual question answering tasks: pre-training on large-scale medical image-text pairs (such as images and titles in medical papers) to achieve cross-modal knowledge integration, which is then transferred to downstream question answering tasks. However, existing solutions suffer from the following key technical problems:

[0003] 1. Insufficient modal interaction in the pre-training stage: Existing models only mine modal associations through single-modal / multimodal contrastive loss, and do not make full use of the attention mechanism between vision and text, resulting in a lack of fine-grained interaction information between image sub-regions and text keywords, and limited cross-modal representation quality;

[0004] 2. Optimization of prompts relies on manual design: In order to adapt to downstream tasks, existing prompt learning schemes (such as MPR) require manual design of text prompts, which is not only time-consuming but also requires professional medical knowledge, making it difficult to guarantee the optimality of prompts, and at the same time, it cannot adapt to the feature differences of visual modalities.

[0005] 3. Limited effectiveness of knowledge editing: Existing knowledge editing technologies mostly adopt the "input editing" mode, which guides the model output by injecting knowledge into the input text. However, this mode is limited by the length of the input text and relies on the model's ability to understand new knowledge from complex inputs, resulting in poor robustness in medical knowledge-intensive scenarios.

[0006] Therefore, there is an urgent need for a medical visual question answering solution that can enhance modal interaction, automate prompting and optimization, and efficiently edit knowledge, in order to improve the accuracy and practicality of the model. Summary of the Invention

[0007] The purpose of this application is to provide a medical question-answering method and related device based on dynamic prompts and decoded knowledge editing, which can improve the accuracy of medical visual question answering.

[0008] To achieve the above objectives, this application provides the following solution:

[0009] Firstly, this application provides a medical question-answering method based on dynamic prompts and decoded knowledge editing, comprising the following steps:

[0010] Obtain a medical visual question-answering dataset; the medical visual question-answering dataset includes several medical visual question-answering tasks; the medical visual question-answering tasks include medical images and corresponding question-answering text; the question-answering text includes question text and answer text.

[0011] Based on a medical visual question answering dataset, a contrastive learning framework was used for visual language pre-training to obtain a pre-trained medical visual language model. The medical visual language model includes a visual encoder, a text encoder, and a multimodal fusion encoder. The minimization objectives of the pre-training include local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss.

[0012] For any medical visual question-answering task, dynamic cues are added to the visual features of the medical image and the text features of the question-answering text, respectively, to obtain visual features with dynamic cues and text features with dynamic cues.

[0013] Visual features with dynamic cues and text features with dynamic cues are input into a pre-trained medical visual language model to obtain an initial predicted answer.

[0014] The initial predicted answer is decoded and edited. An external knowledge base is introduced through a retrieval and verification mechanism to correct the model output and determine the final predicted answer. The external knowledge base is a collection of answer texts for all medical vision question-answering tasks. The retrieval and verification mechanism includes: selecting a set of candidate answers from the external knowledge base based on the text type of the question-answer pair text corresponding to the initial predicted answer, and verifying several candidate answers with similarity higher than a preset similarity threshold. The candidate answer is the answer text with the same text type as the question-answer pair text.

[0015] Secondly, this application provides a medical question-answering system based on dynamic prompts and decoded knowledge editing, including the following functional modules:

[0016] The dataset acquisition module is used to acquire a medical visual question-answering dataset. The medical visual question-answering dataset includes several medical visual question-answering tasks. Each medical visual question-answering task includes medical images and corresponding question-answering text pairs. The question-answering text pairs include question text and answer text.

[0017] The model pre-training module is used to pre-train the visual language based on the medical visual question answering dataset using a contrastive learning framework to obtain a pre-trained medical visual language model. The medical visual language model includes a visual encoder, a text encoder, and a multimodal fusion encoder. The minimization objectives of the pre-training include local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss.

[0018] The dynamic prompting module is used to add dynamic prompts to the visual features of medical images and the text features of question-and-answer pairs for any medical visual question-and-answer task, resulting in visual features with dynamic prompts and text features with dynamic prompts.

[0019] The initial answer generation module is used to input visual features with dynamic cues and text features with dynamic cues into the pre-trained medical visual language model to obtain the initial predicted answer.

[0020] The decoding knowledge editing module is used to decode and edit the initial predicted answer. It introduces an external knowledge base through a retrieval and verification mechanism to correct the model output and determine the final predicted answer. The external knowledge base is a collection of answer texts for all medical vision question-answering tasks. The retrieval and verification mechanism includes: filtering candidate answer sets from the external knowledge base based on the text type of the question-answer pair text corresponding to the initial predicted answer, and verifying several candidate answers with similarity higher than a preset similarity threshold. The candidate answer is the answer text with the same text type as the question-answer pair text.

[0021] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the medical question-and-answer method based on dynamic prompts and decoded knowledge editing described above.

[0022] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the medical question-and-answer method based on dynamic prompts and decoded knowledge editing described above.

[0023] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the medical question-and-answer method based on dynamic prompts and decoded knowledge editing described above.

[0024] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0025] This application provides a medical question-answering method and related apparatus based on dynamic prompts and decoded knowledge editing. The method first acquires a dataset containing medical images and corresponding question-answer pairs to provide high-quality, domain-adaptive medical data support for subsequent pre-training, dynamic prompt adaptation, and knowledge editing. Then, a medical visual language model is pre-trained with the goal of minimizing local contrast loss, global contrast loss, image-text matching loss, and masked language modeling loss. The combination of local and global alignment losses addresses the problem of insufficient modal interaction during the pre-training stage in existing technologies, significantly improving cross-modal representation quality. Image-text matching loss strengthens the model's judgment of the semantic matching between medical images and text, avoiding interference from unpaired data. Masked language modeling loss allows the model to combine medical images to assist in predicting masked medical terms, enhancing medical language understanding capabilities. Ultimately, the pre-trained model possesses superior cross-modal medical semantic understanding capabilities, providing high-quality representation support for downstream question-answering task transfer.

[0026] Subsequently, by adding dynamic cues to both the visual and textual modalities, the need for manually designing professional medical cues is eliminated, saving significant optimization time and expertise costs. This addresses the problem of existing technologies where "cue optimization relies on manual design." Features with dynamic cues guide the model to accurately focus on key information relevant to the medical task. Combined with high-quality cross-modal representations learned during pre-training, this effectively reduces initial prediction bias caused by inappropriate cues or insufficient modal interaction, making the initial predicted answers more aligned with the core needs of the medical problem. Finally, knowledge editing is employed in the decoding stage, rather than the "input editing" of existing technologies. This avoids the limitation of input text length and solves the problem of "limited effectiveness of knowledge editing." Filtering candidate answers by text type narrows the search scope and improves the accuracy of medical knowledge introduction. Validation of highly similar candidate answers ensures that the introduced knowledge conforms to medical standards, avoiding erroneous outputs due to insufficient medical knowledge or misunderstandings on the model's part. Simultaneously, the external knowledge base, built from the answer texts of all tasks, can be expanded with data updates, ensuring the timeliness and completeness of knowledge. Ultimately, this significantly improves the accuracy and medical compliance of the model's output answers and enhances the model's robustness in medical knowledge-intensive scenarios. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating a medical question-and-answer method based on dynamic prompts and decoded knowledge editing, provided as an embodiment of this application.

[0029] Figure 2 This is a flowchart of step A2 in a medical question-answering method based on dynamic prompts and decoded knowledge editing, provided as an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of the MedVL pre-training process in a medical question-answering method based on dynamic prompts and decoded knowledge editing, provided as an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of the dynamic prompting and decoding knowledge editing process in a medical question-answering method based on dynamic prompting and decoding knowledge editing, provided as an embodiment of this application.

[0032] Figure 5 This is a framework diagram for adding dynamic prompts in a medical question-answering method based on dynamic prompts and decoded knowledge editing, provided as an embodiment of this application.

[0033] Figure 6 This application provides an embodiment of a medical question-answering method based on dynamic prompts and decoded knowledge editing to generate a candidate answer list. A A schematic diagram.

[0034] Figure 7 This is a schematic diagram of the functional modules of a medical question-and-answer system based on dynamic prompts and decoded knowledge editing, provided as an embodiment of this application.

[0035] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] Inspired by pre-training work on visual language, recent studies have explored its applications in the low-resource, knowledge-intensive medical field. Medical Visual Language (MedVL) models, pre-trained on medical image-text pairs, incorporate external knowledge and demonstrate good portability in downstream medical visual question answering tasks, becoming an effective approach for learning multimodal representations. Existing pre-trained medical visual language models for medical visual question answering tasks suffer from limited learning of unimodal and multimodal features during the pre-training phase. Previous studies have introduced unimodal and multimodal contrastive losses for masked image and text modeling during the pre-training phase. However, this study finds that the attention mechanism between visual and textual modalities is more beneficial for solving medical visual question answering tasks, and the interaction information between modalities mined solely using contrastive losses is insufficient.

[0038] When large-scale medical visual language models like MedVL are used for downstream tasks, the results of directly fine-tuning the model are not ideal due to its large number of parameters and the relatively small size of medical visual question-answering datasets. Inspired by work on cue learning, recent studies have explored applying cue tuning to adapt MedVL models for downstream tasks. However, current cue tuning efforts still rely on manually designing cue words for text input. For pre-trained medical visual language models, text input plays a crucial role in downstream tasks, and manually designing cue words typically requires significant time for tuning and cannot guarantee optimal performance.

[0039] In addition, knowledge editing technology for large language models has flourished in recent years. Its aim is to effectively modify the behavior of large language models in specific domains while maintaining overall performance across various inputs. Therefore, it is possible to leverage knowledge editing to better transfer general-domain pre-trained language models to downstream tasks related to the medical field. Simultaneously, knowledge editing can provide knowledge guidance and supplementation for answering downstream medical visual question-answering tasks, thereby addressing the performance issues caused by insufficient medical knowledge learned by current generative medical visual question-answering models.

[0040] To address the challenges in the aforementioned research, this application pre-trains a medical visual language model on a large-scale medical image question-answering dataset, enabling the extraction of richer representational information for medical visual question-answering tasks. It also proposes a training process based on Dynamic Prompts and Decoding Knowledge Editing (DPDE), which employs dynamic and learnable prompt optimization, avoiding the time-consuming nature of traditional prompt optimization and the need for professional knowledge to manually design prompt statements. This approach combines the latest relevant technologies and addresses related issues in three aspects: (1) Introducing an attention mechanism in the pre-training stage of the medical visual language model to achieve global and local comparative learning of image-text pairs, thereby deeply mining the interactive information between vision and text; (2) Proposing an efficient and concise dynamic prompting method for automated prompting engineering of visual and text modules, particularly applicable to pre-trained medical visual language models; (3) Exploring a new paradigm of knowledge editing in decoding, performing knowledge editing operations in the output stage, so that the output of the language model can fully consider and combine the input knowledge information. In this embodiment, the medical visual question-answering task is defined as a generation task, using the fusion features of medical images and questions as input to the generator to generate the final predicted answer.

[0041] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] This application provides a medical question-answering method based on dynamic prompts and decoded knowledge editing. In one exemplary embodiment, such as... Figure 1 As shown, it includes the following steps:

[0043] A1. Obtain the medical visual question-answering dataset; the medical visual question-answering dataset includes several medical visual question-answering tasks; the medical visual question-answering tasks include medical images and corresponding question-answering text; the question-answering text includes question text and answer text.

[0044] A2. Based on a medical visual question-answering dataset, a contrastive learning framework is used for visual language pre-training to obtain a pre-trained medical visual language model; the medical visual language model MedVL includes a visual encoder. E v Text encoder E t and multimodal fusion encoder E m All three encoders are transformer-based modules; the visual encoder E v and text encoder Et Deep semantic features of medical images and textual semantic information of medical questions are extracted independently, followed by a multimodal fusion encoder. E m Fine-grained associations are established between image regions and text keywords, and finally, these associations are fused to generate joint representations to produce predicted answers. The minimization objectives of pre-training include local contrast loss, global contrast loss, image-text matching loss, and masked language modeling loss. In this embodiment, as... Figure 2 As shown, step A2 specifically includes the following steps:

[0045] A21. Input the medical image into the visual encoder and output the image embedding. Input the question-and-answer pair text into the text encoder and output the text embedding.

[0046] A22. Based on image embedding and text embedding, calculate local contrast loss and global contrast loss respectively. Inspired by the GLoRIA model, incorporating local and global contrast losses into the pre-training task is more conducive to learning global and local features by comparing paired image sub-regions and words. Therefore, we adopt the approach in GLoRIA to design local and global contrast losses. To learn local representations, a similarity matrix is ​​calculated based on image sub-region features and word-level features to generate attention-weighted image representations. The local contrast target is based on the attention-weighted image representation and the corresponding word representation. The local contrast loss is calculated according to the following formula:

[0047]

[0048] in, For local contrast loss, log e ( ) represents the natural constant. e Logarithmic function with base 0. N This represents the number of medical vision question-answering tasks included in the current training batch. i and j For labeling; l vi For the first i Local visual features of a medical image l ti For the first i Local textual features of the question and answer text. P ( , ) represents the similarity score between local features. T 1 represents the temperature parameter.

[0049] The global comparison loss is calculated using the following formula:

[0050]

[0051] in, For global comparison loss, g vi For the first i Global visual features of a medical image g ti For the first i The global text features of the question-and-answer pair text, where <,> represents the vector inner product.

[0052] A23. Input the image embedding and text embedding into a multimodal fusion encoder, output fused features, and construct an image-text matching task based on the fused features, calculating the image-text matching loss. Following the image-text matching (ITM) strategy designed in most existing works, create a binary classification task with negative text labels randomly sampled from the same mini-batch. Use the fused features output from the multimodal fusion encoder as input to the binary classification head. The ITM task is optimized using cross-entropy loss, and the image-text matching loss is calculated according to the following formula:

[0053]

[0054] in, For image-text matching loss, Represents the mathematical expectation operation. V and T From the dataset respectively D A medical image and question-answer pair text sampled for a medical visual question-answering task. D For medical visual question answering datasets, CE (,) is the cross-entropy function. y label For real labels, p label ( , ) represents the matching probability predicted by the model.

[0055] A24. Randomly mask and replace the text embeddings with mask tags according to a preset probability. Input the image embeddings and the text embeddings containing mask tags into a multimodal fusion encoder, output the predicted probability of the mask tags, and calculate the mask language modeling loss based on the predicted probability. Mask language modeling predicts masking tags in the text based on visual and unmasked context information. For each title text, randomly mask words in the text with a 15% probability and replace them with special word segmentation tags. MASK By embedding images g v With sentence embedding g t Connecting these components and inputting them into a multimodal fusion encoder yields the predicted probabilities of the masking markers at corresponding positions in the output sequence. The masking language modeling loss is calculated using the following formula:

[0056]

[0057] in, For masked language modeling loss, The question-and-answer pair text after being masked. y mask The actual word label at the masked location. p mask ( , ) represents the probability distribution of the masked words predicted by the model.

[0058] A25. Iteratively train the model based on the total training loss until it converges, obtaining a pre-trained medical visual language model; the pre-training process is as follows: Figure 3 As shown, the total training loss is the sum of the local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss. The final total training loss... for:

[0059]

[0060] After obtaining a medical visual language model through pre-training in step A2, the following process, based on dynamic prompting and decoding knowledge editing as proposed in this application, is used to process the medical visual question answering task. Figure 4 As shown, the steps are as follows (A3-A5).

[0061] A3. For any medical visual question-answering task, dynamic cues are added to the visual features of the medical image and the text features of the question-answering text, respectively, to obtain visual features with dynamic cues and text features with dynamic cues. To enable the pre-trained model MedVL to perform better in medical visual question-answering tasks, this embodiment introduces text and visual cues into the model through dynamic cues. The framework diagram is as follows: Figure 5 As shown, dynamic prompts model the context for both text and visual modules by using continuous vectors learned from the data, while freezing a large number of pre-trained parameters, thus avoiding manual prompt tuning.

[0062] 1) Dynamic Text Hints: In this embodiment, dynamic hints are added to the text input. This avoids the significant performance impact of minor changes in the wording of the question template and effectively transforms the pre-trained MedVL model into a data-efficient visual learner. Specifically, adding dynamic hints to the text features of the question-and-answer pair involves constructing dynamic hint vectors for the text features. ,in, L The hyperparameter for the number of word segments in the dynamic cue vector of text features, each... V l For learnable vectors with the same dimension as word embeddings, l ∈(1,..., LThe input to the first layer of the text encoder is... ,in For the embedding of the [CLS] tokenizer, superscript d For embedded dimensions, represent d The real space of dimension 1 ET 0 represents the word embedding of the question text.

[0063] By adding dynamic hints after the text question, the dynamic hints can supplement the answer to the text question with key information through self-learning, without the need for time-consuming manual design and optimization of hint templates.

[0064] 2) Visual Dynamic Cueing: Existing work has demonstrated the limitations of using only unimodal cues for tuning in multimodal tasks. Text cues fail on data with significant intra-class visual differences, while visual cues cannot handle data with small inter-class differences. To combine the advantages of both, this example adds dynamic cues to both the text and visual modal. Specifically, adding dynamic cues to the visual features of medical images involves dividing the medical image into m fixed-size image patches, obtaining an image patch embedding set. Construct a set of visual dynamic cue vectors The visual encoder i The input for layer +1 is ,in, This indicates the embedding of the [CLS] token in the (i+1)th level input space. For the set of natural numbers, For the first i The first layer output k Embedding vectors of image patches, i and k These are the levels of the visual encoder and the numbering of the image blocks, respectively. For the first i The first layer output j An embedding vector for a visual dynamic cue. j The visual cues are numbered, and each visual cue corresponds to a specific image block.

[0065] A4. Input the visual features with dynamic cues and the text features with dynamic cues into the pre-trained medical visual language model to obtain the initial predicted answer.

[0066] A5. Decode the initial predicted answer using knowledge editing, and use a retrieval and verification mechanism to introduce an external knowledge base to correct the model output and determine the final predicted answer. The external knowledge base is a collection of answer texts for all medical vision question-answering tasks. The retrieval and verification mechanism includes: selecting a set of candidate answers from the external knowledge base based on the text type of the question-answer pair text corresponding to the initial predicted answer, and verifying several candidate answers with similarity higher than a preset similarity threshold. The candidate answer is the answer text with the same text type as the question-answer pair text.

[0067] This embodiment transfers knowledge editing as a constrained decoding problem to the language model decoding stage of a medical vision question-answering task. Unlike traditional constrained decoding methods that access the output vocabulary distribution, the decoding-based knowledge editing method proposed in this embodiment retrieves relevant knowledge and verifies and modifies the final output during the decoding stage, achieving effective knowledge editing. This decoding-based knowledge editing method can be flexibly combined with black-box large language models that do not provide vocabulary access. On the one hand, decoding knowledge editing can correct the language model's output answer to the desired ideal answer when it deviates from the ideal answer; on the other hand, when the answer changes and needs to be supplemented and modified, it can also adjust the language model to the updated answer with a small amount of work.

[0068] like Figure 4 As shown, knowledge editing mainly consists of two stages: search and verification. Taking a medical visual question answering task as an example, given a medical image... S v And a medical problem S q Assuming the original language model used for decoding is Ideally, the language model should provide a perfectly matching answer as output. However, the answers generated by language models may deviate from the ideal answers. Decoding knowledge editing pre-generates answers using language models, and during the search phase, it calculates the similarity between the generated answers and candidate answers, selecting the one with the highest similarity. k Several candidate answers. These will be used in the validation phase. k Each candidate answer is assumed to be the true answer. The decoder is used to calculate the matching degree between the candidate answer and the image text pair. The candidate answer with the highest matching degree is selected as the final predicted answer.

[0069] In one specific embodiment, step A5 specifically includes the following steps:

[0070] A51. Input the question-answer pair texts corresponding to the initial predicted answers into the text encoder and classifier sequentially. G Get the problem type label L q and answer type tags La .

[0071] A52, According to the question type label L q and answer type tags L a From external knowledge base A The answer texts with the same question type and answer type are filtered out to form a candidate answer set. A Generate a set of candidate answers. A The process is as follows Figure 6 As shown.

[0072] A53. Use cosine similarity or Euclidean distance to calculate the initial predicted answer and candidate answer set. A The similarity between the candidate answers.

[0073] A54. For several candidate answers with similarity higher than a preset similarity threshold, the decoder is used to calculate the matching degree between each candidate answer and the answer text, and the candidate answer with the highest matching degree is taken as the final predicted answer.

[0074] External knowledge base A From all the training data, all the answer texts in the training data are used as an external knowledge base. When the knowledge in the training data changes or is added, the external knowledge base... A It can be updated and expanded accordingly.

[0075] In another exemplary embodiment of this application, the medical visual language model MedVL is pre-trained on two datasets: ROCO and PMC-OA. The ROCO dataset contains 87,952 image-question-answer pairs. PMC-OA is a high-quality image-text biomedical dataset built based on scientific research papers, containing 1.65M image-text pairs. PMC-CLIP experimentally demonstrated that PMC-OA outperforms large-scale biomedical datasets; therefore, this embodiment uses both ROCO and PMC-OA during MedVL pre-training. For downstream medical visual question-answering tasks, this embodiment conducts experiments on three publicly available medical visual question-answering datasets, including VQA-RAD, SLAKE, and OVQA. VQA-RAD has 315 radiological images and 3,064 question-answer pairs, of which 451 pairs are used for testing. SLAKE has 14,028 sample pairs, which are further divided into a 70% training subset, a 15% validation subset, and a 15% test subset. The OVQA dataset contains 2001 images and 19,020 question-answer pairs.

[0076] The experiment selected traditional medical visual question answering models, CLIP-style pre-trained models, and models using cue-based learning as benchmark methods, specifically including MMQ, MMBERT, MTPT-CMSA, M2I2, M3AE, PMC-CLIP, and PrefixT. Medical LM. Among them, MMQ, MMBERT, and MTPT-CMSA are traditional medical visual question answering models; M2I2, M3AE, and PMC-CLIP are designed with CLIP-style medical visual language pre-training; and PMC-CLIP and PrefixT. Medical LM use cue-based learning for transfer to downstream medical visual question answering tasks.

[0077] MMQ adds metadata through automatic annotation, handles noisy labels, and outputs a meta-model, providing robust features for medical VQA tasks. MMBERT is pre-trained using the ROCO dataset and uses image features from Med-VQA for masked language modeling. MTPT-CMSA redefines image feature pre-training as a multi-task learning paradigm. M2I2 uses masked language modeling, masked image modeling, image-text matching, and image-text contrast learning as pre-training objects, learning unimodal and multimodal feature representations of input images and text through an image caption dataset, and then transferring them to downstream Med-VQA tasks in a data-efficient manner. M3AE proposes a self-supervised learning paradigm with a multimodal masked autoencoder, which learns cross-modal domain knowledge by reconstructing missing pixels and labels from images and text with random masks. Specifically, M3AE uses different masking ratios for the input images and text, with a larger masking ratio for images. It then utilizes different levels of visual and textual features for reconstruction to handle different levels of abstraction in vision and language, while developing different designs for the visual and language decoders. PMC-CLIP uses a self-constructed image caption dataset for pre-training, designs a visual language model in the CLIP style, and employs image-text contrastive learning and masked language modeling as training objectives. In transferring to the medical visual question answering task, PMC-CLIP introduces cue learning. To leverage the pre-trained language model, Prefix T. Medical LM converts medical images into a set of learnable labels using a small-scale mapping network. These labels can then be interpreted as visual prefixes for the language model. Finally, the visual prefixes, along with the question, are used as input to the language model for answer prediction.

[0078] Extensive experiments were conducted on three publicly available datasets: VQA-RAD, SLAKEN-EN, and OVQA. Performance analysis results are shown in Table 1. Following existing practices, the DPDE proposed in this application selects Accuracy as the evaluation metric.

[0079] Table 1 Experimental Results of DPDE Model Performance Analysis

[0080]

[0081] Analysis of the experimental results in Table 1 shows that the DPDE model proposed in this application achieves the best performance in overall accuracy and closed-ended questions on the three datasets VQA-RAD, SLAKE-EN, and OVQA, and improves by 2.3%, 2.3%, and 7.7% respectively compared to the state-of-the-art methods. This indicates that the DPDE model can make full use of dynamic prompts and decoding knowledge editing methods to effectively learn medical image features and understand related questions, thereby improving the accuracy of medical visual question answering tasks. In addition, further analysis of the experimental results shows that: (1) Visual language models (M2I2, M3AE, and PMC-CLIP) pre-trained using the Transformer architecture have better performance in medical visual question answering tasks than traditional medical visual question answering methods (MMQ, MMBERT, and MTPT-CMSA). This indicates that compared to meta-learning, pre-trained visual language models can better learn cross-modal representations by leveraging external medical image-text pairs, thereby transferring them to downstream medical visual question answering tasks to solve the problem of data limitations. (2) In the medical visual question answering methods using CLIP-style pre-training, DPDE and PMC-CLIP further incorporate cue learning in downstream medical visual question answering tasks, demonstrating better performance than M2I2 and M3AE. This also shows that cue learning can further enhance representation learning in medical visual question answering tasks.

[0082] In another exemplary embodiment of this application, an ablation experiment is conducted to remove a component of the DPDE model, and the impact on overall performance is compared to verify the effectiveness of each component. The results of the ablation experiment are shown in Table 2. Compared to the baseline DPDE model (i.e., text dynamic prompts + visual dynamic prompts + decoded knowledge editing), integrating text dynamic prompts, visual dynamic prompts, or decoded knowledge editing can significantly improve performance on medical visual question answering tasks. Notably, simultaneously integrating these three components achieves performance improvements of 2.6%, 1.1%, and 2.5% on the VQA-RAD, SLAKE-EN, and OVQA datasets, respectively.

[0083] Table 2. Ablation Experiment Results of DPDE Model

[0084]

[0085] Analysis of the experimental results of the first to third structures revealed that: (1) Dynamic prompts can effectively improve the performance of medical visual question answering tasks; for different datasets, the degree to which text dynamic prompts and visual dynamic prompts play a role is different, which is mainly related to inter-class differences. (2) Decoding knowledge editing plays a prominent role in improving medical visual question answering tasks. It can improve the model's utilization of new knowledge in the decoding stage, thereby avoiding the problem that input-type knowledge editing largely depends on large language models to understand new knowledge from complex input text.

[0086] Based on the same inventive concept, this application also provides a system for implementing the above-described medical question-answering method based on dynamic prompts and decoded knowledge editing. The solution provided by this system is similar to the implementation described in the above method. In an exemplary embodiment, such as... Figure 7 As shown, a medical question-answering system based on dynamic prompts and decoded knowledge editing is provided, including the following functional modules:

[0087] The dataset acquisition module is used to acquire a medical visual question-answering dataset. The medical visual question-answering dataset includes several medical visual question-answering tasks. Each medical visual question-answering task includes medical images and corresponding question-answering text pairs. The question-answering text pairs include question text and answer text.

[0088] The model pre-training module is used to pre-train the visual language based on the medical visual question answering dataset using a contrastive learning framework to obtain a pre-trained medical visual language model. The medical visual language model includes a visual encoder, a text encoder, and a multimodal fusion encoder. The minimization objectives of the pre-training include local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss.

[0089] The dynamic prompting module is used to add dynamic prompts to the visual features of medical images and the text features of question-and-answer pairs for any medical visual question-and-answer task, resulting in visual features with dynamic prompts and text features with dynamic prompts.

[0090] The initial answer generation module is used to input visual features with dynamic cues and text features with dynamic cues into the pre-trained medical visual language model to obtain the initial predicted answer.

[0091] The decoding knowledge editing module is used to decode and edit the initial predicted answer. It introduces an external knowledge base through a retrieval and verification mechanism to correct the model output and determine the final predicted answer. The external knowledge base is a collection of answer texts for all medical vision question-answering tasks. The retrieval and verification mechanism includes: filtering candidate answer sets from the external knowledge base based on the text type of the question-answer pair text corresponding to the initial predicted answer, and verifying several candidate answers with similarity higher than a preset similarity threshold. The candidate answer is the answer text with the same text type as the question-answer pair text.

[0092] certainly, Figure 7 The architecture shown is merely exemplary; it can be omitted as needed when implementing different functionalities. Figure 7 One or at least two components of the system shown.

[0093] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 8 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it can implement the medical question-and-answer method based on dynamic prompts and decoded knowledge editing provided in the previous embodiment.

[0094] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0095] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0096] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0097] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0099] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A medical question-answering method based on dynamic prompts and decoded knowledge editing, characterized in that, include: Obtain a medical visual question-answering dataset; the medical visual question-answering dataset includes several medical visual question-answering tasks; The medical visual question-answering task includes medical images and question-answer pairs corresponding to the medical images; the question-answer pairs include question text and answer text. Based on the aforementioned medical visual question-answering dataset, a contrastive learning framework is used for visual language pre-training to obtain a pre-trained medical visual language model. The medical visual language model includes a visual encoder, a text encoder, and a multimodal fusion encoder. The minimization objectives of the pre-training include local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss. For any medical visual question-answering task, dynamic prompts are added to the visual features of the medical image and the text features of the question-answering text, respectively, to obtain visual features with dynamic prompts and text features with dynamic prompts. The visual features with dynamic cues and the text features with dynamic cues are input into the pre-trained medical visual language model to obtain the initial predicted answer; The initial predicted answer is decoded and knowledge-edited. An external knowledge base is introduced through a retrieval and verification mechanism to correct the model output and determine the final predicted answer. The external knowledge base is a collection of answer texts for all medical vision question answering tasks. The retrieval verification mechanism includes: filtering a set of candidate answers from an external knowledge base based on the text type of the question-and-answer pair text corresponding to the initial predicted answer, and verifying several candidate answers with a similarity higher than a preset similarity threshold; the candidate answer is the answer text with the same text type as the question-and-answer pair text; Adding dynamic hints to the text features of the question-and-answer text specifically involves constructing dynamic hint vectors for the text features. ,in, L The hyperparameter for the number of word segments in the dynamic cue vector of text features, each... V l For learnable vectors with the same dimension as word embeddings, l ∈(1,..., L The input of the first layer of the text encoder is... ,in For the embedding of the [CLS] tokenizer, superscript d For embedded dimensions, represent d The real space of dimension 1 ET 0 represents the word embedding of the question text; Adding dynamic cues to the visual features of medical images specifically involves dividing the medical image into m fixed-size image blocks, resulting in an image block embedding set. Construct a set of visual dynamic cue vectors The visual encoder i The input for layer +1 is ,in, This indicates the embedding of the [CLS] token in the (i+1)th level input space. For the set of natural numbers, For the first i The first layer output k Embedding vectors of image patches, i and k These are the levels of the visual encoder and the numbering of the image blocks, respectively. For the first i The first layer output j An embedding vector for each visual dynamic cue, with each visual dynamic cue corresponding to a one-to-one image patch.

2. The medical question-answering method based on dynamic prompts and decoded knowledge editing according to claim 1, characterized in that, Based on the aforementioned medical visual question-answering dataset, a contrastive learning framework is used for visual language pre-training to obtain a pre-trained medical visual language model, specifically including: The medical image is input into a visual encoder, which outputs an image embedding, and the question-and-answer pair text is input into a text encoder, which outputs a text embedding. Based on the image embedding and the text embedding, the local contrast loss and the global contrast loss are calculated respectively; The image embedding and the text embedding are input into a multimodal fusion encoder, which outputs fusion features. An image-text matching task is constructed based on the fusion features, and the image-text matching loss is calculated. The text embedding is randomly masked and replaced with a mask marker according to a preset probability. The image embedding and the text embedding containing the mask marker are input into a multimodal fusion encoder, and the predicted probability of the mask marker is output. The mask language modeling loss is calculated based on the predicted probability. The model is iteratively trained based on the total training loss until it converges, resulting in a pre-trained medical visual language model. The total training loss is the sum of local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss.

3. The medical question-answering method based on dynamic prompts and decoded knowledge editing according to claim 2, characterized in that, The local contrast loss is calculated using the following formula: in, For local contrast loss, log e ( ) represents the natural constant. e Logarithmic function with base 0. N This represents the number of medical vision question-answering tasks included in the current training batch. i and j For labeling; l vi For the first i Local visual features of a medical image l ti For the first i Local textual features of the question and answer text. P ( , ) represents the similarity score between local features. T 1 represents the temperature parameter; The global comparison loss is calculated using the following formula: in, For global comparison loss, g vi For the first i Global visual features of a medical image g ti For the first i The global text features of the question-and-answer pairs, where < and > represent the vector inner product.

4. The medical question-answering method based on dynamic prompts and decoded knowledge editing according to claim 2, characterized in that, The image-text matching loss is calculated using the following formula: in, For image-text matching loss, Represents the mathematical expectation operation. V and T From the dataset respectively D A medical image and question-answer pair text sampled for a medical visual question-answering task. D For medical visual question answering datasets, CE ( , ) is the cross-entropy function. y label For real labels, p label ( , ) represents the matching probability predicted by the model; The mask language modeling loss is calculated using the following formula: in, For masked language modeling loss, The question-and-answer pair text after being masked. y mask The actual word label at the masked location. p mask ( , ) represents the probability distribution of the masked words predicted by the model.

5. The medical question-answering method based on dynamic prompts and decoded knowledge editing according to claim 1, characterized in that, Based on the text type of the question-and-answer pair text corresponding to the initial predicted answer, a set of candidate answers is obtained by filtering from an external knowledge base. Several candidate answers with a similarity higher than a preset similarity threshold are then verified, specifically including: The question-answer pair text corresponding to the initial predicted answer is sequentially input into the text encoder and classifier. G Get the problem type label L q and answer type tags L a ; Based on question type tags L q and answer type tags L a From external knowledge base A The answer texts with the same question type and answer type are filtered out to form a candidate answer set. A ; The initial predicted answer and the candidate answer set are calculated using cosine similarity or Euclidean distance. A The similarity between the candidate answers; For several candidate answers with a similarity higher than a preset similarity threshold, a decoder is used to calculate the matching degree between each candidate answer and the answer text, and the candidate answer with the highest matching degree is taken as the final predicted answer.

6. A medical question-answering system based on dynamic prompts and decoded knowledge editing, characterized in that, include: The dataset acquisition module is used to acquire a medical visual question-answering dataset; the medical visual question-answering dataset includes several medical visual question-answering tasks; The medical visual question-answering task includes medical images and question-answer pairs corresponding to the medical images; the question-answer pairs include question text and answer text. The model pre-training module is used to perform visual language pre-training based on the medical visual question-answering dataset using a contrastive learning framework to obtain a pre-trained medical visual language model. The medical visual language model includes a visual encoder, a text encoder, and a multimodal fusion encoder. The minimization objectives of the pre-training include local contrast loss, global contrast loss, image-text matching loss, and mask language modeling loss. The dynamic prompt addition module is used to add dynamic prompts to the visual features of medical images and the text features of question-and-answer pairs for any medical visual question-and-answer task, resulting in visual features with dynamic prompts and text features with dynamic prompts. The initial answer generation module is used to input the visual features with dynamic prompts and the text features with dynamic prompts into the pre-trained medical visual language model to obtain the initial predicted answer; The decoding knowledge editing module is used to decode and edit the initial predicted answer, and to correct the model output by introducing an external knowledge base through a retrieval and verification mechanism to determine the final predicted answer; the external knowledge base is a collection of answer texts for all medical vision question answering tasks. The retrieval verification mechanism includes: filtering a set of candidate answers from an external knowledge base based on the text type of the question-and-answer pair text corresponding to the initial predicted answer, and verifying several candidate answers with a similarity higher than a preset similarity threshold; the candidate answer is the answer text with the same text type as the question-and-answer pair text; Adding dynamic hints to the text features of the question-and-answer text specifically involves constructing dynamic hint vectors for the text features. ,in, L The hyperparameter for the number of word segments in the dynamic cue vector of text features, each... V l For learnable vectors with the same dimension as word embeddings, l ∈(1,..., L The input of the first layer of the text encoder is... ,in For the embedding of the [CLS] tokenizer, superscript d For embedded dimensions, represent d The real space of dimension 1 ET 0 represents the word embedding of the question text; Adding dynamic cues to the visual features of medical images specifically involves dividing the medical image into m fixed-size image blocks, resulting in an image block embedding set. Construct a set of visual dynamic cue vectors The visual encoder i The input for layer +1 is ,in, This indicates the embedding of the [CLS] token in the (i+1)th level input space. For the set of natural numbers, For the first i The first layer output k Embedding vectors of image patches, i and k These are the levels of the visual encoder and the numbering of the image blocks, respectively. For the first i The first layer output j An embedding vector for each visual dynamic cue, with each visual dynamic cue corresponding to a one-to-one image patch.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the medical question-and-answer method based on dynamic prompts and decoded knowledge editing as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the medical question-and-answer method based on dynamic prompts and decoded knowledge editing as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the medical question-and-answer method based on dynamic prompts and decoded knowledge editing as described in any one of claims 1-5.

Images