A conflict protection knowledge editing method for a medical visual question and answer model

Abstract

The application discloses a conflict protection knowledge editing method of a medical visual question and answer model. The method freezes model main body parameters, sets lightweight editable parameters, screens target editing samples and constructs directional medical conflict samples; calculates target editing gradients and conflict gradients, estimates a conflict sensitive subspace corresponding to a current editing task by using the conflict gradients; projects the target editing gradients to the subspace to obtain a conflict risk component, adaptively determines a protection strength according to an overlapping degree and attenuates the component to obtain a safe editing gradient; and fuses and updates the lightweight editable parameters by using the safe editing gradient and a reference answer anchor gradient. The application can identify potential conflict risks and perform targeted protection, correct target errors and effectively avoid conflict drift.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing, natural language processing, and artificial intelligence, specifically to a conflict-preserving knowledge editing method for a medical visual question-answering model. Background Technology

[0002] Medical Visual Question Answering (MedVQA) is an important technology that integrates medical image analysis, natural language processing, and multimodal artificial intelligence. In recent years, Vision-Language Models (VLMs) have shown strong potential in medical image understanding and clinical question answering. However, limitations in the coverage of medical training data, domain expertise, and image quality variations mean that models may still produce incorrect predictions on some medical question-answering samples. Knowledge editing techniques aim to enable models to output new target answers for given medical images and questions without retraining all model parameters. Among these, parameter-efficient fine-tuning methods (such as low-rank fitting) can significantly reduce the computational and storage overhead of knowledge updates by freezing the main model parameters while updating only a small number of additional lightweight parameters.

[0003] However, existing lightweight parametric editing methods typically focus only on optimizing the correctness of the target edited sample, failing to identify and limit the directional impact of the editing gradient on neighboring medical semantic samples. In medical applications, different question-answering samples often share highly similar contexts (such as the same question template, organ location, imaging modality, or candidate answer space), but due to differences in specific medical image evidence, their reference answers must remain distinct. When a model is edited to correct a target sample, the target answer is highly susceptible to directional propagation along the same medical relationships and answer space, leading to the incorrect assimilation of originally correctly predicted neighboring conflicting samples. This phenomenon of the target edited answer propagating directionally to non-target medical samples and causing knowledge contamination is known as conflict drift.

[0004] To address the aforementioned problem of non-target knowledge contamination, existing technologies typically employ random, irrelevant samples or a unified global constraint space (such as a global null space or subspace constraints) to limit parameter updates. However, the correlation between random samples and the current editing target is weak, making it difficult to reflect the directional propagation risk in scenarios with high similarity, such as those involving medical relationships or answer spaces. More limitingly, the conflict drift risk in medical visual question answering is usually determined by the relationship between the current specific editing target and specific medical protection samples. This means that the risk gradient directions corresponding to different editing samples are often dynamic and different. Existing methods, with their pre-fixed or uniformly constructed global protection spaces, lack editing specificity and cannot accurately express and suppress the directional risk of a specific medical target answer propagating to specific conflict samples. Therefore, it is difficult to provide targeted and precise protection for each independent editing task.

[0005] Therefore, how to overcome the shortcomings of existing global constraints or the lack of editing specificity of random samples when using lightweight editable parameters for knowledge editing in medical visual question answering models, and how to identify and protect the unique potential conflict risks in each independent editing task, so as to successfully correct the target medical errors while minimizing the occurrence of conflict drift in non-target medical samples, is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address at least one of the technical deficiencies mentioned in the background art, the present invention aims to provide a conflict-preserving knowledge editing method for a medical visual question-answering model, an electronic device, a computer-readable storage medium, and a computer program product.

[0007] This invention provides a conflict-preserving knowledge editing method for a medical visual question-answering model, the method comprising: A medical visual question answering model was established. Lightweight editable parameters were set while freezing the main parameters of the medical visual question answering model, and basic prediction results of medical question answering samples were obtained. Based on the basic prediction results, target editing samples to be corrected are screened, and medical conflict samples with directional correlation corresponding to the target editing samples are constructed based on medical structural information. Calculate the target editing gradient based on the target editing sample and the conflict gradient based on the medical conflict sample, respectively, and use the conflict gradient to estimate the conflict-sensitive subspace corresponding to the current target editing sample; Projecting the target editing gradient into a conflict-sensitive subspace yields a conflict hazard component. The degree of overlap between the target editing gradient and the conflict-sensitive subspace is calculated to adaptively determine the protection strength. Based on the protection strength, the conflict hazard component in the target editing gradient is attenuated to obtain a safe editing gradient. The secure editing gradient is fused with the reference answer anchoring gradient for the medical conflict sample to obtain the final update gradient, and the lightweight editable parameter is updated using the final update gradient.

[0008] The present invention also provides an electronic device, the electronic device including a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method as described in any of the preceding claims.

[0009] The present invention also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method as described in any of the preceding claims.

[0010] The present invention also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to perform the method as described in any of the preceding claims.

[0011] This invention constructs medical conflict samples with directional correlation to the target editing sample by freezing the main parameters of the model and introducing lightweight editable parameters. Based on this, it dynamically estimates the conflict-sensitive subspace specifically corresponding to the current editing task, adaptively determines the protection strength based on the gradient overlap, and projects the subspace. Finally, it fuses the safe editing gradient after attenuating the dangerous components with the anchored gradient of the reference answer to complete the parameter update. This invention effectively overcomes the shortcomings of existing random samples or global constraints that lack editing specificity, achieving the identification and targeted protection of potential conflict risks unique to each independent editing task. This allows for successful correction of target medical errors with low overhead while minimizing conflict drift in non-target medical samples. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall process of a conflict-preserving knowledge editing method for a medical visual question-answering model disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the structural relationship between the target editing sample and the directional medical sample (including four types of conflict samples, generalized samples, and local samples) in an embodiment of the present invention. Figure 3 This is a flowchart of the conflict-sensitive subspace estimation module in an embodiment of the present invention, which shows the complete process from the calculation of the original conflict gradient, normalized weighting, Gram matrix construction and eigenvalue decomposition to the generation of the conflict-sensitive subspace; Figure 4This is a flowchart illustrating the process of adaptive editing gradient projection and protection gradient fusion in an embodiment of the present invention. It shows the complete process from the target editing gradient through conflict-sensitive subspace projection, conflict hazard component identification, adaptive protection strength determination, hazard component attenuation, to the fusion of the reference answer anchoring gradient and the locality-preserving gradient. Figure 5 This is a schematic diagram comparing the structure of the training / editing phase and the inference phase in an embodiment of the present invention. It shows the operations such as loading conflicting samples, calculating gradients, estimating subspaces and projecting during the training / editing phase, as well as the simplified process of loading only lightweight editable parameters and performing forward computation during the inference phase. Detailed Implementation

[0013] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the specific implementation methods of the conflict-protected knowledge editing method for the medical visual question-answering model provided by this invention will be fully described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0015] Specifically, please refer to Figure 1 This invention provides a conflict-preserving knowledge editing method for a medical visual question-answering model, the method comprising the following steps: Step 1: Establish a medical visual question answering model, set lightweight editable parameters while freezing the main parameters of the medical visual question answering model, and obtain the basic prediction results of medical question answering samples; This step aims to establish a medical visual question-answering model that can be edited later, and to obtain the model's initial prediction performance on various medical question-answering samples, so as to provide a unified and reliable judgment benchmark for subsequent identification of erroneous samples that need to be corrected.

[0016] In practical implementation, the first step is to establish a medical visual question-answering model, which is represented as follows:

[0017] in, This refers to the input medical images, which may include, but are not limited to, X-ray, CT, MRI, ultrasound, pathological sections, endoscopic images, etc. Representing natural language medical questions corresponding to medical images; Let represent the set of candidate answers. For open-ended questions, let ; These represent the main parameters of the medical visual question-answering model; This represents lightweight editable parameters that are allowed to be updated during the knowledge editing process.

[0018] Furthermore, the medical visual question-answering model may include a medical image visual encoder. Multimodal connection module and language decoder It should be understood that the above three modules form the basic architecture for the visual language model to perform medical visual question answering tasks. Its forward computation process can be represented as:

[0019]

[0020]

[0021] in, Visual features or visual tokens representing medical images; Indicated by medical issues With the set of candidate answers The resulting text prompt embedding, for open-ended question and answer commands It degenerates into question text embedding; This represents a multimodal representation that integrates medical image information and question text information. This indicates a medical answer that is yet to be scored or generated.

[0022] Maintaining main parameters during knowledge editing Freeze, only update lightweight editable parameters It should be noted that this strategy is adopted because full parameter fine-tuning requires updating a large number of parameters of the visual language model, which has problems such as high memory usage, large computational overhead, and high model storage costs, and may cause an overall change in the model's original medical question-answering ability; while lightweight parameter updates can achieve accurate writing of target knowledge while reducing training and storage overhead.

[0023] As an example, the lightweight editable parameter is a low-rank adaptation parameter, which is updated by adding a low-rank increment to the frozen pre-trained weight matrix.

[0024] Specifically, lightweight editable parameters employ low-rank adapted (LoRA) parameters. For the pre-trained weight matrix in the model... Set low-rank increment:

[0025]

[0026]

[0027]

[0028] in, This represents the frozen pre-trained weight matrix; and It is a trainable low-rank matrix; and These represent the output and input dimensions of the original weight matrix, respectively. Indicates the rank of the low-rank fit; The scaling parameter represents the low-rank increment. This represents the actual scaling factor.

[0029] By updating only the matrix and This can significantly reduce the GPU memory, computational cost, and parameter storage overhead required for full-parameter training, while avoiding significant changes in the model's overall medical visual understanding capabilities.

[0030] When obtaining the basic prediction results of medical question-answering samples, a candidate answer conditional probability scoring method is adopted. It should be noted that this invention uses a scoring method instead of directly taking the generated sequence as the prediction because medical multiple-choice question answering is a common form in medical visual question answering tasks, and its candidate answer set is fixed and finite. By using a scoring method, the model confidence of different candidate answers can be compared more accurately, and the scoring results can serve as a unified quantitative basis for subsequent error filtering and conflict gradient calculation.

[0031] For medical multiple-choice question-and-answer samples, they are defined as follows:

[0032] in, Indicates the first One medical image input; Indicates a medical issue; Represents a fixed set of candidate answers; This indicates a medical reference answer; This represents sample metadata.

[0033] The fixed set of candidate answers is represented as ,in Indicates the first The number of candidate answers for each sample. Indicates the first One candidate medical answer.

[0034] As an example, when obtaining the basic prediction results of medical question-and-answer samples in step 1, a candidate answer conditional probability scoring method is adopted. Specifically, for each candidate answer in a fixed set of candidate answers, the average conditional log probability of the words contained therein is calculated, and the candidate answer with the highest average conditional log probability is taken as the basic prediction result.

[0035] Specifically, the candidate answer set is preferably fixed before the editing samples are screened, and no new candidate options are added based on the subsequently selected target editing answers, thereby avoiding artificial alteration of the original answer space of conflicting samples. For candidate answers... The score is calculated based on the average conditional logarithmic probability of the answer's word units:

[0036] in, Indicates candidate answers The number of lexical units contained; The first candidate answer Each word element; Indicates that it is located at the th The answer word before each word; This represents the normalized conditional logarithmic probability score of the candidate answer's length. By averaging the number of tokens in the answer, the influence of differences in the length of different candidate answers on the score can be eliminated, making the scoring results comparable among different candidate answers.

[0037] The model's basic prediction is:

[0038] in, This indicates that the basic medical visual question answering model is applicable to the first... The predicted answer for each sample. For short medical question-and-answer sets that do not have a fixed candidate set, the base prediction can be determined based on the conditional probability of the generated answer, the normalized generated text, or the structured category score.

[0039] Step 2: Based on the basic prediction results, filter the target editing samples to be corrected, and construct medical conflict samples with directional correlation corresponding to the target editing samples based on medical structural information; After obtaining the model's basic prediction results on each medical question-and-answer sample, this step aims to select samples that the model predicted incorrectly as editing targets, and construct conflict samples with medical semantic associations around each target editing sample, so as to provide protection for the risk direction of the potential spread of the target answer in the subsequent identification.

[0040] In practice, the first step is to perform answer normalization on the basic predicted answer, the reference answer, and the candidate answers. Let the answer normalization function be... Standardization operations may include, but are not limited to, case unification, punctuation removal, continuous whitespace normalization, unit format unification, mapping of medical abbreviations, and mapping of medical synonyms. For example, mapping "computedtomography" and "CT" to the same standardized representation, and mapping "magnetic cresonance imaging" and "MRI" to the same standardized representation. Through standardization, misjudgments caused by different answer expressions can be eliminated, ensuring the accuracy of screening for incorrect samples.

[0041] When the basic prediction differs from the reference answer, that is:

[0042] The sample was identified as a genuine error editing sample:

[0043]

[0044] in, This represents the base prediction answer of the model before editing. The target edit answer is indicated. Therefore, this invention preferably corrects actual medical question-and-answer errors that have occurred in the basic model, rather than artificially constructing counterfactual edit targets that contradict medical reference information, thereby ensuring that the editing task has real clinical or application value.

[0045] As an example, a generalized sample that preserves the semantics of the target edit answer of the medical image evidence and the target edit sample is also constructed, and a local sample that is correctly predicted based on the pre-editing prediction result is selected; wherein, the local sample includes at least one of the following: local sample in the same image, high semantic local sample, and random local sample; Specifically, construct one or more generalized samples for the target edit sample:

[0046] in, This refers to rewriting questions that are semantically equivalent to the original medical question (such as synonymous expression replacement, sentence structure transformation, etc.). This represents the set of candidate answers after the candidate order has been rearranged. This represents the metadata corresponding to the generalized sample. The generalized sample should maintain the semantics of the medical image evidence and the target answer unchanged, only altering the question expression, candidate order, or other input formats that do not affect the reference medical semantics. It should be noted that the purpose of constructing the generalized sample is to evaluate the semantic generalization ability after editing, ensuring that the model learns the semantic content of the target answer rather than memorizing specific question expressions or option positions.

[0047] Furthermore, a local sample set is constructed:

[0048] in, This represents a set of localized samples within the same image, i.e., samples that use the same medical image but ask different medical questions. It is used to test whether the editor has maintained the ability to answer other questions under the same image. This represents a set of samples with high semantic locality, i.e., samples that have the same imaging modality, organ, anatomical location or medical concept as the edited sample, but with different medical predicates, and are used to test the stability of knowledge in similar medical contexts. This represents a set of random local samples, which prioritizes correctly predicted samples across images, medical relationships, and answer spaces, and is used to test whether editing has a broad impact on the overall model's capabilities.

[0049] Furthermore, based on medical structural information, directionally related medical conflict samples are constructed that correspond to the target editing sample. As an example, the medical structural information includes at least one of medical predicates, normalized medical relations, fixed answer space, question templates, image sources, imaging modalities, and anatomical locations. For each editing sample... Predicting the correct medical protection sample pool from the basic model The set of directional medical conflicts in the construction of the middle structure:

[0050] in, Indicates the relationship with the first The first edit sample corresponding to the first A medical conflict sample, This represents the reference answer for the conflicting sample itself. This represents the medical structure association determination function. When the edited sample and the candidate protected sample satisfy any of the following preset conflict category complete determination rules... If a sample only meets the requirement of having the same single modality, organ, or keyword but does not meet the complete conflict rule, it will not be directly identified as a directional medical conflict sample, in order to avoid introducing weakly associated samples that dilute the protective effect.

[0051] The input to the medical structure association determination function includes at least the normalized medical predicates or relation identifiers of the target edit sample and the candidate protected sample, the answer space identifier, the normalized candidate answer set, the target edit answer, the candidate protected sample reference answer, the question template identifier, the image or case source identifier, and the correctness of the basic prediction before editing; it may also include structural fields such as imaging modality, organ or anatomical location. The determination function outputs a valid association identifier only when the combination of the above fields satisfies any complete determination rule of first-level natural conflict, first-level answer space conflict, second-level relation conflict, or question template conflict.

[0052] like Figure 2 As shown, in this embodiment, the target editing sample and the directional medical sample are constructed according to a preset medical structure association rule. The directional medical sample includes at least one of the following samples whose basic prediction result is correct before editing: A first-level natural conflict sample is one that has the same or highly consistent medical predicates or standardized medical relations as the target edit sample, has the same or a fixed answer space that meets the preset compatibility judgment conditions, and uses different medical image evidence. The target edit answer of the target edit sample exists in the candidate set of the conflict sample, and the reference answer of the conflict sample is different from the target edit answer. A first-level answer space conflict sample is a fixed answer space that has the same or satisfies the preset compatibility judgment conditions as the target editing sample. The target editing answer exists in the candidate set of the conflict sample, and the reference answer of the conflict sample is different from the target editing answer. A second-level relation conflict sample is one that has at least one complete association relationship with the target editing sample, including the same medical predicate, normalized medical relation, or answer space, and the reference answer of the conflict sample is different from the target editing answer. A conflicting sample is one whose question template is the same as that of the target edit sample, or is determined to be semantically equivalent by a preset question template normalization rule or semantic similarity threshold, and uses different medical image evidence. The reference answer of the conflicting sample is different from that of the target edit answer.

[0053] In this embodiment, identical answer spaces preferably refer to those with the same answer space identifier after answer normalization; compatible answer spaces preferably refer to two candidate sets belonging to the same medical attribute category and satisfying a preset set inclusion relationship or overlap condition after normalization. In the current implementation, compatibility can be directly determined by the same answer_space_key; in other implementations, it can also be determined based on the inclusion relationship of the normalized candidate sets or a preset overlap ratio.

[0054] In this embodiment, the semantic equivalence of question templates is preferably determined by preset question template normalization rules, including sentence normalization without changing medical predicates, synonym mapping, and medical entity standardization; when the normalized question_template_keys are the same, they are determined to be semantically equivalent. Optionally, it can also be determined by the similarity between manual annotation or text semantic representations not being lower than a preset threshold.

[0055] When the same candidate protected sample satisfies multiple conflict categories, it is preferred to classify it uniquely according to the order of first-level natural conflict, first-level answer space conflict, second-level relation conflict, and question template conflict, and use the conflict level weight corresponding to the category; the same candidate sample is only counted once in the conflict gradient matrix of the same editing task.

[0056] Specifically, a first-level natural conflict sample must at least satisfy the following conditions: the edited sample and the conflicted sample use different medical image evidence, have the same or highly consistent medical predicates or normalized medical relations, have the same or a fixed answer space that meets the preset compatibility judgment conditions, the edited target answer exists in the original candidate set of the conflicted sample, the reference answer of the conflicted sample is different from the edited target answer, and the conflicted sample is correctly predicted by the base model before editing.

[0057] A first-level answer space conflict sample must at least satisfy the following: both have the same or satisfy the preset compatibility judgment conditions in their answer spaces, the editing target answer exists in the original candidate set of the conflict sample, the reference answer of the conflict sample is different from the editing target answer and the prediction before editing is correct, but the requirement for consistency of medical relationship is lower than that of first-level high-quality natural conflict.

[0058] A conflicting sample of a second-order relation must at least satisfy the following conditions: the two have the same medical predicate, a normalized medical relation, or at least one complete association relationship in the answer space; the reference answer of the conflicting sample is different from the target answer of the editing sample and the prediction before editing is correct; however, it is not required that the target answer of the editing sample exists in the candidate set of conflicting samples.

[0059] Conflicting sample questions must meet at least the following conditions: the question templates are the same or are semantically equivalent according to the aforementioned rules, the medical image evidence is different, the reference answers are different, and the conflicting sample was correctly predicted before editing.

[0060] By constructing the above four hierarchical and multi-dimensional conflict samples, this invention can comprehensively capture the risk paths of potential targeted propagation of the target answer at different semantic similarity levels: first-level natural conflict reflects the conflict risk under the most stringent high similarity scenarios; first-level answer space conflict reflects the risk brought about by the overlap of candidate answer spaces; second-level relational conflict reflects the risk at a broader medical relational level; and question template conflict reflects the propagation risk of the same question semantics under different images. It should be understood that this multi-level construction method can provide rich and discriminative gradient information for the subsequent estimation of conflict-sensitive subspaces.

[0061] As an example, when the current editing task cannot construct any valid conflict samples that satisfy the preset conflict category, subsequent processing will perform corresponding degradation processing as if the set were empty. Specific details will be described in step 5. When the reference answer anchor subset or local sample set is empty, it will also be uniformly processed in step 5 to avoid calculation anomalies caused by averaging or matrix inversion of an empty set. Through the above processing framework, it can be ensured that the method of this invention can still operate stably even when conflict samples or local samples are sparse.

[0062] Step 3: Calculate the target editing gradient based on the target editing sample and the conflict gradient based on the medical conflict sample, respectively, and use the conflict gradient to estimate the conflict-sensitive subspace corresponding to the current target editing sample; After completing the screening of target editing samples, the construction of generalized samples, and the construction of directional medical conflict samples, this step aims to calculate the target editing gradient and the conflict gradient respectively, and dynamically estimate the conflict-sensitive subspace corresponding to the current editing task based on the conflict gradient. This subspace is used to characterize the dangerous update direction of the target editing answer propagating to the conflict samples in the current model state, serving as the basis for the adaptive projection protection in the subsequent step 4.

[0063] In practice, the target editing gradient is calculated as follows: the target editing loss based on the target editing sample and the generalization loss based on the generalization sample are calculated separately, and the two are combined and differentiated with respect to the lightweight editable parameter to obtain the target editing gradient.

[0064] For target editing samples The negative log-likelihood loss of the target answer is expressed as:

[0065] in, This indicates the number of tokens contained in the target edit answer. This indicates the first option in the target edit answer. One token, This indicates the target answer token preceding this token.

[0066] Calculate the generalization loss using the same target answer for generalized samples:

[0067] The editing loss and generalization loss are combined into a target editing objective function:

[0068]

[0069] in, This represents the generalization loss weight, used to adjust the relative contribution of the original target editing sample and the generalized sample to the editing direction; This represents the target editing gradient used for subsequent conflict subspace projection. Preferably, the target editing gradient does not include the ordinary locality preservation gradient and the reference answer anchoring gradient, so that conflict projection only acts on the editing direction that may cause the non-target propagation of the target answer.

[0070] As an example, the conflict gradient estimation is used to estimate the conflict-sensitive subspace corresponding to the current target edit sample, including: Step S31: Calculate the score of the target edit answer of the target edit sample or the reference answer of the medical conflict sample on the medical conflict sample, construct the conflict injection loss or the reference answer preservation loss based on the score, and differentiate the loss with respect to the lightweight editable parameter to obtain the original conflict gradient; Specifically, for conflict samples Calculate the average conditional log probability of the target edit answer and the reference answer of the conflict sample on the conflict sample, respectively:

[0071]

[0072] in, , and These represent the medical images, medical questions, and candidate answer sets of conflicting samples, respectively. This indicates the score of the current edit target answer on the conflicting samples; This represents the score of the reference answer for the conflicting sample.

[0073] Construct the conflict injection loss using a smoothing interval function:

[0074]

[0075] in, Indicates the preset safety interval; This represents a smoothing positive value function, which increases the loss when the score of the target edit answer on the conflict sample is close to or exceeds the score of the reference answer on the conflict sample, and outputs a continuous gradient even before the sorting is reversed, thus characterizing the potential direction of conflict propagation in advance.

[0076] Calculate the gradient of lightweight parameters for the conflict injection loss:

[0077] in, Indicates the first The original conflict gradient generated from each conflict sample.

[0078] For second-order relation conflicts or question template conflicts that do not meet the existence condition of the target answer, the negative log-likelihood loss of the conflict sample reference answer can also be used as the reference answer preservation loss:

[0079] in, This represents the number of tokens in the reference answer of the conflicting sample. The corresponding conflict gradient is:

[0080] Step S32: Normalize each valid non-zero original conflict gradient with a 2-norm with a stabilizing constant, scale it according to the square root of the conflict level weight, and combine them column by column to form a conflict gradient matrix. To prevent individual conflict samples from dominating the subspace direction due to excessively large gradient magnitudes, it is preferable to first normalize each effective non-zero original conflict gradient with a 2-norm normalization and then scale it according to the square root of its corresponding conflict level weight.

[0081] in, Indicates the first The first editing task The weights corresponding to the conflict levels of each conflict sample; This represents a stability constant used to prevent the denominator from being zero during conflict gradient normalization. Let L2 represent the norm. This formula indicates that the original conflict gradient is first normalized to L2 with a stability constant, and then scaled according to the square root of the conflict level weights.

[0082] set up This is a set of subspace estimation samples that have been filtered through preset conflict categories and can generate effective non-zero conflict gradients. These samples may come from first-level high-quality natural conflicts, first-level answer space conflicts, second-level relational conflicts, or question template conflicts; let... The currently edited sample will correspond to... The conflict gradients are concatenated column-wise to form a conflict gradient matrix:

[0083] in, This represents the total dimension after the lightweight editable parameters are flattened. This indicates the number of valid non-zero conflicting gradients used in the current edit sample.

[0084] Step S33: Construct a gradient inner product matrix with regularization terms, i.e., Gram matrix, based on the conflict gradient matrix. Perform eigenvalue decomposition on the Gram matrix and select the eigenvectors corresponding to the first k eigenvalues ​​in descending order of eigenvalues. Map the eigenvectors to the gradient space of the lightweight editable parameters through the conflict gradient matrix to obtain the conflict principal direction. The conflict sensitive subspace is spanned by the conflict principal direction.

[0085] Specifically, since the feature vectors reside in the coefficient space of conflicting samples, they need to be mapped to the gradient space of lightweight editable parameters using the conflict gradient matrix. The mapping formula is:

[0086] in, Indicates the first The selected feature vectors (located in the conflict sample coefficient space, with dimension ) are... ), Represents the conflict gradient matrix (dimension 1). ), This represents the principal conflict direction obtained after mapping (located in the gradient space of the lightweight editable parameters, with dimension ). ).

[0087] Because it usually satisfies To avoid explicitly constructing dimensions of From the high-dimensional matrix, construct a low-dimensional Gram matrix (gradient inner product matrix) with regularization terms:

[0088] in, The regularization coefficients of the Gram matrix are represented by their respective coefficients. express An identity matrix of order 1. Regularization terms are used to improve numerical stability in low-rank or near-singular cases; when... In this case, the subsequent operator belongs to the regularized approximate projection operator, rather than the strictly orthogonal projection operator.

[0089] Perform eigenvalue decomposition on the Gram matrix:

[0090] in, Represents the eigenvector matrix; This represents a diagonal matrix composed of eigenvalues.

[0091] set up To predetermine the dimension of the conflict subspace, The number of valid non-zero conflict gradients retained for the current editing task determines the actual subspace dimension used. Select the first one according to the eigenvalues ​​in descending order. The eigenvalues ​​and their corresponding eigenvectors are denoted as follows:

[0092] in, Indicates the first One selected feature vector; This represents the corresponding eigenvalue.

[0093] Based on the selected eigenvectors and their mappings, the principal direction of the conflict is... The conflict-sensitive subspace corresponding to each editing task is defined as follows:

[0094] in, Indicates the principal direction of the conflict after mapping. This represents the eigenvectors obtained from the Gram matrix decomposition. This subspace will be used in subsequent steps to project constraints onto the target editing gradient.

[0095] Figure 3 The complete process of the above-mentioned conflict-sensitive subspace estimation is illustrated schematically.

[0096] This step dynamically constructs a conflict-sensitive subspace corresponding to each editing task, based on the current model state and the current set of conflict samples. This subspace changes dynamically depending on the editing target answer, the conflict samples, and the model state, rather than using a pre-fixed uniform subspace, thus accurately reflecting the unique risky update directions of different editing tasks.

[0097] Step 4: Project the target editing gradient onto a conflict-sensitive subspace to obtain a conflict hazard component; calculate the overlap between the target editing gradient and the conflict-sensitive subspace to adaptively determine the protection strength; and attenuate the conflict hazard component in the target editing gradient based on the protection strength to obtain a safe editing gradient. Please refer to the following: Figure 4 This illustrates the complete process of adaptive editing gradient projection and protective gradient fusion in this step.

[0098] After estimating the conflict-sensitive subspace corresponding to the current editing task, this step aims to quantify the overlap between the target editing gradient and this subspace, adaptively determine the protection strength accordingly, and selectively attenuate dangerous components in the target editing gradient that may cause conflict drift, ultimately obtaining a safe editing gradient. Through this adaptive mechanism, the present invention can retain more effective editing directions to ensure the success rate of target editing when the conflict risk is low, and enhance the suppression of dangerous components to protect conflict samples when the conflict risk is high, thereby avoiding insufficient protection or excessive restriction caused by a fixed projection strength.

[0099] As an example, the target editing gradient is projected onto a conflict-sensitive subspace to obtain a conflict hazard component. The overlap between the target editing gradient and the conflict-sensitive subspace is calculated to adaptively determine the protection strength. Based on the protection strength, the conflict hazard component in the target editing gradient is attenuated to obtain a safe editing gradient, including: Step S41: Construct a regularized approximate projection operator based on the conflict-sensitive subspace, calculate the projection component of the target editing gradient in the conflict-sensitive subspace, and use it as the conflict danger component; Specifically, based on the aforementioned definition of conflict-sensitive subspace Construct a regularized approximate projection operator that acts on this subspace:

[0100] in, This represents the conflict gradient matrix corresponding to the current editing task. and These respectively represent the aforementioned selections. Each eigenvector matrix and its corresponding eigenvalue diagonal matrix. When When, the operator belongs to the regularized approximate projection operator; when Furthermore, when the columns of the conflict gradient matrix are full rank, the operator degenerates into the standard orthogonal projection operator.

[0101] Based on the projection operator described above, the target editing gradient is calculated. Projected components in the conflict-sensitive subspace:

[0102] Will As a conflict-prone component, this component reflects the portion of the target editing gradient that falls into the conflict-sensitive subspace, i.e., the dangerous update direction that may trigger the directional propagation of the target editing answer to conflict samples.

[0103] Step S42: Calculate the ratio of the length of the conflict-prone component to the length of the target editing gradient with a stability constant, as the degree of overlap, i.e., the relative overlap rate; determine an adaptive protection coefficient between 0 and 1 based on the ratio of the relative overlap rate to a preset overlap threshold, as the protection strength; wherein, the larger the relative overlap rate, the larger the adaptive protection coefficient. First, calculate the relative overlap rate of the target editing gradient in the conflict-sensitive subspace:

[0104] in, Indicates the first The degree of relative overlap between the target editing gradient and the conflict-sensitive subspace for each editing task; This represents a stability constant used to limit the lower bound of the denominator, preventing the ratio from being abnormally amplified when the target editing gradient is close to zero.

[0105] The adaptive protection strength, i.e., the adaptive protection coefficient, is determined based on the overlap rate:

[0106] in, Indicates the preset overlap threshold; This indicates the conflict protection coefficient for the current editing task. When... When smaller, Smaller size allows for more target editing directions to ensure a higher success rate; when When it is large, The relative overlap rate is increased, thereby strengthening the suppression of conflict risk components to protect conflict samples. Thus, the higher the relative overlap rate, the larger the adaptive protection coefficient, achieving adaptive adjustment of the protection strength.

[0107] Step S43: Subtract the conflict risk component multiplied by the adaptive protection coefficient from the target editing gradient to attenuate the conflict risk component and obtain the safe editing gradient.

[0108] Specifically, the protected target editing gradient is:

[0109] in, This represents the safe edit gradient after constraint of the conflict-sensitive subspace. When... When the safe edit gradient is equal to the original target edit gradient, no decay is performed; when At that time, the dangerous component of conflict was completely removed; when At this time, the conflict risk component is partially attenuated. Through the above adaptive attenuation processing, the present invention achieves a dynamic balance between target knowledge writing and conflict knowledge protection.

[0110] Step 5: Fuse the secure editing gradient with the reference answer anchoring gradient for the medical conflict sample to obtain the final update gradient, and use the final update gradient to update the lightweight editable parameters.

[0111] Please see Figure 4 , Figure 4 This is a schematic diagram of the adaptive editing gradient projection and protective gradient fusion process in an embodiment of the present invention.

[0112] Before performing gradient fusion, we first address the potential for empty set degradation. This includes handling cases where there are no valid conflicting samples in the current editing task, or the number of valid conflicting gradients remaining after deleting zero gradients. Then, the effective number of conflict gradients and the actual subspace dimension are both set to 0, and the conflict approximate projection operator, conflict projection components, overlap rate, and adaptive protection coefficient are all set to 0. At this time, the safe editing gradient degenerates into the target editing gradient. When the reference answer anchor subset is empty, the reference answer anchor gradient is set to a zero vector. When the locality sample set is empty, the locality preservation gradient is set to a zero vector. When the L2 norm of the target editing gradient is not higher than the preset stability threshold, the overlap rate and adaptive protection coefficient are also set to 0. The specific formula is as follows:

[0113]

[0114] (Anchor subset is empty); (The local sample set is empty) The above processing ensures that the algorithm can still execute stably even with sparse samples, and that each variable has a definite value. After completing the degradation determination, subsequent gradient fusion and parameter updates are performed.

[0115] After obtaining the safe edit gradient after the conflict risk component has been attenuated, this step aims to fuse the safe edit gradient with the reference answer anchoring gradient to form the final gradient used to update the lightweight editable parameters, thus completing the parameter update. The role of reference answer anchoring is to maintain or enhance the model's predictive ability for the reference answer of the conflict sample itself during the parameter update process, avoiding unintentional suppression of the reference answer direction that might be caused by the conflict subspace projection operation. This ensures that the correct answer of the conflict sample itself remains unaffected while suppressing the propagation of the target answer to the conflict sample.

[0116] As an example, in step 5, when fusing the secure editing gradient with the reference answer anchoring gradient for the medical conflict sample, the locality preservation gradient obtained by differentiating the locality sample is further fused to obtain the final update gradient.

[0117] Specifically, let the subset of conflicting samples applicable to anchoring the reference answer be: ={ | Valid and its model score is computable} The subset can contain samples that meet the above conditions from the following categories: first-level high-quality natural conflict, first-level answer space conflict, second-level relation conflict, and question template conflict. Its reference answer anchoring loss is:

[0118] in, This indicates the number of conflicting samples that participated in anchoring the reference answer.

[0119] The reference answer's anchored gradient is:

[0120] At the same time, from independent local sample sets For samples that were correctly predicted by the base model in the mid-sampling phase, define the ordinary locality preservation loss:

[0121] in, This indicates the number of locality samples used in the current editing task; This represents the reference answer for local samples. When At that time, in accordance with the aforementioned degradation treatment order .

[0122] The locality-preserving gradient is:

[0123] The method of this invention integrates the safe editing gradient, the reference answer anchoring gradient, and the general locality-preserving gradient:

[0124] in, This represents the gradient ultimately used to update the lightweight editable parameters; This indicates the anchor weight of the reference answer; This indicates that the weights preserve general locality. Anchoring, as a necessary component mechanism, has weights that satisfy... The weight for maintaining general locality is set according to implementation requirements. In the method of this invention, the reference answer anchoring gradient and the ordinary locality-preserving gradient are added after the conflict projection is completed, without going through the regularized approximate projection operator. This processing order can prevent the effective gradient components in the reference answer preservation direction and the ordinary locality-preserving direction from being mistakenly deleted by the conflict projection.

[0125] Next, the lightweight editable parameters are updated using the final update gradient. Let the... The lightweight editable parameter at the start of each optimizer update cycle is: The gradient accumulation steps are , No. The index of the edit task corresponding to each micro-training step is: Then the average cumulative gradient used in this update cycle is:

[0126] When using stochastic gradient descent, Adam, AdamW, or other gradient optimizers, the parameter update can be expressed as:

[0127] Among them, the most basic gradient descent degenerates into:

[0128] As an example, for multiple consecutive medical knowledge editing tasks, the conflict-sensitive subspaces corresponding to different target editing samples are dynamically estimated based on the current model state and the current medical conflict samples, rather than using a pre-fixed unified subspace. For sequential medical knowledge editing, let the sequence of edited samples to be processed be . ,in Indicates the number of consecutive editing tasks. For the ... Each editing task is based on the current model state. and the current conflict set Recalculate:

[0129] And complete the parameter update:

[0130] in, This represents a knowledge editing and update process comprised of target editing, conflict-sensitive subspace projection, reference answer anchoring, and locality preservation. The conflict-sensitive subspaces corresponding to different editing tasks are dynamically estimated based on the current model state, rather than using a pre-fixed uniform subspace throughout the entire editing process. This ensures that each editing task receives targeted and specific protection.

[0131] Save the lightweight, editable parameters after editing. In the low-rank adaptation implementation, the matrix can be stored separately. sum matrix The scaled low-rank increment can also be used. Merge into the original model weights, i.e. .

[0132] In the inference phase after the medical visual question answering model completes knowledge editing, the medical visual question answering model only loads the updated lightweight editable parameters to perform the forward computation process. The inference phase does not load the medical conflict samples, does not calculate the conflict-sensitive subspace, and does not perform gradient backpropagation and subspace projection operations.

[0133] In the reasoning phase, given a new medical image Medical issues and candidate answer set The model outputs the probability distribution of the answer:

[0134] For multiple-choice questions in the medical field, the inferred answer is:

[0135] in, The set of candidate answers representing the inference sample; Indicates the index of the candidate answer; This indicates the edited model's response to the candidate answers. The average conditional logarithmic probability score. For open-ended questions, let The score is generated using the sequence of answers.

[0136] The inference phase does not require loading the medical conflict samples and locality samples used during training, does not calculate the edit loss, conflict loss or reference answer preservation loss, does not perform gradient backpropagation, Gram matrix construction, eigenvalue decomposition and conflict subspace projection, and does not require calling the external edit memory retrieval module.

[0137] like Figure 5 As shown, the training / editing phase and the inference phase of this invention have a clear division of labor: the training / editing phase performs operations such as loading conflicting samples, calculating gradients, estimating subspaces, and projecting; while the inference phase only loads the updated lightweight editable parameters and performs the forward computation process.

[0138] Therefore, the conflict protection mechanism of this invention only increases the computation process during the training or knowledge editing stage, without changing the basic reasoning chain of the edited medical visual question answering model, thus ensuring the simplicity and efficiency of deployment.

[0139] The technical solution of the present invention will now be described in its entirety by way of example: This embodiment illustrates the specific implementation process of the knowledge editing method for medical visual question answering models based on conflict-sensitive subspace constraints proposed in this invention on the PMC-VQA multiple-choice visual question answering dataset. This embodiment uses Qwen3-VL-8B-Instruct as the basic visual language model, the actual prediction errors of the basic model on the PMC-VQA dataset as the knowledge editing object, low-rank adaptation parameters as lightweight editable parameters, and MedConSPEdit, which includes conflict-sensitive subspace projection, post-projection reference answer anchoring, and locality-preserving gradient fusion, as the implementation method. The conflict-sensitive subspace dimension is also specified. The value is set to 2, meaning the conflict-sensitive subspace is constructed by selecting the principal directions corresponding to the two largest eigenvalues ​​of the Gram matrix. Data construction, sample extraction, and model training all use a fixed random seed of 42. Training and inference computations use BF16 (BrainFloatingPoint16, 16-bit floating-point calculation format). In PMC-VQA, the candidate answer set for each multiple-choice question is fixed before the edit sample screening, and candidate options are not added or replaced subsequently based on the selected editing target. The model determines the prediction result by calculating the average conditional log probability of the answer token for each candidate, thus reducing the impact of differences in candidate text length and fixed option positions on the results.

[0140] Phase 1: Initialization of the Medical Visual Question Answering Model and Scoring of Candidate Answers This phase uses PMC-VQA multiple-choice training data and its corresponding medical images, natural language medical questions, fixed candidate answer sets, reference answers, and sample source information as input, while retaining image identifiers, article or case source identifiers, question types, answer spaces, and structural meta-information required for subsequent conflict construction. The base model adopts Qwen3-VL-8B-Instruct. After loading the model weights and multimodal processor, the original parameters of the visual encoder, multimodal connection module, and language decoder are frozen. Only the LoRA lightweight parameter is inserted into the q_proj (query projection), v_proj (value projection), and o_proj (output projection) modules of the last 16 layers of the language decoder. The rank of LoRA is set to 8, the scaling parameter is set to 16, the dropout (random inactivation rate) is set to 0, the model calculation type is set to BF16, and the maximum number of new tokens allowed during candidate answer scoring is set to 32.

[0141] In the processing, medical images and questions are first input into the model according to a unified multimodal prompt format. Then, each candidate answer from a fixed candidate set is input into the question prompt as the answer to be scored. The conditional log probability of all answer terms contained in the candidate answer is calculated, and the average is taken according to the number of answer terms. The model determines the candidate with the highest average conditional log probability as the basic predicted answer. To avoid the influence of differences in answer expression on the incorrect selection, the model prediction, reference answer, and candidate answers are also subjected to case uniformity, punctuation and whitespace normalization, medical abbreviation mapping, and common synonym mapping. For example, "computed tomography" and "CT", and "magnetic resonance imaging" and "MRI" are unified as the same medical concept.

[0142] After completing the above processing, this stage outputs the frozen Qwen3-VL main model, the initialized LoRA parameters, and the basic prediction records of PMC-VQA candidate samples. Each record contains the medical image path, medical question, fixed candidate answer, each candidate score, model prediction answer, reference answer, and source identifier, and serves as the input for the next stage to screen real error samples and construct medical protection samples.

[0143] Phase Two: Construction of Real Error Editing Samples, Generalized Samples, and Directional Medical Protection Samples This stage uses the PMC-VQA basic prediction records, the original fixed candidate answer set, and sample metadata generated in the previous stage as input. From the samples whose images can be read normally, whose reference answers are clear, and whose basic model prediction results can be verified, the editing candidate pool and the protection candidate pool are divided. Among them, the samples whose basic predictions are inconsistent with the standardized reference answers are entered into the editing candidate pool, and the samples whose basic predictions are correct are entered into the medical protection candidate pool.

[0144] In this embodiment, 1000 real error samples are selected from the edit candidate pool to form the formal edit set. The dataset reference answer of each sample is set as the target edit answer, and its original medical image, original question, fixed candidate set, error prediction before editing, target answer and source metadata are retained.

[0145] For each edited sample, a generalized sample with rearranged candidate options is first constructed. During the rearrangement, only the order of the candidate options and their corresponding numbers are changed, without altering the medical image, question semantics, candidate answer text, or target answer. This is to test whether the model has learned the semantics of the target answer rather than memorizing fixed option positions. Subsequently, from the protection pool correctly predicted by the base model, directional medical conflict samples are constructed based on information such as medical predicates, medical relations, answer space, question templates, image sources, imaging modalities, and anatomical locations. The first-level natural conflict (NaturalL1-HQ) requires that the edited sample and the protection sample have the same or highly consistent medical relations and answer space. The target edited answer naturally exists in the original candidate set of the protection sample, but the reference answer of the protection sample is different from the target edited answer. The first-level answer space conflict (L1-AS) preserves the existence of the same answer space and the target answer, but the requirement for relation matching is relatively weak. The second-level relation conflict (L2-Rel) is used to maintain relevant knowledge in the same medical predicates, medical relations, or answer space. The question template conflict (Question-C) is used to test whether the same or semantically equivalent questions are incorrectly assimilated under different medical image evidence.

[0146] Simultaneously, localized samples, highly semantically localized samples, and randomized localized samples are constructed. Localized samples use the same medical images as the edited samples but query different medical attributes. Highly semantically localized samples are similar to the edited samples in imaging modality, organs, anatomical locations, or medical concepts but do not belong to the current directional conflict. Randomized localized samples are preferentially extracted across images, across medical relationships, and across answer spaces. Data partitioning and conflict extraction both use a random seed of 42. Each edited sample is assigned a maximum of 3 training protection conflicts, and the reuse of protection samples is limited. Samples are isolated according to article, image, or other source identifiers. The training protection pool, development pool, and evaluation pool are separated. The training protection candidate pool contains 79,409 samples, the development candidate pool contains 18,247 samples, and the evaluation candidate pool contains 24,182 samples, corresponding to 9,995 evaluation article sources.

[0147] After completing the overlap check of sample keys, image keys, and question keys, this stage ultimately outputs 1000 real error edit samples, 1000 candidate order rearrangement generalization samples, 3000 training conservation conflicts, and 1427 independent conflict evaluation samples. All 1000 edit samples have usable training conservation samples. Among them, the NaturalL1-HQ strict natural conflict evaluation set contains 449 samples. The number of sample keys overlapping between the training conservation set and the evaluation set is 0. At the same time, hierarchical training or evaluation sets such as L1-Core, L1-AS, L2-Rel, Question-C, Same-Image, HS-Loc, and Random-Loc are formed.

[0148] Phase 3: Target Editing Gradient Calculation and Conflict-Sensitive Subspace Estimation This phase uses the current model and LoRA parameter state, one true error edit sample, one corresponding candidate order rearrangement generalization sample, and up to three directional medical conflict samples as input for a micro-training step. Samples capable of generating effective non-zero conflict gradients are selected from preset conflict categories such as NaturalL1-HQ, L1-AS, L2-Rel, and Question-C, according to the extraction configuration fixed in the data construction phase. Same-Image, HS-Loc, and Random-Loc samples used only for general locality preservation do not enter the conflict-sensitive subspace.

[0149] First, on the target medical image, the original medical question, and a fixed set of candidate answers, the target editing loss is calculated using the reference answer as the supervised target. Then, the generalization loss is calculated on the same medical image and the input after the candidate order has been rearranged. The weights of the generalization loss are... Setting it to 0.35, the combination of the two is differentiated only with respect to the LoRA parameters to obtain the edit gradient used to write the target knowledge while maintaining the generalization ability of the option order.

[0150] Then, directional conflicts are processed one by one: for first-level conflicts where the target edit answer naturally exists in the fixed candidate set of conflict samples, the average conditional log probability difference between the target edit answer and the reference answer of the conflict samples is compared, and a safety interval is used. The original conflict gradients are generated using a softplus smoothing margin loss of 0.5. For second-order relation conflicts or question template conflicts that do not require the target edit answer to exist in the candidate set, the original conflict gradients are formed using the negative log-likelihood of the reference answer of the conflict sample itself. The weights of the first-order conflict gradients are set to 1.5, and the weights of the second-order conflict gradients are set to 1.0. Each valid non-zero original conflict gradient is first used... Perform L2 normalization with a stability constant, then scale by the square root of the conflict level weights. Combine up to three conflict gradients column-wise into a conflict gradient matrix, and calculate the regularization coefficient. The low-dimensional Gram matrix is ​​decomposed, and the top eigenvalues ​​are selected in descending order. Each characteristic direction. This embodiment sets... The actual dimension is ,in .

[0151] This stage outputs the target editing gradient, the normalized and hierarchically weighted conflict gradient, the conflict gradient matrix, a conflict-sensitive subspace of at most two dimensions, the number of effective conflict gradients, and the actual subspace dimension.

[0152] Phase 4: Adaptive Gradient Projection Editing and Preservative Gradient Fusion This stage takes the target editing gradient, conflict-sensitive subspace and its projection components obtained in the previous stage as the main input, and also receives the reference answer anchor sample corresponding to the current editing sample and the sample that was originally correctly predicted by the base model and extracted from the independent locality pool.

[0153] First, use stable parameters. The proportion of the projected length of the target edit gradient in the current conflict-sensitive subspace to the total length of the target edit gradient is calculated and used as the overlap rate; the overlap threshold is also calculated. Set to 0.18. When the overlap rate is low, more of the original edit gradient is preserved; when the overlap rate is close to or exceeds 0.18, the protection factor is increased accordingly and reaches a maximum of 1 to strengthen the attenuation of dangerous components. The projection operation only applies to the target edit gradient; the ordinary locality-preserving gradient and the reference answer anchoring gradient do not participate in this projection.

[0154] Subsequently, samples correctly predicted by the base model are drawn from the independent locality sample pool, and the locality preservation loss is calculated using these samples as reference answers, along with the locality preservation weights. Set to 0.10; for directional conflict samples where the reference answer is valid and the model score is computable. The anchoring loss is calculated using its own reference answer, and the anchoring weight of the reference answer is... Set to 0.15. Both types of protection gradients are added to the safety editing gradient after the conflict projection is completed.

[0155] This stage records the overlap rate, adaptive protection coefficient, and actual subspace dimension. And the number of samples that participated in anchoring the reference answer.

[0156] Phase 5: Lightweight Parameter Updates, Continuous Knowledge Editing, and Model Inference This phase uses 1000 real error edit samples, the corresponding generalized samples and training protection samples for each edit sample, and the final update gradient generated in each micro-training step as input, employing the AdamW optimizer to continuously update the LoRA parameters. The learning rate is set to 8×10⁻⁶. -5 The calculation type is BF16, and the gradient accumulation steps are set to... The average cumulative gradient is formed every 4 micro-training steps. One optimizer update is performed, with the total number of optimizer update steps set to 2000. Therefore, the training process handles approximately 8000 micro-training sample exposures.

[0157] Specifically, in this embodiment, with a fixed random seed of 42, 1000 edit samples are randomly shuffled in rounds and iterated through 8 rounds, with each edit sample being accessed once in principle in each round; thus, approximately 8000 micro-training steps are formed. The gradient accumulation step is 4, and the optimizer is updated once every 4 micro-training steps, corresponding to approximately 2000 optimizer updates.

[0158] Each micro-training step recalculates the target editing gradient, conflict gradient, and conflict-sensitive subspace corresponding to the current editing task based on the current model state. These are then accumulated after adaptive projection and preservative gradient fusion. The entire continuous editing process does not use a pre-fixed unified conflict subspace; the original parameters of the visual encoder, multimodal connectivity module, and language model remain frozen throughout. After 2000 optimizer updates, the edited LoRA adapter is saved. During deployment, it can be loaded together with the frozen base model, or the scaled low-rank increment can be used. Merge into the corresponding projection weights.

[0159] Evaluation indicators and results: The evaluation phase still uses the candidate-by-candidate average conditional log probability, and the following indicators are statistically analyzed: ES (Edit Success Rate), Gen-Shuffle (Generalization Rate of Candidate Order Rearrangement), R-Loc (Stochastic Locality Preservation Rate), HS-Loc (High Semantic Locality Preservation Rate), Strict-C (Strict Conflict Reference Answer Preservation Rate), CDR (Conflict Drift Rate), and OER (Other Error Rate).

[0160] To quantify the implicit score drift that has not yet caused the final answer to be flipped, we define the average score interval change of the reference answer relative to the target edited answer. for:

[0161] in, This represents the set of independent conflict assessment samples; and These represent conflict samples before and after editing, respectively. The score for the reference answer; and These represent the current target edited answer before and after the edit in the conflict samples, respectively. The score.

[0162] when When the score is higher, it indicates that the average score advantage of the edited reference answer relative to the target edited answer has increased.

[0163] To ensure verifiability of the comparison, the comparison method in this embodiment uses the same basic model, LoRA editable module, target editing set, training steps, and main optimization configuration as the present invention. Specifically, RandomProjection replaces the main conflict direction estimated by the conflict gradient with a randomly generated orthogonal direction of the parameter space; the dimension of the random subspace is consistent with the main settings of the present invention, and other comparable training configurations remain consistent. AlphaEditNull-space, following the null space constraint concept, uses correctly predicted protection samples from the base model in the protection candidate pool to estimate a unified constraint space shared by multiple editing tasks, and applies this constraint to the same LoRA editable module as the present invention. This protection space is not dynamically reconstructed line by line for the current target editing sample. PreservationReplay does not perform conflict-sensitive subspace estimation and gradient projection; instead, it directly adds a preservation loss based on the reference answer of the protection sample in addition to the target editing loss, and uses data from the same source as the training protection samples of the present invention for replay.

[0164] Table 1 shows the comparison of experimental metrics for each method on the PMC-VQA dataset.

[0165] Table 1. Experimental metrics of this invention on the PMC-VQA dataset compared with other models.

[0166] As shown in Table 1, MedConSPEdit has a preset conflict-sensitive subspace dimension. At that time, the edit success rate (ES) was 93.6%, the candidate order reordering generalization rate (Gen-Shuffle) was 53.8%, the random locality retention rate (R-Loc) was 57.0%, and the high semantic locality retention rate (HS-Loc) was 67.7%. Among 449 high-quality first-level natural conflict samples, the strict conflict reference answer retention rate (Strict-C) was 67.0%, the conflict drift rate (CDR) was 13.4%, and the other error rate (OER) was 19.6%. The score interval change between the reference answer and the target answer was also observed. It is +0.1790.

[0167] In comparison, standard LoRA achieves a Strict-C of 24.7% and a CDR of 29.0% on the PMC-VQA dataset. The value is -2.388; using the random projection constraint method, the Strict-C is 29.0%, and the CDR is 24.1%. The value is -2.228; using the AlphaEdit zero-space constraint method, Strict-C achieves 24.9%, and CDR achieves 26.9%. The value is -2.427. The MedConSPEdit of this invention achieves significant improvements in all conflict protection metrics while maintaining a high editing success rate.

[0168] Table 2 shows the comparison of experimental metrics for each method on the SLAKE dataset.

[0169] Table 2 Comparison of experimental metrics of this invention with other models on the SLAKE dataset.

[0170] As shown in Table 2, MedConSPEdit achieves the following results on the SLAKE dataset: ES 96.0%, Gen-Shuffle 90.0%, R-Loc 92.0%, HS-Loc 86.0%, Strict-C 91.1%, CDR 2.5%, and OER 6.3%. The result of +1.083 further verifies the effectiveness of the method of the present invention on different medical visual question answering datasets.

[0171] Table 3 shows the comparison of experimental metrics for each method on the OmniMedVQA dataset.

[0172] Table 3 Comparison of experimental metrics of this invention with other models on the OmniMedVQA dataset.

[0173] As shown in Table 3, MedConSPEdit achieves the following results on the OmniMedVQA dataset: ES 100.0%, Gen-Shuffle 99.6%, R-Loc 96.3%, HS-Loc 95.0%, Strict-C 92.4%, CDR 6.8%, and OER 0.8%. It is +1.206. For comparison, the standard LoRA on OmniMedVQA has a Strict-C of 77.5% and a CDR of 21.2%. The value is -0.366; using the random projection constraint method, the Strict-C is 83.1%, and the CDR is 16.2%. The value is 0.300; using the AlphaEdit zero-space constraint method, Strict-C achieves 78.5%, and CDR achieves 20.2%. The value is -0.431. Experimental results on the OmniMedVQA dataset show that the method of this invention can significantly improve the protection of conflicting samples and maintain a high editing success rate on larger-scale and more diverse medical visual question answering data, further verifying the generalizability and effectiveness of the method.

[0174] Regarding ablation experiments, the comparison of ablation experiment metrics on the PMC-VQA dataset is shown in Table 4.

[0175] Table 4 Comparison of ablation experimental indicators of the present invention on the PMC-VQA dataset.

[0176] Main settings selection =2 is a comprehensive trade-off between editing success rate, conflict protection effectiveness, other error rates, and computational overhead. Although When the value is 3, the CDR decreases further and the ΔMargin increases further, but its ES decreases from 93.6% to 92.6% and OER increases from 19.6% to 21.2%. At the same time, more conflicting principal directions need to be retained and the corresponding computational overhead increases. =2 in =3 achieves higher ES and lower OER while maintaining the same Strict-C, therefore... =2 is used as the main setting in this embodiment.

[0177] As shown in Table 4, after removing the reference answer anchoring mechanism, MedConSPEdit-P's Strict-C score on PMC-VQA decreased to 48.3%, while the CDR score increased to 18.3%. The value of -0.5024 indicates that the reference answer anchoring mechanism plays an important role in protecting conflicting samples; retaining the reference answer anchoring but adopting At that time, the Strict-C level was 65.9%, and the CDR level was 12.9%. -0.0051; using At that time (primary setting), Strict-C was 67.0%, and CDR was 13.4%. +0.1790; using At that time, the Strict-C level was 67.0%, and the CDR level was 11.8%. It is +0.4004.

[0178] The above results demonstrate that this embodiment, while maintaining a high target editing success rate, significantly improves the preservation of reference answers for conflict samples, locality within the same graph, and candidate score intervals, verifying the technical effectiveness of the method of this invention in correcting target medical errors while effectively avoiding conflict drift. During the inference phase, no training conflict samples or locality samples are loaded, and backpropagation, Gram matrix factorization, conflict-sensitive subspace estimation, or reference answer anchoring are performed. Only the edited LoRA adapter is loaded for forward computation, ensuring the simplicity and efficiency of deployment. The basic model, data scale, parameter configuration, training steps, and evaluation set listed in this embodiment are only used to illustrate specific feasible implementations of the invention and do not constitute a limitation on the type of basic visual language model, lightweight parameter form, knowledge editing scale, or source of medical visual question answering data.

[0179] Based on the same inventive concept, another aspect of this application provides an electronic device. This electronic device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores a computer program / instructions executable by the at least one processor. When executed by the at least one processor, the computer program / instructions enable the at least one processor to execute and implement the various operational steps in the conflict-preserving knowledge editing method for the medical visual question-answering model described in any of the above embodiments. In practical deployment, this electronic device can be a standalone deep learning training server, a cloud-based distributed computing node cluster, or an intelligent computing terminal built into an image-assisted diagnostic system within a medical institution.

[0180] Another aspect of this application provides a computer-readable storage medium storing a computer program or instructions thereon. When the computer program or instructions are read and executed by a processor of a computing device, the computing device is able to implement the various steps of the conflict-preserving knowledge editing method for the medical visual question-answering model described in any of the above embodiments. Those skilled in the art should understand that the computer-readable storage medium can be a non-volatile computer-readable medium, such as, but not limited to, read-only memory (ROM), random access memory (RAM), flash memory, solid-state drive (SSD), or optical data storage media, to ensure the continuous and stable transmission of the data flow and control logic required for model training, gradient calculation, and knowledge editing.

[0181] Another aspect of this application provides a computer program product containing instructions. When the computer program product is downloaded and installed on a computer or artificial intelligence processing device and is run, the computer or processing device causes the computer or processing device to perform the corresponding steps of the conflict-preserving knowledge editing method for the medical visual question-answering model described in any of the above method embodiments.

[0182] Furthermore, it must be pointed out that the various embodiments, preferred embodiments, and technical features described in this disclosure possess a high degree of compatibility and interoperability within the overall algorithm design and technical concept. Without violating the basic algorithmic logic and the common sense of multimodal data processing, those skilled in the art can flexibly extract, combine, nest, or equivalently replace the aforementioned technical features (such as conflict sample construction dimensions, subspace estimation methods, and gradient protection fusion strategies) according to specific medical clinical question-and-answer application scenarios, the architectural characteristics of basic visual language models, and actual computational constraints. Any logical modifications, recombinations, or derivations based on the aforementioned technical features should undoubtedly be considered within the scope of protection claimed in this application.

Claims

1. A conflict-preserving knowledge editing method for a medical visual question-answering model, characterized in that, The method includes: Step 1: Establish a medical visual question answering model, set lightweight editable parameters while freezing the main parameters of the medical visual question answering model, and obtain the basic prediction results of medical question answering samples; Step 2: Based on the basic prediction results, filter the target editing samples to be corrected, and construct medical conflict samples with directional correlation corresponding to the target editing samples based on medical structural information; Step 3: Calculate the target editing gradient based on the target editing sample and the conflict gradient based on the medical conflict sample, respectively, and use the conflict gradient to estimate the conflict-sensitive subspace corresponding to the current target editing sample; Step 4: Project the target editing gradient onto a conflict-sensitive subspace to obtain a conflict hazard component; calculate the overlap between the target editing gradient and the conflict-sensitive subspace to adaptively determine the protection strength; and attenuate the conflict hazard component in the target editing gradient based on the protection strength to obtain a safe editing gradient. Step 5: Fuse the secure editing gradient with the reference answer anchoring gradient for the medical conflict sample to obtain the final update gradient, and use the final update gradient to update the lightweight editable parameters.

2. The conflict-preserving knowledge editing method for the medical visual question-answering model according to claim 1, characterized in that, The medical structural information includes at least one of the following: medical predicates, standardized medical relations, fixed answer space, question templates, image sources, imaging modalities, and anatomical locations; The construction of directionally related medical conflict samples corresponding to the target editing sample based on medical structural information includes constructing at least one of the following samples whose basic prediction result is correct before editing: A first-level natural conflict sample is one that has the same or highly consistent medical predicates or normalized medical relations as the target edit sample, has the same or a fixed answer space that meets the preset compatibility judgment conditions, and uses different medical image evidence. The target edit answer of the target edit sample exists in the candidate set of the conflict sample, and the reference answer of the conflict sample is different from the target edit answer. A first-level answer space conflict sample is a fixed answer space that has the same or satisfies the preset compatibility judgment conditions as the target editing sample. The target editing answer exists in the candidate set of the conflict sample, and the reference answer of the conflict sample is different from the target editing answer. A second-level relation conflict sample is one that has at least one complete association relationship with the target editing sample, including the same medical predicate, normalized medical relation, or answer space, and the reference answer of the conflict sample is different from the target editing answer. A conflicting sample is one whose question template is the same as the target edit sample's question template, or is determined to be semantically equivalent by a preset question template normalization rule or semantic similarity threshold, and uses different medical image evidence. The reference answer of the conflicting sample is different from the target edit answer.

3. The conflict-preserving knowledge editing method for the medical visual question-answering model according to claim 1 or 2, characterized in that, The conflict-sensitive subspace corresponding to the current target edit sample is estimated using the conflict gradient, including: Step S31: Calculate the score of the target edit answer of the target edit sample or the reference answer of the medical conflict sample on the medical conflict sample, construct the conflict injection loss or the reference answer preservation loss based on the score, and differentiate the loss with respect to the lightweight editable parameter to obtain the original conflict gradient; Step S32: Normalize each valid non-zero original conflict gradient with a 2-norm with a stabilizing constant, scale it according to the square root of the conflict level weight, and combine them column by column to form a conflict gradient matrix. Step S33: Construct a gradient inner product matrix with regularization terms, i.e., Gram matrix, based on the conflict gradient matrix. Perform eigenvalue decomposition on the Gram matrix and select the eigenvectors corresponding to the first k eigenvalues ​​in descending order of eigenvalues. Map the eigenvectors to the gradient space of the lightweight editable parameters through the conflict gradient matrix to obtain the conflict principal direction. The conflict sensitive subspace is spanned by the conflict principal direction.

4. The conflict-preserving knowledge editing method for the medical visual question-answering model according to claim 1, characterized in that, Projecting the target editing gradient onto a conflict-sensitive subspace to obtain a conflict hazard component, calculating the overlap between the target editing gradient and the conflict-sensitive subspace to adaptively determine the protection strength, and attenuating the conflict hazard component in the target editing gradient based on the protection strength to obtain a safe editing gradient, including: Step S41: Construct a regularized approximate projection operator based on the conflict-sensitive subspace, calculate the projection component of the target editing gradient in the conflict-sensitive subspace, and use it as the conflict danger component; Step S42: Calculate the ratio of the length of the conflict-prone component to the length of the target editing gradient with a stability constant, as the degree of overlap, i.e., the relative overlap rate; determine an adaptive protection coefficient between 0 and 1 based on the ratio of the relative overlap rate to a preset overlap threshold, as the protection strength; wherein, the larger the relative overlap rate, the larger the adaptive protection coefficient. Step S43: Subtract the conflict risk component multiplied by the adaptive protection coefficient from the target editing gradient to attenuate the conflict risk component and obtain the safe editing gradient.

5. The conflict-preserving knowledge editing method for the medical visual question-answering model according to claim 1, characterized in that, The method further includes: In step 2, a generalized sample is further constructed that preserves the semantics of the target edit answer of the medical image evidence and the target edit sample, and a local sample that is correctly predicted based on the pre-editing prediction result is selected; wherein, the local sample includes at least one of the following: local sample in the same image, high semantic local sample, and random local sample; In step 3, the target editing loss based on the target editing sample and the generalization loss based on the generalization sample are calculated respectively. The two are combined and differentiated with respect to the lightweight editable parameter to obtain the target editing gradient. In step 5, when fusing the safe editing gradient with the reference answer anchoring gradient for the medical conflict sample, the locality preservation gradient obtained by differentiating the locality sample is further fused to obtain the final update gradient.

6. The conflict-preserving knowledge editing method for the medical visual question-answering model according to claim 1, characterized in that, The lightweight editable parameters are low-rank adaptation parameters, which are updated by adding low-rank increments to the frozen pre-trained weight matrix. When obtaining the basic prediction results of the medical question-and-answer sample in step 1, the candidate answer conditional probability scoring method is adopted. Specifically, for each candidate answer in the fixed candidate answer set, the average conditional log probability of the word elements contained therein is calculated, and the candidate answer with the highest average conditional log probability is taken as the basic prediction result.

7. The conflict-preserving knowledge editing method for the medical visual question-answering model according to claim 1, characterized in that, For multiple consecutive medical knowledge editing tasks, the conflict-sensitive subspaces corresponding to different target editing samples are dynamically estimated based on the current model state and the current medical conflict samples, rather than using a pre-fixed unified subspace. In the inference phase after the medical visual question answering model completes knowledge editing, the medical visual question answering model only loads the updated lightweight editable parameters to perform the forward computation process. The inference phase does not load the medical conflict samples, does not calculate the conflict-sensitive subspace, and does not perform gradient backpropagation and subspace projection operations.

8. An electronic device, characterized in that, The electronic device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-7.

Images