Large model closed-loop knowledge forgetting editing method and system based on multi-modal knowledge graph constraint

Abstract

The application discloses a kind of big model closed loop knowledge forgetful editing method and system based on multi-modal knowledge graph constraint, it is related to artificial intelligence and knowledge engineering technical field, big model closed loop knowledge forgetful editing method based on multi-modal knowledge graph constraint mainly includes: the influence subgraph of to-be-forgotten knowledge in multi-modal knowledge graph is constructed, in combination with text, image and atlas path etc. Variety of detection methods, the target knowledge residue in big model is detected, and the big model is edited based on atlas constraint and localized forgetfulness, finally whether the target knowledge is reliably forgotten is judged by closed loop verification mechanism. The reliability and applicability of knowledge forgetful editing can be improved by the big model closed loop knowledge forgetful editing method and system based on multi-modal knowledge graph constraint provided by the application.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and knowledge engineering technology, and more specifically, to a method and system for editing closed-loop knowledge forgetting in large models based on multimodal knowledge graph constraints. Background Technology

[0002] With the development of large language models and multimodal large models, these models have demonstrated strong capabilities in tasks such as natural language understanding, image understanding, cross-modal question answering, and knowledge reasoning. Meanwhile, knowledge graphs, as a structured knowledge representation method, can describe knowledge relationships in the real world through entities, relations, and attributes; multimodal knowledge graphs further incorporate multimodal information such as images, text descriptions, and video clips, providing richer semantic evidence to support the same entity or relation. Therefore, combining multimodal knowledge graphs with large models has become an important way to improve the knowledge representation and reasoning capabilities of models.

[0003] However, in applications such as privacy protection, copyright compliance, sensitive information removal, harmful knowledge removal, and outdated knowledge correction, some knowledge already learned or memorized by large models may need to be deleted, blocked, or forgotten. Existing methods for knowledge forgetting and editing in large models typically reduce the model's memorization of target knowledge through fine-tuning, parameter editing, reverse optimization, and cue constraints. However, these methods are mostly geared towards single textual facts, question-and-answer samples, or structured triples, and are difficult to adapt to multimodal knowledge graph scenarios.

[0004] Specifically, existing technologies have at least the following shortcomings: First, the localization of knowledge to be forgotten is incomplete. In multimodal knowledge graphs, a target knowledge may simultaneously exist in entity nodes, relation edges, attribute nodes, text descriptions, image evidence, and adjacency reasoning paths. Forgetting only a single text or a single triple makes it difficult to determine the true scope of impact. Second, cross-modal knowledge residue is prone to occur. The model may no longer output the target knowledge in text question answering, but it may still recover the knowledge through inference from related images, entity attributes, or graph paths, resulting in superficial forgetting rather than true forgetting. Third, coarse-grained forgetting can easily damage non-target knowledge. Directly updating a large range of parameters may damage the non-sensitive attributes of the target entity, neighbor entity relationships, and the model's general language understanding, image understanding, and graph reasoning capabilities.

[0005] Therefore, there is a need for a large-scale closed-loop knowledge forgetting and editing method and system based on multimodal knowledge graph constraints, which can perform multimodal association localization, cross-modal residue detection and localized forgetting editing of knowledge to be forgotten, and improve forgetting reliability through a closed-loop verification mechanism, while maintaining non-target knowledge and model generality. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for editing closed-loop knowledge forgetting in large models based on multimodal knowledge graph constraints, which can improve the reliability and applicability of knowledge forgetting editing.

[0007] This invention provides a method for editing closed-loop knowledge forgetting in large models based on multimodal knowledge graph constraints, comprising the following steps: S1: Transform knowledge forgetting requests in natural language form into structured forgetting targets; S2: Align the multimodal knowledge graph by mapping the knowledge representations of different modalities to a unified semantic space to obtain the aligned multimodal knowledge graph; S3: Based on the forgetting target and the multimodal knowledge graph, construct a subgraph of influence to be forgotten, and calculate the forgetting weights of nodes and edges in the subgraph of influence to be forgotten; S4: Generate probe samples based on the subgraph of influence to be forgotten, input the probe samples into the large model to be edited to obtain the output results, and calculate the target knowledge residual score based on the probe samples and the output results; S5: Based on the residual score of the target knowledge, the forgetting influence subgraph and its node and edge forgetting weights, determine the editable parameter region in the large model to be edited, perform localized forgetting update on the editable parameter region, so that the large model to be edited reduces the output probability of the target knowledge, and obtain the edited large model; S6: Construct a knowledge set to be retained based on the multimodal knowledge graph, and apply output consistency constraints to the edited large model using the retention loss function and the original large model to be edited, to obtain a new edited large model; S7: Input the cross-modal validation sample into the new edited large model, recalculate the target knowledge residual score, confirm that the target knowledge residual score is higher than the preset threshold, and return to step S5; confirm that the target knowledge residual score is not higher than the preset threshold, and output the new edited large model and the corresponding forgetting validation results.

[0008] Furthermore, the method for constructing the subgraph of influence to be forgotten includes neighborhood expansion, path search, and cross-modal evidence retrieval; the subgraph of influence to be forgotten includes target triples, first-order neighbors of the target entity, second-order neighbors of the target entity, relevant attribute nodes, relevant image nodes, relevant text nodes, and inferenceable paths.

[0009] Furthermore, the formula for calculating the forgetting weight is as follows:

[0010] in, Represents a node Forgetting weights; For nodes semantic representation Semantic representation of the target to be forgotten The similarity between them; For nodes To the set of entities to be forgotten The shortest graph distance; Represents a node The relevance of cross-modal evidence; These are adjustable weighting coefficients.

[0011] Furthermore, the formula for calculating the residual score of the target knowledge is as follows:

[0012] in, Residual score for the target knowledge; Indicates the number of samples detected; This indicates that the large model to be edited is in the input field. A cross-modal detection sample Output the target answer or target knowledge to be forgotten under the given conditions. The probability of.

[0013] Furthermore, the editable parameter region includes at least one of the following from the large model to be edited: attention layer, feedforward network layer, cross-modal projection layer, knowledge injection layer, retrieval enhancement module, LoRA parameters, adapter parameters, cue parameters, and learnable forgetting gating parameters.

[0014] Furthermore, step S5 specifically includes: Based on the forgetting weights of the subgraph to be forgotten and its nodes and edges, different editing intensities are applied to different sources of evidence: for nodes, relations, and cross-modal evidence with higher forgetting weights, the system applies stronger forgetting constraints; for nodes and relations with lower forgetting weights or those within the scope of knowledge retention, the system reduces the editing intensity. Freeze the main parameters of the large model to be edited, introduce only the learnable forgetting edit parameters, and use the forgetting loss function to reduce the output probability of the model for the target knowledge to obtain the model parameters of the edited large model, as shown in the formula:

[0015] in, This represents the model parameters of the original large model to be edited. This indicates that the edit parameters were forgotten. This indicates that the edited model parameters have been forgotten.

[0016] Furthermore, the formula for calculating the forgetting loss function is as follows:

[0017] in, Forgetting loss function; Indicates the number of samples to be forgotten; This indicates that the large model to be edited is in the input field. A sample to be forgotten Output the target answer or target knowledge to be forgotten under the given conditions. The probability of.

[0018] Furthermore, the formula for calculating the retention loss function is as follows:

[0019] in, To preserve the loss function; Indicates the number of samples retained; Indicates KL divergence; Indicates the first in the knowledge set to be retained A sample of retained knowledge The output distribution of the original model; Indicates the first in the knowledge set to be retained A sample of retained knowledge The output distribution of the model after editing.

[0020] Furthermore, the aforementioned large-scale model closed-loop knowledge forgetting editing method based on multimodal knowledge graph constraints also includes: updating the model parameters of the edited large-scale model using a joint loss function, wherein the calculation formula for the joint loss function is:

[0021]

[0022] in, For the joint loss function; This represents the loss due to forgetting, used to reduce the probability of outputting the target knowledge. This represents retention loss, used to preserve non-target knowledge; This represents the cross-modal consistent forgetting loss, used to constrain the forgetting of target knowledge in text, image, and graph paths; This represents the sparse editing loss, used to limit the size of editing parameters and reduce their impact on the overall capabilities of the model. , , , For the corresponding loss weight coefficients; where, Represents text modality, Represents image modality, Represents the path mode of the graph. Indicates the first Forgetting loss in various modalities This represents the weights corresponding to different modes.

[0023] This invention also provides a large-scale closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints, comprising the following modules: The forgetting request parsing module is used to convert knowledge forgetting requests in natural language form into structured forgetting targets; The multimodal knowledge graph construction and alignment module is used to: align the multimodal knowledge graph, map the knowledge representations of different modalities to a unified semantic space, and obtain the aligned multimodal knowledge graph; The module for locating the subgraph of influence to be forgotten is used to: construct the subgraph of influence to be forgotten based on the forgetting target and the multimodal knowledge graph, and calculate the forgetting weights of nodes and edges in the subgraph of influence to be forgotten; The cross-modal knowledge residue detection module is used to: generate detection samples based on the subgraph of influence to be forgotten, input the detection samples into the large model to be edited to obtain the output results, and calculate the target knowledge residue score based on the detection samples and the output results; The graph-constrained forgetting editing module is used to: determine the editable parameter region in the large model to be edited based on the residual score of the target knowledge, the forgetting influence subgraph and its nodes and edges, and perform localized forgetting update on the editable parameter region to reduce the output probability of the target knowledge in the large model to be edited, thereby obtaining the edited large model; The knowledge preservation module is used to: construct a set of preserved knowledge based on the multimodal knowledge graph, apply output consistency constraints to the edited large model using the preservation loss function and the original large model to be edited, and obtain a new edited large model; The cross-modal closed-loop verification module is used to: input the cross-modal verification sample into the new edited large model, recalculate the target knowledge residual score, confirm that the target knowledge residual score is higher than the preset threshold, and return to step S5; confirm that the target knowledge residual score is not higher than the preset threshold, and output the new edited large model and the corresponding forgetting verification result.

[0024] The method and system for editing closed-loop knowledge based on multimodal knowledge graph constraints in large models, as provided in this invention, have the following beneficial effects: This invention addresses the problems of incomplete localization of knowledge to be forgotten, significant cross-modal knowledge residue, and the easy misidentification of non-target knowledge in multimodal knowledge graph scenarios in existing large-scale knowledge forgetting and editing methods. By constructing an influence subgraph of the knowledge to be forgotten in the multimodal knowledge graph, and combining multiple detection methods such as text, image, and graph path, the invention detects the residual target knowledge in the large model, performs localized forgetting and editing of the large model based on graph constraints, and finally uses a closed-loop verification mechanism to determine whether the target knowledge has been reliably forgotten.

[0025] Specifically, this invention addresses the shortcomings of existing large-scale knowledge forgetting editing methods, which typically process single textual facts, question-and-answer samples, or structured triples, making it difficult to cover the associated evidence of target knowledge in multimodal knowledge graphs. By using a forgetting request parsing module and a subgraph localization module for the influence of the knowledge to be forgotten, the present invention expands the knowledge to be forgotten into an influence subgraph composed of target entities, target relationships, attribute nodes, text descriptions, image evidence, and adjacency reasoning paths. This allows for a more complete determination of the actual influence range of the knowledge to be forgotten, reducing knowledge residue caused by incomplete localization of the forgotten target, and solving the problem of incomplete localization of knowledge to be forgotten in multimodal knowledge graph scenarios.

[0026] This invention addresses the shortcomings of existing methods, which may cause the model to no longer output target knowledge in direct text question answering, but the target knowledge can still be recovered through related images, attribute completion, entity association, or graph path reasoning. It sets up a cross-modal knowledge residue detection module, which generates text detection samples, image detection samples, and graph path detection samples based on the subgraph of the influence to be forgotten. This allows for multi-angle detection of target knowledge residue in large models. In this way, implicit knowledge residue that is difficult to detect with single text detection can be discovered, improving the reliability of knowledge forgetting results and solving the problems of cross-modal residue and surface forgetting of target knowledge.

[0027] This invention addresses the shortcomings of existing large-scale model knowledge forgetting methods, which often involve extensive updates to model parameters during forgetting, impacting the non-sensitive attributes of target entities, adjacent entity relationships, and the model's general language understanding, image understanding, and graph reasoning capabilities. By employing a graph-constrained forgetting editing module and a knowledge preservation protection module, the invention distinguishes between knowledge to be forgotten and knowledge to be retained. Forgetting constraints are applied to target knowledge, while retention constraints are applied to non-target knowledge. This approach achieves the forgetting of target knowledge while preserving non-target knowledge and the model's general capabilities as much as possible, thus solving the problem that coarse-grained forgetting easily damages non-target knowledge.

[0028] This invention addresses the shortcomings of existing methods that often employ a one-time editing approach, making it difficult to promptly determine whether target knowledge can still be recovered through other modalities or graph paths. By employing a cross-modal closed-loop verification module, the residual score of the target knowledge is re-detected after each round of forgetting editing. If the residual score is higher than a preset threshold, the residual samples, residual paths, and residual modalities are fed back to the graph-constrained forgetting editing module to continue local forgetting editing. If the residual score is lower than the preset threshold, the large model after forgetting editing and the verification results are output. This forms a closed-loop process of "location—detection—editing—verification—feedback," improving the controllability, verifiability, and stability of the knowledge forgetting process, and solving the problems of lacking closed-loop verification and uncontrollable forgetting results in existing knowledge forgetting editing processes.

[0029] This invention enables multimodal association localization and constrained forgetting editing of target knowledge in knowledge environments where text, images, and graph structures coexist. It is applicable to scenarios such as deletion of personal privacy information, updating of enterprise private knowledge, shielding of sensitive entity relationships, correction of outdated knowledge, and removal of non-compliant content. Compared to methods relying solely on text question answering or single-point triple editing, this invention better adapts to the practical application needs of large-scale multimodal knowledge graph enhancement models, improving its applicability in scenarios such as privacy protection, copyright compliance, deletion of sensitive knowledge, and correction of outdated knowledge. Attached Figure Description

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the large-scale closed-loop knowledge forgetting and editing method based on multimodal knowledge graph constraints provided by the present invention; Figure 2 This is a schematic diagram of the framework of a large-scale closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints provided by the present invention. Detailed Implementation

[0031] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] Figure 1 This diagram illustrates a large-model closed-loop knowledge forgetting and editing method based on multimodal knowledge graph constraints, as described in this embodiment. In this embodiment, the large-model closed-loop knowledge forgetting and editing method based on multimodal knowledge graph constraints includes the following steps: S1: Transform knowledge forgetting requests in natural language form into structured forgetting targets; In one exemplary embodiment, the method for converting the forgetting target is to perform entity recognition, relation recognition, attribute recognition, and intent recognition on the knowledge forgetting request; The forgetting targets include a set of entities to be forgotten, a set of relations to be forgotten, a set of attributes to be forgotten, a set of triples to be forgotten, and a corresponding set of multimodal evidence.

[0033] S2: Align the multimodal knowledge graph by mapping the knowledge representations of different modalities to a unified semantic space to obtain the aligned multimodal knowledge graph; In one exemplary embodiment, the multimodal knowledge graph includes entity nodes, relation edges, attribute nodes, image nodes, text nodes, and cross-modal alignment edges.

[0034] S3: Based on the forgetting target and the multimodal knowledge graph, construct a subgraph of influence to be forgotten, and calculate the forgetting weights of nodes and edges in the subgraph of influence to be forgotten; In one exemplary embodiment, the method for constructing the subgraph of influence to be forgotten includes neighborhood expansion, path search, and cross-modal evidence retrieval; The subgraph of influence to be forgotten includes target triples, first-order neighbors of the target entity, second-order neighbors of the target entity, related attribute nodes, related image nodes, related text nodes, and inference paths.

[0035] In one exemplary embodiment, the formula for calculating the forgetting weight is:

[0036] in, Represents a node Forgetting weights; For nodes semantic representation Semantic representation of the target to be forgotten The similarity between them; For nodes To the set of entities to be forgotten The shortest graph distance; Represents a node The relevance of cross-modal evidence; These are adjustable weighting coefficients.

[0037] S4: Generate probe samples based on the subgraph of influence to be forgotten, input the probe samples into the large model to be edited to obtain the output results, and calculate the target knowledge residual score based on the probe samples and the output results; In one exemplary embodiment, the probe samples include direct text question-and-answer samples, rewritten text question-and-answer samples, image question-and-answer samples, entity attribute completion samples, relationship prediction samples, graph path reasoning samples, and counterfactual probe samples.

[0038] In one exemplary embodiment, the formula for calculating the target knowledge residual score is:

[0039] in, Residual score for the target knowledge; Indicates the number of samples detected; This indicates that the large model to be edited is in the input field. A cross-modal detection sample Output the target answer or target knowledge to be forgotten under the given conditions. The probability of.

[0040] S5: Based on the residual score of the target knowledge, the forgetting influence subgraph and its node and edge forgetting weights, determine the editable parameter region in the large model to be edited, perform localized forgetting update on the editable parameter region, so that the large model to be edited reduces the output probability of the target knowledge, and obtain the edited large model; In one exemplary embodiment, the editable parameter region includes at least one of the following from the large model to be edited: a partial attention layer, a feedforward network layer, a cross-modal projection layer, a knowledge injection layer, a retrieval enhancement module, LoRA parameters, adapter parameters, cue parameters, and learnable forgetting gating parameters.

[0041] In one exemplary embodiment, step S5 specifically includes: Based on the forgetting weights of the subgraph to be forgotten and its nodes and edges, different editing intensities are applied to different sources of evidence: for nodes, relations, and cross-modal evidence with higher forgetting weights, the system applies stronger forgetting constraints; for nodes and relations with lower forgetting weights or those within the scope of knowledge retention, the system reduces the editing intensity. Freeze the main parameters of the large model to be edited, introduce only the learnable forgetting edit parameters, and use the forgetting loss function to reduce the output probability of the model for the target knowledge to obtain the model parameters of the edited large model, as shown in the formula:

[0042] in, This represents the model parameters of the original large model to be edited. This indicates that the edit parameters were forgotten. This indicates that the edited model parameters were forgotten; In one exemplary embodiment, the formula for calculating the forgetting loss function is:

[0043] in, Forgetting loss function; Indicates the number of samples to be forgotten; This indicates that the large model to be edited is in the input field. A sample to be forgotten Output the target answer or target knowledge to be forgotten under the given conditions. The probability of.

[0044] S6: Construct a knowledge set to be retained based on the multimodal knowledge graph, and apply output consistency constraints to the edited large model using the retention loss function and the original large model to be edited, to obtain a new edited large model; In one exemplary embodiment, the retained knowledge set includes entity relationships unrelated to the target to be forgotten, non-sensitive attributes of the target entity, normal relationships of adjacent entities, ordinary text question-answering capabilities, image understanding capabilities, and graph reasoning capabilities.

[0045] In one exemplary embodiment, the formula for calculating the retention loss function is:

[0046] in, To preserve the loss function; Indicates the number of samples retained; Indicates KL divergence; Indicates the first in the knowledge set to be retained A sample of retained knowledge The output distribution of the original model; Indicates the first in the knowledge set to be retained A sample of retained knowledge The output distribution of the model after editing.

[0047] S7: Input the cross-modal validation sample into the new edited large model, recalculate the target knowledge residual score, confirm that the target knowledge residual score is higher than the preset threshold, and return to step S5; confirm that the target knowledge residual score is not higher than the preset threshold, and output the new edited large model and the corresponding forgetting validation results.

[0048] In one exemplary embodiment, the method further includes: updating the model parameters of the edited large model using a joint loss function, wherein the formula for calculating the joint loss function is:

[0049]

[0050] in, For the joint loss function; This represents the loss due to forgetting, used to reduce the probability of outputting the target knowledge. This represents retention loss, used to preserve non-target knowledge; This represents the cross-modal consistent forgetting loss, used to constrain the forgetting of target knowledge in text, image, and graph paths; This represents the sparse editing loss, used to limit the size of editing parameters and reduce their impact on the overall capabilities of the model. , , , For the corresponding loss weight coefficients; where, Represents text modality, Represents image modality, Represents the path mode of the graph. Indicates the first Forgetting loss in various modalities This represents the weights corresponding to different modes.

[0051] This embodiment provides a large-scale closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints, including the following modules: The forgetting request parsing module is used to convert knowledge forgetting requests in natural language form into structured forgetting targets; The multimodal knowledge graph construction and alignment module is used to: align the multimodal knowledge graph, map the knowledge representations of different modalities to a unified semantic space, and obtain the aligned multimodal knowledge graph; The module for locating the subgraph of influence to be forgotten is used to: construct the subgraph of influence to be forgotten based on the forgetting target and the multimodal knowledge graph, and calculate the forgetting weights of nodes and edges in the subgraph of influence to be forgotten; The cross-modal knowledge residue detection module is used to: generate detection samples based on the subgraph of influence to be forgotten, input the detection samples into the large model to be edited to obtain the output results, and calculate the target knowledge residue score based on the detection samples and the output results; The graph-constrained forgetting editing module is used to: determine the editable parameter region in the large model to be edited based on the residual score of the target knowledge, the forgetting influence subgraph and its nodes and edges, and perform localized forgetting update on the editable parameter region to reduce the output probability of the target knowledge in the large model to be edited, thereby obtaining the edited large model; The knowledge preservation module is used to: construct a set of preserved knowledge based on the multimodal knowledge graph, apply output consistency constraints to the edited large model using the preservation loss function and the original large model to be edited, and obtain a new edited large model; The cross-modal closed-loop verification module is used to: input the cross-modal verification sample into the new edited large model, recalculate the target knowledge residual score, confirm that the target knowledge residual score is higher than the preset threshold, and return to step S5; confirm that the target knowledge residual score is not higher than the preset threshold, and output the new edited large model and the corresponding forgetting verification result.

[0052] In some embodiments, the above-described large-model closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints can also be implemented in the following ways.

[0053] like Figure 2 As shown, the large-scale closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints mainly includes the following modules: forgetting request parsing module; multimodal knowledge graph construction and alignment module; subgraph localization module for the influence of the to-be-forgotten knowledge graph; cross-modal knowledge residue detection module; graph constraint forgetting and editing module; knowledge preservation module; and cross-modal closed-loop verification module.

[0054] The working principles, connection relationships, and cooperation relationships of each module are as follows.

[0055] 1. Forgot Request Parsing Module The forgetting request parsing module is mainly used to receive knowledge forgetting requests input by users and convert forgetting requests in natural language form into structured forgetting targets.

[0056] Specifically, users can input forgetting requests in the following forms: delete a sensitive attribute of an entity, forget the relationship between an entity and another entity, prevent the model from recognizing specific identity information based on an image, or prevent the model from outputting outdated, erroneous, or non-compliant knowledge. Upon receiving a forgetting request, the system first performs entity recognition, relationship recognition, attribute recognition, and intent recognition on the request text to obtain a set of entities to be forgotten, a set of relationships to be forgotten, a set of attributes to be forgotten, and a set of triples to be forgotten.

[0057] For example, for the forgetting request "make the model no longer answer the relationship R between entity A and entity B", this module parses it into a set of entities to be forgotten. Set of relations to be forgotten and the set of triples to be forgotten If the forgetting request contains images, text descriptions, or other modal evidence, then further parsing yields a set of image evidence to be forgotten. and the set of textual evidence to be forgotten .

[0058] The output of this module serves as the input to the multimodal knowledge graph construction and alignment module and the module for locating the subgraph of influence to be forgotten.

[0059] 2. Multimodal Knowledge Graph Construction and Alignment Module The multimodal knowledge graph construction and alignment module is mainly used to construct or call existing multimodal knowledge graphs and to uniformly represent entities, relationships, attributes, images, and text descriptions within them. The multimodal knowledge graph can be constructed from external knowledge bases, enterprise private knowledge bases, public multimodal knowledge bases, or user-uploaded data.

[0060] The multimodal knowledge graph includes entity nodes, relation edges, attribute nodes, image nodes, text nodes, and cross-modal alignment edges. Entity nodes represent objects such as people, places, organizations, goods, and events; relation edges represent semantic relationships between entities; attribute nodes represent attributes such as category, time, location, identity, and function of entities; image nodes represent visual evidence related to entities or relationships; text nodes represent entity descriptions, event descriptions, web page fragments, document content, or question-and-answer knowledge; and cross-modal alignment edges represent the correspondence between entity nodes and their corresponding image or text nodes.

[0061] In this module, a multimodal knowledge graph can be represented as:

[0062] in, Represents a set of entity nodes. Represents a set of image nodes. Represents a collection of text nodes. Represents a collection of attribute nodes. Represents the set of edges that represent relationships between entities. This represents the set of cross-modal aligned edges.

[0063] This module obtains text features, image features, and graph structure features through a text encoder, an image encoder, and a graph structure encoder, respectively, and maps them to a unified semantic space for subsequent subgraph localization, residual detection, and knowledge editing.

[0064] 3. Subgraph localization module for the influence of being forgotten The subgraph localization module for the influence of the to-be-forgotten knowledge is mainly used to determine the influence range of the target knowledge in the multimodal knowledge graph based on the entities, relations, attributes and triples to be forgotten output by the forgetting request parsing module.

[0065] In multimodal knowledge graphs, knowledge to be forgotten does not only exist in a single triple, but may also exist in related images, text descriptions, entity aliases, neighboring entities, and graph reasoning paths. Therefore, this module focuses on the set of triples to be forgotten. Centered on this, neighborhood expansion, path search, and cross-modal evidence retrieval are performed in a multimodal knowledge graph to construct a subgraph of influences to be forgotten. .

[0066] The subgraph of influence to be forgotten includes target triples, first-order neighbors of the target entity, second-order neighbors of the target entity, relevant attribute nodes, relevant image nodes, relevant text nodes, and inference paths. For each node and relation edge, this module further calculates its relevance to the target to be forgotten and generates the corresponding forgetting weight.

[0067] Specifically, for the nodes in the subgraph that affect Its forgetting weight It can be calculated based on the semantic similarity between nodes and target entities, graph distance, and cross-modal evidence relevance:

[0068] in, Represents a node semantic representation, Semantic representation of the target to be forgotten. Represents a node To the set of entities to be forgotten The shortest graph distance, Represents a node Cross-modal evidence relevance, These are adjustable weighting coefficients.

[0069] The output of this module is the subgraph to be forgotten, along with its nodes and edge forgetting weights, providing a basis for subsequent cross-modal knowledge residue detection and graph-constrained forgetting editing.

[0070] 4. Cross-modal knowledge residual detection module The cross-modal knowledge residue detection module is mainly used to automatically construct detection samples based on the influence subgraph to be forgotten, and to detect whether the large model to be edited still retains the target knowledge.

[0071] This module generates various types of probe samples based on the subgraph of influence to be forgotten, including direct text question-and-answer samples, text rewriting question-and-answer samples, image question-and-answer samples, entity attribute completion samples, relationship prediction samples, graph path reasoning samples, and counterfactual probe samples.

[0072] For example, for the triples to be forgotten This module not only generates direct text questions such as "What is the relationship between entity A and entity B?", but also generates indirect probing questions such as "What entities can be inferred to be related to entity A based on its image?", "What relationships can be deduced from the adjacent entities of entity A?", and "If the attribute description of entity B is given, can the model recover entity B?".

[0073] After inputting the aforementioned probe samples into the large model to be edited, this module calculates the target knowledge residual score based on the model's output. The target knowledge residual score is used to measure whether the model can still recover the forgotten knowledge through text, images, or graph paths.

[0074] The residual score of the target knowledge can be expressed as:

[0075] in, Indicates the first A cross-modal detection sample, This indicates the target answer or target knowledge to be forgotten. This indicates that the large model to be edited is in the input... Output target knowledge under certain conditions The probability, This indicates the number of samples detected.

[0076] like If the value exceeds a preset threshold, it indicates that the model still has knowledge residue and forgetting editing is required; if... If the value is below the preset threshold, it indicates that the residual target knowledge is low, and the cross-modal closed-loop verification stage can be entered.

[0077] 5. Graph Constraint Forgotten Editing Module The graph-constrained forgetting editing module is mainly used to perform localized forgetting updates on the large model to be edited based on the subgraph of influence to be forgotten, forgetting weights, and knowledge residue scores, so that the model reduces or eliminates its ability to output target knowledge.

[0078] This module does not perform indiscriminate full parameter updates on the large model. Instead, it determines the model regions that need to be edited and the intensity of the edit based on the cross-modal knowledge residue detection results. The editable regions may include some attention layers, feedforward network layers, cross-modal projection layers, knowledge injection layers, retrieval enhancement modules, LoRA parameters, adapter parameters, cue parameters, or learnable forgetting gating parameters in the large model.

[0079] In one implementation, this module freezes the main parameters of the large model and only introduces learnable forgettable edit parameters. The edited model parameters are represented as follows:

[0080] in, Represents the parameters of the original large model. This indicates that the edit parameters were forgotten. This indicates that the edited model parameters have been forgotten.

[0081] This module reduces the model's output probability of the target knowledge through forgetting loss. Forgetting loss can be expressed as:

[0082] in, Indicates the first A sample to be forgotten during detection. Indicates the target answer to be forgotten. This represents the number of samples to be forgotten. By minimizing this loss, the model reduces the output probability of the target knowledge.

[0083] Meanwhile, this module combines the forgetting weights of nodes and edges in the subgraph to be forgotten to apply different editing intensities to different sources of evidence. For nodes, relationships, and cross-modal evidence with high forgetting weights, the system applies stronger forgetting constraints; for nodes and relationships with low forgetting weights or those within the scope of knowledge retention, the system reduces the editing intensity to minimize false negatives.

[0084] 6. Retain the knowledge protection module The knowledge preservation and protection module is mainly used to preserve non-target knowledge and the model's general capabilities without significant impairment while forgetting target knowledge.

[0085] This module constructs a knowledge retention set based on a multimodal knowledge graph. The knowledge set to be retained includes entity relationships unrelated to the target to be forgotten, non-sensitive attributes of the target entity, normal relationships between adjacent entities, ordinary text question-answering ability, image understanding ability, and graph reasoning ability.

[0086] During the forgetting and editing process, this module applies a consistency constraint to the outputs of the original and edited large models on the retained samples, ensuring that the outputs of the edited model on non-target knowledge are as close as possible to the original model. The retention loss can be expressed as:

[0087] in, Indicates the first A preserved knowledge sample, Indicates the number of samples to retain. Denotes KL divergence, This represents the original model output distribution. This indicates the output distribution of the model after editing.

[0088] This module helps prevent the model from destroying non-target knowledge when forgetting target knowledge, thus improving the selectivity and stability of knowledge forgetting and editing.

[0089] 7. Cross-modal closed-loop verification module The cross-modal closed-loop verification module is mainly used to perform cross-modal consistency verification on the large model after forgetting the edit, and to determine whether further editing is needed based on the verification results.

[0090] After each round of forgetting and editing, this module regenerates or calls cross-modal validation samples, inputs them into the edited large model, and calculates the new target knowledge residual score. .like Still higher than the preset threshold If so, the residual samples, residual paths, and residual modes are fed back to the map-constrained forgetting editing module to continue the next round of local forgetting editing. Below the preset threshold If so, it is determined that the target knowledge has met the forgetting requirements.

[0091] Simultaneously, this module calculates a retained knowledge impairment score to assess the changes in the model's ability to handle non-target knowledge after editing. If the retained knowledge impairment score exceeds a preset threshold, the system reduces the intensity of forgetting edits or reselects the editable parameter area.

[0092] Finally, when the residual score of the target knowledge is lower than the preset threshold and the degree of knowledge impairment meets the preset requirements, this module determines that the forgetting editing is complete and outputs the large model after forgetting editing and the corresponding forgetting verification results.

[0093] 8 Joint Optimization Objectives To simultaneously achieve target knowledge forgetting, cross-modal residue suppression, and non-target knowledge retention, this invention employs a joint optimization objective to update the forgetting editing parameters. The joint loss function can be expressed as:

[0094] in, This represents the loss due to forgetting, used to reduce the probability of outputting the target knowledge. This represents retention loss, used to preserve non-target knowledge; This represents the cross-modal consistent forgetting loss, used to constrain the forgetting of target knowledge in text, image, and graph paths; This represents the sparse editing loss, used to limit the size of editing parameters and reduce their impact on the overall capabilities of the model. This is the loss weighting coefficient.

[0095] The cross-modal consistent forgetting loss can be expressed as:

[0096] in, Represents text modality, Represents image modality, Represents the path mode of the graph. Indicates the first Forgetting loss in various modalities This represents the weights corresponding to different modes.

[0097] Through the above joint optimization, the present invention can avoid achieving superficial forgetting only in a single text question-and-answer scenario, but simultaneously suppress the knowledge recovery ability in images, text and graph paths.

[0098] It should be noted that the large-model closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints provided by this invention can be deployed in servers, workstations, edge computing devices, or cloud computing platforms. These devices include at least a processor, a memory, and a computing acceleration unit. The memory stores the multimodal knowledge graph, the large model to be edited, forgetting requests, probe samples, training samples, and forgetting verification results. The processor and computing acceleration unit perform operations such as knowledge parsing, subgraph retrieval, model reasoning, parameter editing, and verification calculations. The large model to be edited can be a large language model, a multimodal large model, a visual language model, or a large model enhanced with a knowledge graph. Before forgetting editing, the large model to be edited receives cross-modal probe samples and outputs knowledge residue results. During forgetting editing, it receives parameter updates from the graph-constrained forgetting editing module. After forgetting editing, it receives verification samples from the cross-modal closed-loop verification module and outputs verification results.

[0099] In some embodiments, the above-described large-model closed-loop knowledge forgetting and editing method based on multimodal knowledge graph constraints can also be implemented in the following ways.

[0100] In this embodiment, the large-scale closed-loop knowledge forgetting and editing method based on multimodal knowledge graph constraints includes the following steps: Step 1. Forgetting Request Parsing: Receive the knowledge forgetting request input by the user, and perform entity recognition, relation recognition, attribute recognition, and intent recognition on the knowledge forgetting request to obtain the set of entities to be forgotten, the set of relations to be forgotten, the set of attributes to be forgotten, the set of triples to be forgotten, and the corresponding multimodal evidence set.

[0101] Step 2. Multimodal Knowledge Graph Construction and Alignment: Construct or invoke a multimodal knowledge graph containing entity nodes, relation edges, attribute nodes, image nodes, text nodes, and cross-modal alignment edges, and map the knowledge representations of different modalities to a unified semantic space.

[0102] Step 3. Localization of the subgraph of influence to be forgotten: Using the triple to be forgotten as the center, perform neighborhood expansion, path search and cross-modal evidence retrieval in the multimodal knowledge graph to obtain the subgraph of influence to be forgotten, and calculate the forgetting weights of nodes and edges in the subgraph of influence.

[0103] Step 4. Cross-modal knowledge residue detection: Generate detection samples based on the subgraph of influence to be forgotten, such as direct text question answering, text rewriting question answering, image question answering, entity attribute completion, relation prediction and graph path reasoning. Input the detection samples into the large model to be edited and calculate the target knowledge residue score.

[0104] Step 5. Graph-constrained forgetting editing: Based on the residual score of the target knowledge, the subgraph of influence to be forgotten, and the forgetting weight, determine the editable parameter region in the large model, and perform localized forgetting update on the editable parameter region to reduce the output probability of the target knowledge by the model.

[0105] Step 6. Knowledge Preservation Protection: Construct a knowledge preservation set and apply output consistency constraints or feature consistency constraints to the edited large model to keep the model's capabilities stable in non-target knowledge, general entity relations, general language understanding, image understanding, and graph reasoning tasks.

[0106] Step 7. Closed-loop verification: Input the cross-modal verification samples into the edited large model and recalculate the target knowledge residual score. If the target knowledge residual score is higher than the preset threshold, feed back the residual samples and residual paths to the graph constraint forgetting editing module and return to step S5 to continue editing; if the target knowledge residual score is lower than the preset threshold, proceed to step S8.

[0107] Step 8. Output Results: When the target knowledge residual score is lower than the preset threshold and the degree of knowledge impairment meets the preset requirements, output the large model after forgetting and editing, and the corresponding forgetting verification results.

[0108] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for editing closed-loop knowledge forgetting in a large model based on multimodal knowledge graph constraints, characterized in that, Includes the following steps: S1: Transform knowledge forgetting requests in natural language form into structured forgetting targets; S2: Align the multimodal knowledge graph by mapping the knowledge representations of different modalities to a unified semantic space to obtain the aligned multimodal knowledge graph; S3: Based on the forgetting target and the multimodal knowledge graph, construct a subgraph of influence to be forgotten, and calculate the forgetting weights of nodes and edges in the subgraph of influence to be forgotten; S4: Generate probe samples based on the subgraph of influence to be forgotten, input the probe samples into the large model to be edited to obtain the output results, and calculate the target knowledge residual score based on the probe samples and the output results; S5: Based on the residual score of the target knowledge, the forgetting influence subgraph and its node and edge forgetting weights, determine the editable parameter region in the large model to be edited, perform localized forgetting update on the editable parameter region, so that the large model to be edited reduces the output probability of the target knowledge, and obtain the edited large model; S6: Construct a knowledge set to be retained based on the multimodal knowledge graph, and apply output consistency constraints to the edited large model using the retention loss function and the original large model to be edited, to obtain a new edited large model; S7: Input the cross-modal validation sample into the new edited large model, recalculate the target knowledge residual score, confirm that the target knowledge residual score is higher than the preset threshold, and return to step S5; confirm that the target knowledge residual score is not higher than the preset threshold, and output the new edited large model and the corresponding forgetting validation results.

2. The method according to claim 1, characterized in that, The method for constructing the subgraph of influence to be forgotten includes neighborhood expansion, path search, and cross-modal evidence retrieval; the subgraph of influence to be forgotten includes target triples, first-order neighbors of the target entity, second-order neighbors of the target entity, relevant attribute nodes, relevant image nodes, relevant text nodes, and inference paths.

3. The method according to claim 1, characterized in that, The formula for calculating the forgetting weight is: in, Represents a node Forgetting weights; For nodes semantic representation Semantic representation of the target to be forgotten The similarity between them; For nodes To the set of entities to be forgotten The shortest graph distance; Represents a node The relevance of cross-modal evidence; These are adjustable weighting coefficients.

4. The method according to claim 1, characterized in that, The formula for calculating the residual score of the target knowledge is as follows: in, Residual score of the target knowledge; Indicates the number of samples detected; This indicates that the large model to be edited is in the input field. A cross-modal detection sample Output the target answer or target knowledge to be forgotten under the given conditions. The probability of.

5. The method according to claim 1, characterized in that, The editable parameter region includes at least one of the following from the large model to be edited: attention layer, feedforward network layer, cross-modal projection layer, knowledge injection layer, retrieval enhancement module, LoRA parameters, adapter parameters, cue parameters, and learnable forgetting gating parameters.

6. The method according to claim 1, characterized in that, Step S5 specifically includes: Based on the forgetting weights of the subgraph to be forgotten and its nodes and edges, different editing intensities are applied to different sources of evidence: for nodes, relations, and cross-modal evidence with higher forgetting weights, the system applies stronger forgetting constraints; for nodes and relations with lower forgetting weights or those within the scope of knowledge retention, the system reduces the editing intensity. Freeze the main parameters of the large model to be edited, introduce only the learnable forgetting edit parameters, and use the forgetting loss function to reduce the output probability of the model for the target knowledge to obtain the model parameters of the edited large model, as shown in the formula: in, This represents the model parameters of the original large model to be edited. This indicates that the edit parameters were forgotten. This indicates that the edited model parameters have been forgotten.

7. The method according to claim 6, characterized in that, The formula for calculating the forgetting loss function is as follows: in, Forgetting loss function; Indicates the number of samples to be forgotten; This indicates that the large model to be edited is in the input field. A sample to be forgotten Output the target answer or target knowledge to be forgotten under the given conditions. The probability of.

8. The method according to claim 1, characterized in that, The formula for calculating the retention loss function is as follows: in, To preserve the loss function; Indicates the number of samples retained; Indicates KL divergence; Indicates the first in the knowledge set to be retained A sample of retained knowledge The output distribution of the original model; Indicates the first in the knowledge set to be retained A sample of retained knowledge The output distribution of the model after editing.

9. The method according to claim 1, characterized in that, Also includes: The model parameters of the edited large model are updated using a joint loss function, the formula for which the joint loss function is calculated is as follows: in, For the joint loss function; This represents the loss due to forgetting, used to reduce the probability of outputting the target knowledge. This represents retention loss, used to preserve non-target knowledge; This represents the cross-modal consistent forgetting loss, used to constrain the forgetting of target knowledge in text, image, and graph paths; This represents the sparse editing loss, used to limit the size of editing parameters and reduce their impact on the overall capabilities of the model. , , , For the corresponding loss weight coefficients; where, Represents text modality, Represents image modality, Represents the path mode of the graph. Indicates the first Forgetting loss in various modalities This represents the weights corresponding to different modes.

10. A large-scale closed-loop knowledge forgetting and editing system based on multimodal knowledge graph constraints, characterized in that, Includes the following modules: The forgetting request parsing module is used to convert knowledge forgetting requests in natural language form into structured forgetting targets; The multimodal knowledge graph construction and alignment module is used to: align the multimodal knowledge graph, map the knowledge representations of different modalities to a unified semantic space, and obtain the aligned multimodal knowledge graph; The module for locating the subgraph of influence to be forgotten is used to: construct the subgraph of influence to be forgotten based on the forgetting target and the multimodal knowledge graph, and calculate the forgetting weights of nodes and edges in the subgraph of influence to be forgotten; The cross-modal knowledge residue detection module is used to: generate detection samples based on the subgraph of influence to be forgotten, input the detection samples into the large model to be edited to obtain the output results, and calculate the target knowledge residue score based on the detection samples and the output results; The graph-constrained forgetting editing module is used to: determine the editable parameter region in the large model to be edited based on the residual score of the target knowledge, the forgetting influence subgraph and its nodes and edges, and perform localized forgetting update on the editable parameter region to reduce the output probability of the target knowledge in the large model to be edited, thereby obtaining the edited large model; The knowledge preservation module is used to: construct a set of preserved knowledge based on the multimodal knowledge graph, apply output consistency constraints to the edited large model using the preservation loss function and the original large model to be edited, and obtain a new edited large model; The cross-modal closed-loop verification module is used to: input the cross-modal verification sample into the new edited large model, recalculate the target knowledge residual score, confirm that the target knowledge residual score is higher than the preset threshold, and return to step S5; confirm that the target knowledge residual score is not higher than the preset threshold, and output the new edited large model and the corresponding forgetting verification result.

Images