Multi-modal generative model client cascading federated forgetting method, device, equipment and medium

Abstract

The application provides a multi-modal generative model client cascade federated forgetting method and device, equipment and medium, and the method comprises the following steps: based on the cumulative update record of the target client in the initial multi-modal generative model saved by the server in the federated learning training stage, the cross-modal shared parameter set, the modal specific parameter set and the cross-modal dependency parameter set in the initial multi-modal generative model, the contribution representation of the target client in the initial multi-modal generative model is determined; based on the architecture-aware dependency graph corresponding to the initial multi-modal generative model and the contribution representation, the parameter region related to the target client in the architecture-aware dependency graph is determined; the architecture-aware dependency graph is used for representing the forward calculation structure of the initial multi-modal generative model; based on the parameter region, the initial multi-modal generative model is subjected to forgetting processing to obtain the multi-modal generative model after forgetting. The application accurately identifies and locates the contribution of the target client, and eliminates the historical influence of the target client on the global model.

Description

Technical Field

[0001] This invention relates to the field of federated forgetting technology, and in particular to a multimodal generation model client-level federated forgetting method, apparatus, device and medium. Background Technology

[0002] Visual language models are increasingly becoming a crucial foundation for intelligent transportation systems. In such systems, distributed clients continuously perceive road scenes and achieve environmental understanding and intelligent decision-making through interaction with text commands. For example, manually driven vehicles, autonomous vehicles, and drones can utilize visual observation information and text commands for navigation guidance. These applications require models capable of jointly processing visual input and linguistic conditions; therefore, multimodal generative models have emerged as a potential solution for visual language generation tasks in traffic scenarios.

[0003] In federated learning, each client trains its local model using its private data and uploads the updated model to the server. The server aggregates the data through multiple rounds to obtain a global model; therefore, the contributions from each client's local dataset are integrated into the global model. Based on the right to be forgotten, clients may subsequently request the removal of the impact of all their local data on the model. This requirement has driven the development of client-level federated forgetting, which aims to satisfy the target client's forgetting request while preserving the utility of the other client models.

[0004] Existing federated forgetting methods mainly include retraining methods, impact estimation methods, and direct model modification methods. However, these methods are primarily geared towards discriminative models and unimodal scenarios, and are not applicable to multimodal generative models. During federated learning, multiple rounds of model aggregation couple the local updates of the target client with the local updates of other clients, making it difficult to accurately identify and locate the contribution of the target client in multimodal generative models and effectively eliminate the historical influence of the target client on the global model. Summary of the Invention

[0005] This invention provides a client-level federated forgetting method, apparatus, device, and medium for multimodal generative models. It addresses the shortcomings of existing federated forgetting methods, which are primarily designed for discriminative models and unimodal scenarios and are unsuitable for multimodal generative models. In federated learning, multiple rounds of model aggregation couple the local updates of the target client with those of other clients, making it difficult to accurately identify and locate the target client's contribution in multimodal generative models and effectively eliminate the historical impact of the target client on the global model. This invention utilizes the cumulative update records of the target client during the federated learning process to characterize the target client's contribution from three aspects: cross-modal shared representation, modality-specific representation, and cross-modal dependency relationships. Furthermore, it constructs an architecture-aware dependency graph based on the computational structure of the multimodal generative model to locate the representation regions related to the target client and their propagation paths within the model, accurately identifying and locating the target client's contribution and effectively eliminating the historical impact of the target client on the global model.

[0006] This invention provides a client-level federated forgetting method for multimodal generative models, comprising the following steps: Based on the cumulative update records of the target client in the initial multimodal generation model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generation model, the contribution representation of the target client in the initial multimodal generation model is determined. Based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation, the parameter region related to the target client in the architecture-aware dependency graph is determined; the architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generation model. The parameter region is subjected to forgetting processing to obtain the forgotten multimodal generative model.

[0007] According to a client-level federated forgetting method for multimodal generative models provided by the present invention, the contribution representation includes cross-modal shared contribution representation, modality-specific contribution representation, and cross-modal dependency contribution representation. The method for determining the contribution representation of the target client in the initial multimodal generative model based on the cumulative update records of the target client in the initial multimodal generative model stored by the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generative model, includes: respectively based on the cross-modal shared parameter set, the modality-specific parameter set, and the modality-specific dependency parameter set... Based on the state-specific parameter set and the cross-modal dependency parameter set, determine the cross-modal shared mask corresponding to the cross-modal shared parameter set, the modality-specific mask corresponding to the modality-specific parameter set, and the cross-modal dependency mask corresponding to the cross-modal dependency parameter set; based on the cumulative update record and the cross-modal shared mask, determine the cross-modal shared contribution representation; based on the cumulative update record and the modality-specific mask, determine the modality-specific contribution representation; based on the cumulative update record and the cross-modal dependency mask, determine the cross-modal dependency contribution representation.

[0008] According to a client-level federated forgetting method for a multimodal generative model provided by the present invention, the step of determining the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generative model and the contribution representation includes: determining the binary mask corresponding to each node and the binary mask corresponding to each edge in the architecture-aware dependency graph; and determining the parameter region based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation.

[0009] According to a client-level federated forgetting method for a multimodal generative model provided by the present invention, the step of determining the parameter region based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation includes: determining a contribution component based on the cross-modal shared contribution representation and the modality-specific contribution representation in the contribution representation; for each node, determining a first relevance score of the node based on the binary mask corresponding to the contribution component; for each edge, determining a second relevance score of the edge based on the cross-modal dependency contribution representation and the binary mask corresponding to the edge; and determining the parameter region based on the first relevance score of each node and the second relevance score of each edge.

[0010] According to a client-level federated forgetting method for a multimodal generative model provided by the present invention, the step of determining the parameter region based on the first relevance score of each node and the second relevance score of each edge includes: determining nodes whose first relevance score is greater than or equal to a first relevance threshold as target nodes; determining edges whose second relevance score is greater than or equal to a second relevance threshold as target edges; and determining the region where the target nodes and the target edges are located in the architecture-aware dependency graph as the parameter region.

[0011] According to a client-level federated forgetting method for a multimodal generation model provided by the present invention, the method further includes: determining a data flow based on the forward computation structure of the initial multimodal generation model; and constructing the architecture-aware dependency graph based on the data flow and the cross-modal shared representation function module and the modality-specific representation function module in the initial multimodal generation model.

[0012] According to a client-level federated forgetting method for a multimodal generative model provided by the present invention, the step of performing forgetting processing on the parameter region to obtain the forgotten multimodal generative model includes: converting the parameter region into a parameter region binary mask; determining a forgetting objective function based on the parameter region binary mask, model parameters, and the cumulative update record; and performing forgetting processing on the parameter region by maximizing the forgetting objective function to obtain the forgotten multimodal generative model.

[0013] According to a client-level federated forgetting method for a multimodal generative model provided by the present invention, the forgetting process includes gradient ascent update and knowledge distillation recovery. The step of performing forgetting processing on the parameter region by maximizing the forgetting objective function to obtain the forgotten multimodal generative model includes: performing gradient ascent update on the parameter region by maximizing the forgetting objective function to obtain an intermediate multimodal generative model; inputting general generation conditions that do not contain feature information of the target client into the initial multimodal generative model to obtain a synthesis anchor point output by the initial multimodal generative model; and performing knowledge distillation recovery on the intermediate multimodal generative model based on the synthesis anchor point, the initial multimodal generative model, and the intermediate multimodal generative model to obtain the forgotten multimodal generative model.

[0014] According to a client-level federated forgetting method for a multimodal generative model provided by the present invention, the step of performing knowledge distillation recovery on the intermediate multimodal generative model based on the synthesis anchor, the initial multimodal generative model, and the intermediate multimodal generative model to obtain the forgotten multimodal generative model includes: inputting the synthesis anchor into the initial multimodal generative model to obtain a first soft probability distribution output by the multimodal generative model; inputting the synthesis anchor into the intermediate multimodal generative model to obtain a second soft probability distribution output by the intermediate multimodal generative model; determining the knowledge distillation recovery loss based on the first soft probability distribution and the second soft probability distribution; and performing knowledge distillation recovery on the intermediate multimodal generative model with the objective of minimizing the knowledge distillation recovery loss to obtain the forgotten multimodal generative model.

[0015] The present invention also provides a client-level federated forgetting device for a multimodal generative model, comprising the following modules: The first determining module is used to determine the contribution representation of the target client in the initial multimodal generation model based on the cumulative update records of the target client, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set of the target client in the initial multimodal generation model stored by the server during the federated learning training phase. The second determining module is used to determine the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation; the architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generation model; The forgetting module is used to perform forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the client-level federated forgetting method for multimodal generative models as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the client-level federated forgetting method for multimodal generative models as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the client-level federated forgetting method for multimodal generative models as described above.

[0019] This invention provides a client-level federated forgetting method, apparatus, device, and medium for multimodal generative models. It determines the target client's contribution to the initial multimodal generative model by using the cumulative update records of the target client in the initial multimodal generative model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generative model. Based on the architecture-aware dependency graph corresponding to the initial multimodal generative model and the contribution representation, it determines the parameter region related to the target client in the architecture-aware dependency graph. The architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generative model. The parameter region is then subjected to forgetting processing to obtain the forgotten multimodal generative model. In this way, by utilizing the cumulative update records of the target client in the federated learning process, the contribution of the target client is characterized from three aspects: cross-modal shared representation, modality-specific representation, and cross-modal dependency. In addition, an architecture-aware dependency graph is constructed by combining the computational structure of the multimodal generative model to locate the representation regions related to the target client and their propagation paths in the model, accurately identify and locate the contribution of the target client, and effectively eliminate the historical influence of the target client on the global model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the client-level federated forgetting method for multimodal generation models provided by this invention.

[0022] Figure 2 This is the second flowchart of the client-level federated forgetting method for multimodal generation models provided by this invention.

[0023] Figure 3 This is a schematic diagram of the structure of the client-level federated forgetting system of the multimodal generation model provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the client-level federated forgetting device for the multimodal generation model provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] Data is naturally distributed across different clients and may contain sensitive visual and textual information. Collecting this data directly to a central server can raise privacy concerns. To address this, federated learning provides a suitable training paradigm, enabling multiple clients to collaboratively optimize a shared multimodal generative model while keeping the original data locally. In federated learning, each client trains its local model using its private data and uploads the updated model to the server. The server aggregates the data through multiple rounds to obtain a global model; therefore, the contributions from each client's local dataset are integrated into the global model. Based on the right to be forgotten, clients may subsequently request the removal of the influence of all their local data on the model. This requirement has driven the development of client-level federated forgetting, which aims to satisfy the forgetting request of the target client while preserving the utility of the models from other clients.

[0028] Currently, retraining reconstructs the global model using data from retained clients, thus eliminating the contribution of target clients; influence estimation methods approximate the impact of target clients on model parameters and offset this impact through correction updates; direct model modification methods optimize the trained model using specially designed optimization objectives, suppressing target information while maintaining model utility on non-target data. In recent years, federated forgetting has been further extended to generative models. For example, the Full Trajectory Guidance Unit (F2GU) eliminates identity information in face generation models by redirecting the generated trajectory of the target identity; the Full Trajectory Unified Latent Model (FULM) separates the knowledge forgetting and knowledge retention processes in large language models through dedicated adapters and hierarchical merging mechanisms.

[0029] While the aforementioned methods have proven effective in their respective original scenarios, the applicability of existing federated forgetting methods in multimodal generative models remains limited. This limitation is mainly reflected in two aspects. First, methods designed for discriminative models do not explicitly consider the impact of target client contributions on the learned generative distribution. Second, existing federated forgetting methods for generative models are primarily geared towards unimodal scenarios, thus ignoring residual target client information in cross-modal shared representations, modality-specific representations, and cross-modal dependencies. Consequently, in multimodal generative models, it is difficult to accurately identify and locate the contribution of target clients, and they cannot effectively eliminate the historical influence of target clients on the global model.

[0030] Furthermore, the contributions of the target client may be encoded in cross-modal shared representations and modality-specific representations, and propagate within the model through cross-modal dependencies. Updating only a small number of parameters may result in residual target client information, while updating too much may destroy the multimodal knowledge retained by the model, thereby reducing the generative utility of non-target clients and cross-modal consistency.

[0031] To address the aforementioned problems, this invention proposes a client-level federated forgetting method for multimodal generative models. This method utilizes the cumulative update records of the target client during the federated learning process to characterize the target client's contribution from three aspects: cross-modal shared representation, modality-specific representation, and cross-modal dependency. Furthermore, it constructs an architecture-aware dependency graph based on the computational structure of the multimodal generative model to locate the representation regions related to the target client and their propagation paths within the model. This accurately identifies and locates the target client's contribution, effectively eliminating the historical impact of the target client on the global model.

[0032] The following is combined with Figures 1 to 2 This invention describes a client-level federated unlearning framework guided by causal analysis for multimodal generative models (CaMU). This method can be applied to intelligent transportation, visual question answering, multimodal medical analysis, intelligent recommendation, and other scenarios involving collaborative training of multimodal data and client-side data deletion. The execution entity of this method can be an electronic device or a federated learning server, or a client-level federated unlearning device based on a multimodal generative model installed in the electronic device. This client-level federated unlearning device can be implemented through software, hardware, or a combination of both, as provided by this invention.

[0033] Figure 1 This is one of the flowcharts illustrating the client-level federated forgetting method for multimodal generation models provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Based on the cumulative update records of the target client in the initial multimodal generation model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generation model, determine the contribution representation of the target client in the initial multimodal generation model.

[0034] Here, the target client is the client that sends the forget request to the server.

[0035] Here, the initial multimodal generative model is obtained through federated learning, that is, the client trains the model locally using data and uploads the model parameters to the server, and the server aggregates the model parameters uploaded by all clients.

[0036] Here, the cumulative update record is used to characterize the target client's participation in federated learning training rounds, model updates, and aggregate weights.

[0037] Here, the cross-modal shared parameter set, modality-specific parameter set, and modality-specific parameter set are parameter index sets pre-divided according to the network structure of the multimodal generative model. In the multimodal generative model, different functional modules correspond to different parameter regions. For example, the cross-modal shared representation module, the modality-specific representation module, and the cross-modal interaction / dependency module correspond to different model parameters. Therefore, after obtaining the structure of the trained multimodal generative model, the server can directly determine the above three parameter sets based on the model architecture information.

[0038] Specifically, based on the structural division of each functional module in the model, the server categorizes the parameter indices contained in the corresponding modules into the cross-modal shared parameter set. Modal-specific parameter set and modal-specific parameter set .

[0039] Here, contribution representations can include cross-modal shared contribution representations, modality-specific contribution representations, and cross-modal dependency contribution representations.

[0040] Furthermore, determining the contribution representation of the target client in the initial multimodal generation model based on the cumulative update record of the target client in the initial multimodal generation model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generation model, includes: determining the cross-modal shared mask corresponding to the cross-modal shared parameter set, the modality-specific mask corresponding to the modality-specific parameter set, and the cross-modal dependency mask corresponding to the cross-modal dependency parameter set, respectively, based on the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set; determining the cross-modal shared contribution representation based on the cumulative update record and the cross-modal shared mask; determining the modality-specific contribution representation based on the cumulative update record and the modality-specific mask; and determining the cross-modal dependency contribution representation based on the cumulative update record and the cross-modal dependency mask.

[0041] Here, cross-modal shared masks, modality-specific masks, and cross-modal dependency masks are all binary masks. The mask corresponding to the parameter set can be a parameter selection marker directly generated from the parameter set, without the need for additional training or data acquisition. The server first determines the parameter index set corresponding to the three functional modules based on the multimodal generative model structure. For any parameter set, the server traverses the index positions of all trainable parameters in the entire model, marking the positions of parameters belonging to the set as 1 and the positions of parameters not belonging to the set as 0, thereby forming the binary mask corresponding to each parameter aggregation.

[0042] For example, for the model parameter vector If the parameter set corresponding to a certain functional module is The generated binary mask is ,therefore, Essentially, it's used to select index mapping relationships for specific functional areas of the model, which are then used to accumulate update records for the target client. Project onto the corresponding parameter area.

[0043] Here, the cross-modal shared contribution representation, modality-specific contribution representation, and cross-modal dependency contribution representation can be calculated by multiplying their respective binary masks element-wise with the cumulative update records. The calculation of each contribution representation is as follows: (1) (1) in, Indicates contribution representation, Indicates cross-modal shared contribution representation, Represents modal-specific contribution characterization, The cross-modal dependency contribution representations represent the impact of the target client in the cross-modal shared representation, modality-specific representation, and the functional region corresponding to the cross-modal dependency, respectively. Represents the parameter set The corresponding binary mask, This indicates the cumulative update records. This indicates element-wise multiplication.

[0044] further, The calculation is as follows: (2) (2) in, This represents the set of training rounds in which the target client participates in federated learning. Indicates the corresponding aggregate weight. This indicates the model update uploaded by the target client in round t.

[0045] In this embodiment of the invention, the cumulative update records of the target client maintained by the server during the federated learning process are used to characterize the contribution of the target client from three aspects: cross-modal shared representation, modality-specific representation, and cross-modal dependency relationship, thereby realizing the structured decomposition of the target contribution.

[0046] Step 102: Based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation, determine the parameter region in the architecture-aware dependency graph that is related to the target client.

[0047] The architecture-aware dependency graph is used to characterize the forward computation structure of the initial multimodal generative model.

[0048] Here, the architecture-aware dependency graph can be constructed based on the forward computation structure of the initial multimodal generative model.

[0049] Furthermore, the method also includes: determining the data flow based on the forward computation structure of the initial multimodal generation model; and constructing the architecture-aware dependency graph based on the data flow and the cross-modal shared representation function module and modality-specific representation function module in the initial multimodal generation model.

[0050] Here, the nodes in the architecture-aware dependency graph can correspond to cross-modal shared representation functional modules or modality-specific representation functional modules, and the edges in the architecture-aware dependency graph can represent data flow, which is used to describe the information propagation relationship between different functional modules within the multimodal generative model.

[0051] For example, architecture-aware dependency graph It can be expressed as the following formula (3): (3) in, Represents a set of nodes, nodes This corresponds to a cross-modal shared representation function module or a modality-specific representation function module. The edge set is also called the cross-modal dependency edge set, where if the module The output is used as a module For inputs used in cross-modal computation, a directed edge is created in the dependency graph from module to module. .

[0052] In this embodiment of the invention, an architecture-aware dependency graph is constructed by combining the computational structure of the multimodal generative model to locate representation nodes, cross-modal dependency paths, and parameter regions related to the target client, thereby improving the accuracy of locating representation regions related to the target client and their propagation paths in the model.

[0053] Furthermore, determining the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation includes: determining the binary mask corresponding to each node and the binary mask corresponding to each edge in the architecture-aware dependency graph; and determining the parameter region based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation.

[0054] It should be noted that nodes correspond to cross-modal shared representation function modules or modality-specific representation function modules. Therefore, the binary mask of a node is the binary mask corresponding to the cross-modal shared parameter set or the modality-specific parameter set. Edges correspond to cross-modal dependency relationship modules. Therefore, the corresponding binary mask is the binary mask corresponding to the cross-modal dependency relationship parameter set.

[0055] Here, the parameter area includes representation nodes and dependency paths related to the target client; in other words, the parameter area is the area where the target client contributes the most.

[0056] Optionally, the parameter region can be determined directly based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation; or the target node can be determined first based on the binary mask corresponding to each node and the contribution representation, then the target edge can be determined based on the binary mask corresponding to each edge and the contribution representation, and finally the parameter region can be determined based on the target node and the target edge.

[0057] In this embodiment of the invention, based on the cumulative update records of the target client, its contribution in cross-modal shared representation, modality-specific representation, and cross-modal dependency is characterized. In conjunction with the computational structure of the multimodal generative model, an architecture-aware dependency graph is constructed to locate representation nodes, cross-modal dependency paths, and parameter regions related to the target client, accurately identify and locate the contribution of the target client, and effectively eliminate the historical impact of the target client on the global model.

[0058] Further, determining the parameter region based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation includes: determining a contribution component based on the cross-modal shared contribution representation and the modality-specific contribution representation in the contribution representation; for each node, determining a first relevance score of the node based on the binary mask representing the contribution component and the node; for each edge, determining a second relevance score of the edge based on the cross-modal dependency contribution representation in the contribution representation and the binary mask corresponding to the edge; and determining the parameter region based on the first relevance score of each node and the second relevance score of each edge.

[0059] Here, the contribution component is used to calculate node correlation, and can be a combination of cross-modal shared contribution representation and modality-specific contribution representation in the contribution representation, or a weighted combination of the two.

[0060] For example, representing contribution components The calculation is as follows: (4) (4) in, This represents the cross-modal shared contribution characterization. This represents the modality-specific contribution.

[0061] For example, The value is (1,2,3). If it is (4,5), then The answer is (1,2,3,4,5).

[0062] It should be noted that each node has its own corresponding binary mask, and the correlation score of each node is calculated based on the binary mask of each node and the contribution component.

[0063] Here, the first correlation score can be calculated based on the product or element-wise product of the binary mask representing the contribution component and the node.

[0064] For example, the first relevance score is calculated using the following formula (5): (5) in, Represents a node The first correlation score, Represents a node binary mask, It represents the 2-norm.

[0065] Here, the second relevance score can be calculated as the product or element-wise product of the cross-modal dependency contribution representation and the binary mask of the edge.

[0066] For example, the second relevance score is calculated using the following formula (6): (6) in, Representing an edge The second correlation score, Representing an edge The binary mask.

[0067] Here, the target node can be determined based on the first correlation score, the target edge can be determined based on the second correlation score, and the parameter region can be determined based on the target node and the target edge.

[0068] In this embodiment of the invention, the specific influence area and propagation path of the target client contribution in the multimodal generation model are located by the correlation scores of nodes and edges, providing a parameter update range for subsequent selective forgetting.

[0069] Furthermore, determining the parameter region based on the first relevance score of each node and the second relevance score of each edge includes: determining nodes with the first relevance score greater than or equal to the first relevance threshold as target nodes; determining edges with the second relevance score greater than or equal to the second relevance threshold as target edges; and determining the region where the target nodes and target edges are located in the architecture-aware dependency graph as the parameter region.

[0070] It should be noted that the higher the relevance score, the higher the relevance to the target client, which means the target client contributes more to that node; conversely, the lower the relevance score, the lower the relevance to the target client, which means the target client contributes less to that node.

[0071] Here, the first correlation threshold and the second correlation threshold can be preset based on experience.

[0072] If the first relevance score is greater than or equal to the first relevance threshold, it indicates that the target client has made a significant contribution to that node. If the second relevance score is greater than or equal to the second relevance threshold, it indicates that the target client has made a significant contribution to that edge. The nodes and edges with significant contributions are then used as the parameter regions.

[0073] For example, the parameter region is defined as follows: (7) (7) in, Represents the target node set. The threshold for filtering nodes is the first relevance threshold. Describes the target edge set. This represents the threshold for filtering cross-modal dependency edges, also known as the second relevance threshold.

[0074] It should be noted that the node set and cross-modal dependency edge set This is used to locate the specific area of ​​influence and propagation path of the target client's contribution in the multimodal generative model, providing a parameter update range for subsequent selective forgetting.

[0075] Specifically, node set This represents functional modules highly correlated with the target client's contribution, including a cross-modal shared representation module and a modality-specific representation module, used to determine the model representation region where the target client's influence resides; and a cross-modal dependency edge set. This indicates the propagation relationship of the target client's contribution across different modal functional modules, used to characterize the scope of the target's influence generated through cross-modal interaction paths.

[0076] In this embodiment of the invention, the specific influence area and propagation path of the target client contribution in the multimodal generation model are located by the correlation scores of nodes and edges, providing a parameter update range for subsequent selective forgetting.

[0077] Step 103: Perform forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

[0078] Here, the forgotten multimodal generative model is the model with the model parameters of the target client removed.

[0079] Here, we can either perform forgetting on the initial multimodal generative model to obtain the forgotten multimodal generative model, or we can first perform forgetting on the initial multimodal generative model, and then perform knowledge distillation to recover the forgotten model to obtain the forgotten multimodal generative model.

[0080] Furthermore, the step of performing forgetting processing on the parameter region to obtain the forgotten multimodal generative model includes: converting the parameter region into a parameter region binary mask; determining a forgetting objective function based on the parameter region binary mask, model parameters, and the cumulative update record; and performing forgetting processing on the parameter region by maximizing the forgetting objective function to obtain the forgotten multimodal generative model.

[0081] Here, the binary mask of the parameter region is used to indicate whether an element is located within the parameter region. If the element is located within the parameter region, it is 1; if the element is not located within the parameter region, it is 0.

[0082] For example, the binary mask of the parameter region is defined as follows: (8) (8) in, Indicates the first The mask value of the element. This represents the parameter range corresponding to the selected nodes and dependency paths.

[0083] The model loss can be calculated based on the objective function.

[0084] Here, the model loss can be the difference between the constant and the first ratio, or the weighted difference.

[0085] For example, the target loss function is as follows (9): (9) in, Indicates the model loss. Indicates the inner product. Indicates the first Model parameters during training rounds, This represents a smaller constant used to ensure numerical stability.

[0086] Here, the target loss function is maximized. This can make the alignment degree between the model parameters and the cumulative updates of the target client approach zero, thereby weakening the components in the model parameters that are in the same direction as the cumulative updates of the target client without forcing the relevant parameter components to align in the opposite direction.

[0087] In obtaining and Subsequently, the present invention further constructs a target parameter mask based on the corresponding model parameter regions. Selective gradient ascent forgetting is performed only on parameters relevant to the target client, thereby avoiding updates to irrelevant parameters and reducing the impact on non-target client knowledge and cross-modal consistency.

[0088] In this embodiment of the invention, selective gradient ascent is performed in the parameter subspace related to the target client to reduce the degree of square cosine alignment between the model parameters and the cumulative update records of the target client, thereby weakening the influence of the target client in cross-modal shared representations, modality-specific representations and cross-modal dependencies, and limiting perturbations to non-target knowledge.

[0089] Furthermore, the forgetting process includes gradient ascent update and knowledge distillation recovery. The step of performing forgetting processing on the parameter region by maximizing the forgetting objective function to obtain the forgotten multimodal generative model includes: performing gradient ascent update on the parameter region by maximizing the forgetting objective function to obtain an intermediate multimodal generative model; inputting general generation conditions that do not contain feature information of the target client into the initial multimodal generative model to obtain the synthesis anchor point output by the initial multimodal generative model; and performing knowledge distillation recovery on the intermediate multimodal generative model based on the synthesis anchor point, the initial multimodal generative model, and the intermediate multimodal generative model to obtain the forgotten multimodal generative model.

[0090] Here, gradient ascent update refers to increasing the model loss by increasing the gradient of the model parameters.

[0091] Here, gradient ascent updates are used to make the model forget the contributions of the target client.

[0092] Here, the gradient ascent update is calculated as follows: (10) (10) in, Indicates the first Model parameters for each round, Indicates the first Model parameters for each round, Indicates the forgotten step size. This represents the gradient of the model loss. This ensures that only model parameters related to the selected representation nodes and dependency paths are updated, while the remaining model parameters remain unchanged.

[0093] After updating the contribution of the forgotten target client through gradient ascent, an intermediate multimodal generative model is obtained.

[0094] Here, gradient ascent updates reduce the alignment between the model parameters and the cumulative updates of the target client, i.e., they reduce the similarity in their update directions, rather than simply reducing the parameter values. The cumulative update record of the target client represents the cumulative change direction of the model parameters by the target client during federated training.

[0095] It should be noted that maximizing the objective function using gradient ascent gradually reduces the square cosine similarity between the model parameters and the cumulative updates of the target client. As gradient ascent optimizes the objective function, the model parameters will follow this principle... Adjust the direction of increase, that is, along the decrease. The direction of the change weakens the component in the model parameters that is consistent with the historical update direction of the target client, thereby reducing the influence of the target client.

[0096] At the same time, through Gradient updates are restricted to the target client-related parameter region, adjusting only the target-related module parameters to avoid causing significant disturbances to non-target knowledge regions.

[0097] For example, the current model parameters are The cumulative updates for the target client are Then the two are in the same direction, that is This indicates that there is a clear target client update direction in the model parameters. After gradient ascent... Although the parameter value did not simply return to zero, the direction changed. This indicates that the model parameters are no longer along the direction of contribution from the target client.

[0098] Here, the synthetic anchor points are training data that are independent of the target client.

[0099] It should be noted that while selective gradient descent can mitigate the residual influence of the target client, this process may also perturb non-target representations due to the potential coupling between target and non-target client information in cross-modal shared representations and cross-modal dependencies. This could reduce the model's generation quality and cross-modal consistency. Therefore, knowledge distillation recovery is employed to restore the model's non-target generation utility while maintaining the effectiveness of forgetting.

[0100] Here, the general generation conditions do not include target client-specific identification information or semantic information related to the target client.

[0101] Here, the initial multimodal generative model is used as the teacher model, and the intermediate multimodal generative model is used as the student model. The parameters of the teacher model are kept fixed, and only the parameters of the student model are updated.

[0102] In this embodiment of the invention, selective gradient ascent is performed within the target-related parameter region obtained by localization to weaken the influence of the target client and limit the perturbation of non-target knowledge. At the same time, knowledge distillation is performed using synthetic anchors that are unrelated to the target client to restore the generative performance of the model and maintain cross-modal consistency.

[0103] Further, the step of performing knowledge distillation recovery on the intermediate multimodal generation model based on the synthetic anchor point, the initial multimodal generation model, and the intermediate multimodal generation model to obtain the forgotten multimodal generation model includes: inputting the synthetic anchor point into the initial multimodal generation model to obtain a first soft probability distribution output by the multimodal generation model; inputting the synthetic anchor point into the intermediate multimodal generation model to obtain a second soft probability distribution output by the intermediate multimodal generation model; determining the knowledge distillation recovery loss based on the first soft probability distribution and the second soft probability distribution; and performing knowledge distillation recovery on the intermediate multimodal generation model with the goal of minimizing the knowledge distillation recovery loss to obtain the forgotten multimodal generation model.

[0104] Here, the soft probability distribution is a probability vector in which multiple categories simultaneously possess non-zero probabilities after being processed by high-temperature softmax.

[0105] The first soft probability distribution represents the identification results of the synthetic anchor points by the initial multimodal generation model, and the second soft probability distribution represents the identification results of the synthetic anchor points by the intermediate multimodal generation model.

[0106] Here, the knowledge distillation recovery loss can be calculated based on the KL divergence.

[0107] For example, the knowledge distillation recovery loss function is as follows: (11) (11) in, This indicates the knowledge distillation recovery loss. This represents an intermediate multimodal generative model. This represents the initial multimodal generative model. This represents the training data in the synthetic anchor. This represents the model parameters of the intermediate multimodal generative model. This represents the model parameters of the initial multimodal generative model. Indicates temperature.

[0108] During the recovery process, the parameters of the initial multimodal generative model (teacher model) remain fixed, and only the parameters of the intermediate multimodal generative model (student model) are updated. In the next distillation iteration, by minimizing the recovery loss The student model parameters are minimized according to the following rules as shown in formula (12): (12) in, Indicates the first Model parameters of the intermediate multimodal generative model in round 1. Indicates the first Model parameters of the intermediate multimodal generative model in each round. Indicates the distillation step size. This represents the loss gradient recovered by knowledge distillation.

[0109] In this embodiment of the invention, knowledge distillation is employed to restore the non-target generation utility of the model while maintaining the effectiveness of forgetting. By minimizing the recovery loss on the synthetic anchor set, the student model is prompted to recover the non-target generation behavior represented by the teacher model, thereby improving the generation quality and cross-modal consistency of the model.

[0110] Figure 2 This is the second flowchart illustrating the client-level federated forgetting method for the multimodal generation model provided by this invention. Figure 2 As shown, it includes: Step 201: Based on the cumulative update records of the target client in the initial multimodal generation model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generation model, determine the contribution representation of the target client in the initial multimodal generation model.

[0111] Step 202: Based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation, determine the parameter region in the architecture-aware dependency graph that is related to the target client.

[0112] Step 203: Convert the parameter region into a parameter region binary mask.

[0113] Step 204: Determine the forgetting objective function based on the binary mask of the parameter region, the model parameters, and the cumulative update record.

[0114] Step 205: By maximizing the forgetting objective function, the parameter region is updated by gradient ascent to obtain the intermediate multimodal generation model.

[0115] Step 206: Input the general generation conditions that do not contain the feature information of the target client into the initial multimodal generation model to obtain the synthetic anchor points output by the initial multimodal generation model.

[0116] Step 207: Based on the synthetic anchor point, the initial multimodal generation model, and the intermediate multimodal generation model, perform the knowledge distillation recovery on the intermediate multimodal generation model to obtain the forgotten multimodal generation model.

[0117] In this embodiment of the invention, the cumulative update records of the target client are first used to characterize its contribution to cross-modal shared representations, modality-specific representations, and cross-modal dependencies. An architecture-aware dependency graph is then constructed using the computational structure of the multimodal generative model to locate representation nodes, cross-modal dependency paths, and parameter regions related to the target client. Subsequently, selective gradient ascent is performed within the located parameter regions to weaken the influence of the target client. Knowledge distillation is then performed using synthetic anchors unrelated to the target client to restore the model's generative performance and cross-modal consistency. This method effectively eliminates the historical contributions of the target client without accessing the client's original data or increasing communication overhead during the forgetting phase, while maintaining the utility of non-target client models and cross-modal consistency, and significantly reducing the time cost compared to retraining.

[0118] Figure 3 This is a schematic diagram of the structure of the client-level federated forgetting system for the multimodal generation model provided by the present invention, as shown below. Figure 3 As shown, it includes: a federated learning module and a multimodal generative model client-side cascaded forgetting module. The federated learning module includes multiple clients, a target client, and a server. The multimodal generative model client-side cascaded forgetting module includes a contribution representation determination unit, a parameter region determination unit, a forgetting processing unit, and a knowledge distillation unit. Specifically, in the federated learning module, multiple clients update their local models through federated learning and upload their local models to the server. The server aggregates the model parameters uploaded by each client to obtain a global model (i.e., the initial multimodal generative model). When the target client sends a forgetting request to the server, the server uses the multimodal generative model client-side cascaded forgetting module to perform client-level federated forgetting. Specifically, the client-level federated forgetting process includes the contribution representation determination unit mapping the target client's cumulative update records to a cross-modal shared parameter set, a modality-specific parameter set, and a cross-modal dependency parameter set, respectively, and obtaining the target client's contribution based on these parameters. The parameter region determination unit determines the target client's contribution based on the architecture-aware dependency graph corresponding to the initial multimodal generative model (i.e., the parameter region determination unit). Figure 3 In the dependency graph), determine the parameter region containing the target node and the target path (i.e. Figure 3 The forgetting processing unit selects target parameters based on the parameter region, determines the target parameter region in the initial multimodal generation model based on the target parameters, constructs a binary mask, and then performs selective updates based on the binary mask to obtain the intermediate multimodal generation model. The knowledge distillation unit uses the teacher model (i.e., the initial multimodal generation model) to generate synthetic anchors that are independent of the target client, and uses the student model (i.e., the intermediate multimodal generation model) corresponding to the synthetic anchors to perform knowledge distillation to obtain the forgotten multimodal generation model.

[0119] The following describes the client-level federated forgetting device for the multimodal generation model provided by the present invention. The client-level federated forgetting device for the multimodal generation model described below can be referred to in correspondence with the client-level federated forgetting method for the multimodal generation model described above.

[0120] Figure 4 This is a schematic diagram of the structure of the client-level federated forgetting device for the multimodal generation model provided by the present invention, as shown below. Figure 4 As shown, the multimodal generation model client-level federated forgetting device 300 includes the following: The first determining module 310 is used to determine the contribution representation of the target client in the initial multimodal generation model based on the cumulative update record of the target client, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set of the target client in the initial multimodal generation model stored by the server during the federated learning training phase. The second determining module 320 is used to determine the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation; the architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generation model. The forgetting module 330 is used to perform forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

[0121] In this embodiment of the invention, the contribution representation includes a cross-modal shared contribution representation, a modality-specific contribution representation, and a cross-modal dependency contribution representation. The first determining module 310 is specifically used to: determine, based on the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set, respectively, a cross-modal shared mask corresponding to the cross-modal shared parameter set, a modality-specific mask corresponding to the modality-specific parameter set, and a cross-modal dependency parameter set; determine the cross-modal shared contribution representation based on the cumulative update record and the cross-modal shared mask; determine the modality-specific contribution representation based on the cumulative update record and the modality-specific mask; and determine the cross-modal dependency contribution representation based on the cumulative update record and the cross-modal dependency mask.

[0122] In this embodiment of the invention, the second determining module 320 is specifically used to: determine the binary mask corresponding to each node and the binary mask corresponding to each edge in the architecture-aware dependency graph; and determine the parameter region based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation.

[0123] In this embodiment of the invention, the second determining module 320 is further specifically configured to: determine a contribution component based on the cross-modal shared contribution representation and the modality-specific contribution representation in the contribution representation; for each node, determine a first correlation score of the node based on the binary mask corresponding to the contribution component and the node; for each edge, determine a second correlation score of the edge based on the cross-modal dependency contribution representation and the binary mask corresponding to the edge; and determine the parameter region based on the first correlation score of each node and the second correlation score of each edge.

[0124] In this embodiment of the invention, the second determining module 320 is further specifically used to: determine the node whose first correlation score is greater than or equal to the first correlation threshold as the target node; determine the edge whose second correlation score is greater than or equal to the second correlation threshold as the target edge; and determine the region where the target node and the target edge are located in the architecture-aware dependency graph as the parameter region.

[0125] In this embodiment of the invention, the multimodal generation model client-level federated forgetting device 300 further includes a construction module, specifically used for: determining a data flow based on the forward computation structure of the initial multimodal generation model; and constructing the architecture-aware dependency graph based on the data flow and the cross-modal shared representation function module and the modality-specific representation function module in the initial multimodal generation model.

[0126] In this embodiment of the invention, the forgetting module 330 is specifically used to: convert the parameter region into a parameter region binary mask; determine a forgetting objective function based on the parameter region binary mask, model parameters, and the cumulative update record; and perform forgetting processing on the parameter region by maximizing the forgetting objective function to obtain the forgotten multimodal generative model.

[0127] In this embodiment of the invention, the forgetting module 330 is specifically used for: updating the parameter region by gradient ascent by maximizing the forgetting objective function to obtain an intermediate multimodal generation model; inputting general generation conditions that do not contain feature information of the target client into the initial multimodal generation model to obtain the synthesis anchor point output by the initial multimodal generation model; and performing knowledge distillation recovery on the intermediate multimodal generation model based on the synthesis anchor point, the initial multimodal generation model, and the intermediate multimodal generation model to obtain the forgotten multimodal generation model.

[0128] In this embodiment of the invention, the forgetting module 330 is specifically configured to: input the synthetic anchor point into the initial multimodal generation model to obtain a first soft probability distribution output by the multimodal generation model; input the synthetic anchor point into the intermediate multimodal generation model to obtain a second soft probability distribution output by the intermediate multimodal generation model; determine the knowledge distillation recovery loss based on the first soft probability distribution and the second soft probability distribution; and perform knowledge distillation recovery on the intermediate multimodal generation model with the goal of minimizing the knowledge distillation recovery loss to obtain the forgotten multimodal generation model.

[0129] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a client-level federated forgetting method for multimodal generative models. This method includes: determining the contribution representation of the target client in the initial multimodal generative model based on the cumulative update record of the target client in the initial multimodal generative model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generative model; determining the parameter region related to the target client in the architecture-aware dependency graph corresponding to the initial multimodal generative model and the contribution representation; the architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generative model; and performing forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

[0130] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the client-level federated forgetting method for multimodal generative models provided by the above methods. This method includes: determining the contribution representation of the target client in the initial multimodal generative model based on the cumulative update record of the target client in the initial multimodal generative model stored by the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generative model; determining the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generative model and the contribution representation; the architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generative model; and performing forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

[0132] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the client-level federated forgetting method for multimodal generative models provided by the methods described above. The method includes: determining the contribution representation of the target client in the initial multimodal generative model based on the cumulative update record of the target client in the initial multimodal generative model stored by the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generative model; determining the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generative model and the contribution representation; the architecture-aware dependency graph is used to represent the forward computation structure of the initial multimodal generative model; and performing forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal generative model client cascaded federated forgetting method, characterized in that, include: Based on the cumulative update records of the target client in the initial multimodal generation model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generation model, the contribution representation of the target client in the initial multimodal generation model is determined. Based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation, determine the parameter region in the architecture-aware dependency graph that is related to the target client; The architecture-aware dependency graph is used to characterize the forward computation structure of the initial multimodal generation model; The parameter region is subjected to forgetting processing to obtain the forgotten multimodal generative model.

2. The multi-modal generative model client federated forgetting method according to claim 1, wherein, The contribution representations include cross-modal shared contribution representations, modality-specific contribution representations, and cross-modal dependency contribution representations. The contribution representation of the target client in the initial multimodal generation model is determined based on the cumulative update records of the target client in the initial multimodal generation model stored on the server during the federated learning training phase, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set in the initial multimodal generation model, including: Based on the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set, respectively, determine the cross-modal shared mask corresponding to the cross-modal shared parameter set, the modality-specific mask corresponding to the modality-specific parameter set, and the cross-modal dependency parameter set; Based on the cumulative update record and the cross-modal sharing mask, the cross-modal sharing contribution characterization is determined; Based on the cumulative update record and the modality-specific mask, the modality-specific contribution characterization is determined; Based on the cumulative update records and the cross-modal dependency mask, the cross-modal dependency contribution representation is determined.

3. The multi-modal generative model client federated forgetting method of claim 1, wherein, The step of determining the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation includes: Determine the binary mask corresponding to each node and the binary mask corresponding to each edge in the architecture-aware dependency graph; The parameter region is determined based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation.

4. The multi-modal generative model client federated forgetting method of claim 3, wherein, The determination of the parameter region based on the binary mask corresponding to each node, the binary mask corresponding to each edge, and the contribution representation includes: Based on the cross-modal shared contribution representation and the modality-specific contribution representation in the contribution representation, the contribution components are determined. For each node, a first relevance score is determined based on the binary mask representing the contribution component and the node. For each edge, a second relevance score is determined based on the cross-modal dependency contribution representation in the contribution representation and the binary mask corresponding to the edge; The parameter region is determined based on the first relevance score of each node and the second relevance score of each edge.

5. The multi-modal generative model client federated forgetting method of claim 4, wherein, The determination of the parameter region based on the first relevance score of each node and the second relevance score of each edge includes: Nodes whose first correlation score is greater than or equal to the first correlation threshold are identified as target nodes; Edges whose second relevance score is greater than or equal to the second relevance threshold are identified as target edges; The region where the target node and the target edge are located in the architecture-aware dependency graph is determined as the parameter region.

6. The multi-modal generative model client federated forgetting method according to any one of claims 1 to 5, characterized in that, The method further includes: Based on the forward computation structure of the initial multimodal generation model, the data flow is determined; Based on the data stream and the cross-modal shared representation function module and modality-specific representation function module in the initial multimodal generation model, the architecture-aware dependency graph is constructed.

7. The multi-modal generative model client federated forgetting method according to any one of claims 1 to 5, characterized in that, The step of performing forgetting processing on the parameter region to obtain the forgotten multimodal generative model includes: Convert the parameter region into a parameter region binary mask; Based on the binary mask of the parameter region, the model parameters, and the cumulative update records, the forgetting objective function is determined; By maximizing the forgetting objective function, the parameter region is subjected to forgetting processing to obtain the forgotten multimodal generative model.

8. The multi-modal generative model client federated forgetting method of claim 7, wherein, The forgetting process includes gradient ascent update and knowledge distillation recovery. The forgetting process, which involves maximizing the forgetting objective function to perform forgetting on the parameter region to obtain the forgotten multimodal generative model, includes: By maximizing the forgetting objective function, the parameter region is updated using the gradient ascent method to obtain an intermediate multimodal generation model. The general generation conditions that do not contain the feature information of the target client are input into the initial multimodal generation model to obtain the synthetic anchor points output by the initial multimodal generation model; Based on the synthetic anchor point, the initial multimodal generation model, and the intermediate multimodal generation model, the knowledge distillation recovery is performed on the intermediate multimodal generation model to obtain the forgotten multimodal generation model.

9. The multi-modal generative model client federated forgetting method of claim 8, wherein, The process of performing knowledge distillation recovery on the intermediate multimodal generation model based on the synthetic anchor point, the initial multimodal generation model, and the intermediate multimodal generation model to obtain the forgotten multimodal generation model includes: The synthetic anchor point is input into the initial multimodal generation model to obtain the first soft probability distribution output by the multimodal generation model; The synthetic anchor point is input into the intermediate multimodal generation model to obtain the second soft probability distribution output by the intermediate multimodal generation model; Based on the first soft probability distribution and the second soft probability distribution, the knowledge distillation recovery loss is determined; With the goal of minimizing the knowledge distillation recovery loss, knowledge distillation recovery is performed on the intermediate multimodal generation model to obtain the forgotten multimodal generation model.

10. A client-level federated forgetting device for a multimodal generative model, characterized in that, include: The first determining module is used to determine the contribution representation of the target client in the initial multimodal generation model based on the cumulative update records of the target client, the cross-modal shared parameter set, the modality-specific parameter set, and the cross-modal dependency parameter set of the target client in the initial multimodal generation model stored by the server during the federated learning training phase. The second determining module is used to determine the parameter region related to the target client in the architecture-aware dependency graph based on the architecture-aware dependency graph corresponding to the initial multimodal generation model and the contribution representation; The architecture-aware dependency graph is used to characterize the forward computation structure of the initial multimodal generation model; The forgetting module is used to perform forgetting processing on the parameter region to obtain the forgotten multimodal generative model.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal generative model client-level federated forgetting method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multimodal generative model client-level federated forgetting method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multimodal generative model client-level federated forgetting method as described in any one of claims 1 to 9.

Images