A copyright protection method and system based on a federated learning model with feature fusion

Abstract

This invention provides a copyright protection method and system for federated learning models based on feature fusion. The method includes: Step 1, receiving watermark trigger samples and inputting them into a pre-established client model for training; Step 2, updating and aggregating the local shared model parameters and local classification layer parameters to obtain updated global model shared parameters and personalized classification layer parameters; extracting watermark features from the updated global model shared parameters based on a feature layer watermark embedding mechanism to obtain a global watermark; Step 3, performing multi-bit quantization on the personalized classification layer parameters to obtain a set of binary signatures; verifying copyright based on the black-box watermark and the global watermark, and identifying malicious clients based on the set of binary signatures. This invention can effectively curb the illegal copying and distribution of models, providing key technical support for promoting the healthy and sustainable development of data and model collaboration paradigms.

Description

Technical Field

[0001] This invention relates to the field of federated learning security technology, specifically to a method and system for copyright protection of federated learning models based on feature fusion. Background Technology

[0002] Federated learning, as an emerging distributed machine learning paradigm, allows multiple participants to collaboratively train a shared deep neural network model without directly sharing their local raw data. In this framework, each participant trains the model locally, and a central server then aggregates and updates the model parameters uploaded by each participant. This mechanism, which uploads only parameters or gradients rather than the original data, fundamentally avoids the risk of raw data leakage, thus possessing an inherent advantage in ensuring data privacy. Therefore, it is widely used in data privacy-sensitive fields such as healthcare, finance, and the military. However, the distributed training nature of federated learning grants each participant access to and control over their local model, which also introduces new security risks: malicious participants may steal, copy, or illegally distribute the model, posing a significant challenge to the intellectual property protection of the model owner.

[0003] To address the risks associated with model copyright protection and malicious activity tracing, federated learning model watermarking technology has emerged. This technology embeds verifiable watermark information into the model, aiming to verify model copyright and track malicious actors without compromising the model's original performance. Currently, federated learning model watermarking technologies are mainly divided into two categories: client-side model watermarking and server-side model watermarking. Specifically, client-side watermarking schemes utilize the client's contribution to federated learning training for verification. However, when a client maliciously steals the model, this type of scheme faces difficulties in tracing the specific responsible party, allowing the leaker to escape accountability. Server-side watermarking schemes, to achieve accurate tracing of the leak source, typically require generating and embedding a unique personalized watermark for each client. However, this process often requires the server to repeatedly perform model training or fine-tuning for different clients to inject the watermark, resulting in huge and unnecessary computational overhead, severely limiting the system's scalability and the total number of clients it can support. An ideal federated learning copyright protection framework should recognize the dual ownership of the server and client as co-participants. Building upon this, the server, acting as the central coordinator, should be granted higher-level verification permissions to achieve authoritative arbitration and precise tracing of malicious behavior. However, existing solutions generally grant verification authority to a single party, failing to establish an effective multi-party collaborative mechanism for confirming rights and assigning responsibility. Furthermore, due to the inherent data heterogeneity in federated learning environments, the data from each client typically exhibits non-independent and identically distributed (Non-IID) characteristics. This causes a fundamental conflict between existing model watermarking technologies, which prioritize global robustness and accuracy of watermark information, and the goals of federated learning. The global model needs to consider the adaptability of local data from each client, but watermark embedding operations often force the model to sacrifice fitting accuracy for specific local data. This contradiction directly undermines the fairness principle of federated learning and reduces the performance of the global model on some clients. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a copyright protection method and system for federated learning models based on feature fusion, which addresses the shortcomings of the existing technology.

[0005] The method includes the following steps:

[0006] Step 1: Receive watermark trigger samples and input them into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding the black-box watermark inside the convolutional neural network model, the local shared model parameters and local classification layer parameters are obtained.

[0007] Step 2: Based on the personalized parameter distribution mechanism, update and aggregate the local shared model parameters and local classification layer parameters respectively to obtain the updated global model shared parameters and personalized classification layer parameters; extract watermark features from the updated global model shared parameters based on the feature layer watermark embedding mechanism to obtain the global watermark.

[0008] Step 3: Based on the personalized signature extraction mechanism, multi-bit quantization is performed on the personalized classification layer parameters to obtain a set of binary signatures; copyright verification is performed based on black-box watermark and global watermark; and malicious clients are identified based on the set of binary signatures.

[0009] In step 1, normal training samples and corresponding triggers for the client are obtained. The normal training samples are combined with the corresponding triggers to obtain watermark trigger samples, specifically including:

[0010] For each client's normal training samples ,in Represents the original sample. express Corresponding tags This represents normal training samples, for each client. Received a client-specific local trigger from the server. , N represents the total number of clients, i.e., the total number of local triggers; by overlaying client-specific triggers onto the original sample Above, generate watermark trigger samples , This indicates an overlay operation, while simultaneously assigning a watermark target label. Watermark trigger sample used by each client Compared with normal training samples Both are used together to train the client model.

[0011] In step 1, the client model is trained simultaneously on both normal training samples and watermark-triggered samples, with a total training loss of The calculation formula is:

[0012] ,

[0013] Where M(⋅) represents the client model, It is the classification loss of the main task. It is the classification loss of the watermarking task. It is a hyperparameter that balances the losses of the main task and the watermarking task. By minimizing While learning the main task, the client model also embeds the feature responses unique to the corresponding watermark-triggered samples.

[0014] Step 2 includes: the server processing all participating clients. , Locally shared model parameters reported in each round Perform weighted average aggregation to obtain the globally shared model parameters at the (t+1)th training epoch. :

[0015] ,

[0016] in, This indicates the size of the client's local dataset. Where t represents the training round of federated learning, This indicates the final round of federated learning training for each client. Reported local classification layer parameters Perform federated averaging to obtain the global classification layer parameters at the (t+1)th training epoch. :

[0017] ,

[0018] Calculate the local classification layer parameter vector for each client. With the global classification layer parameter vector Cosine similarity between Then based on cosine similarity Dynamically calculate the weights of the parameters used for client-side personalized classification layer. :

[0019] ,

[0020] ,

[0021] ,

[0022] in, Indicates the parameters of the personalized classification layer. and These are preset hyperparameters. =0.3, ;

[0023] The server packages the aggregated globally shared model parameters and the personalized classification layer parameters generated for each client, and sends them to the corresponding clients. The process of packaging and distributing is as follows:

[0024] ,

[0025] in, This indicates that an operation has been issued.

[0026] Step 2 also includes: the server using a sample generator Generate a batch of samples for watermark activation. And the server defines a global trigger. Local triggers are a combination of all positions of global triggers. By superimposing global triggers onto 640 generated samples, activation samples with global triggers are generated. The server uses the globally shared model parameters in the final round. Extract activation samples separately exist Activation features of layers and generate samples exist Activation features of layers :

[0027] ,

[0028] ,

[0029] in, This represents a layer in a convolutional neural network model. and This indicates the operation of extracting features;

[0030] The server will extract the activation features respectively and Input to watermark extraction model Extract the global watermark bit sequence respectively and arbitrary random watermark bit sequence Meanwhile, the watermark extraction model was trained. The goal is to minimize the following contrast loss function :

[0031] ,

[0032] Among them, the watermark extraction model It is a model pre-built on the server to train the global watermark. It is a fully connected neural network model, MLP; contrast loss function The constrained watermark extraction model will incorporate global fusion features. Mapped to global watermark bit sequence And simultaneously suppress the activation features generated by the watermark extraction model without a global trigger. Mistakenly judged as a legitimate watermark And map to any random watermark bit sequence. , Represents the binary cross-entropy loss. Watermark extraction model Trainable parameters.

[0033] Step 3 includes:

[0034] Obtain personalized classification layer parameters from a convolutional neural network model generated by calculating cosine similarity, dynamic weight allocation, and weighted summation. There are a total of classification layers One channel, each channel Each corresponds to a set of parameters Then calculate the L2 norm of each channel j. Meanwhile, during training, the server treats the same channels of each client's local model as a group and saves the minimum norm value of each group. and maximum norm value Then, for each group, use the minimum norm value. and maximum norm value For the channel norm Perform interval normalization:

[0035] ,

[0036] ,

[0037] in, This represents the normalized L2 norm of the j-th channel of the i-th client;

[0038] Normalized L2 norm Perform multi-bit quantization to generate a binary signature, first by quantizing each client... The The norm of each channel Mapped to the corresponding index The corresponding operation is as follows:

[0039] , ,

[0040] The number of quantization intervals K is determined by the bit parameter bits of each preset quantization index mapping: bits = 2 or 4, each quantization index A binary string corresponding to one bit;

[0041] Then, Operation by group each Mapping to the corresponding The following bits are the binary string, and finally each client will have all of them. The binary strings of each channel are concatenated to form a string of length . The complete binary signature, which each client possesses, is represented as follows:

[0042] ,

[0043] in, Sign the final personalized category layer for the client. This is a string concatenation operation. This indicates that the quantization index is encoded as A binary string.

[0044] Step 3 also includes:

[0045] Each client prepares a set containing A stacked dedicated local trigger Black box watermark verification set For each watermark trigger sample in the verification set The input is fed into the suspicious model M to obtain the predicted label. Subsequently through Function to perform model copyright verification:

[0046] ,

[0047] Among them, the suspicious model M is a model defined as one where malicious client copying or abusing the client exists. This represents the nth original sample. , It is an indicator function; if the matching rate is greater than a preset threshold... =0.9, then the function A value of true indicates that the copyright of a suspicious model can be quickly verified; otherwise, a value of false indicates a false value.

[0048] The server performs a higher level of model copyright verification through global watermarking: First, it uses a watermark with a global trigger. Generate samples Extract suspicious models exist Activation features of layers Then, The watermark extraction model is input into the server. The extracted watermark bit sequence is obtained. Finally, calculate Preset global watermark bit sequence The watermark extraction rate between them, through Function to perform model copyright verification:

[0049] ,

[0050] ,

[0051] Where B=256 is the number of bits; This represents the k-th bit of the extracted watermark bit sequence. This represents the k-th bit of the preset watermark bit sequence. If the watermark extraction rate is greater than the threshold =0.95, then the function If the result is true, it means that the copyright of the suspicious model can be quickly verified; otherwise, it is false, indicating a false value.

[0052] After confirming that the template copyright has been stolen, the malicious client is traced and located: the classification layer parameters are extracted from the classification layer of the suspicious model M, and the signature to be detected of the suspicious model is extracted. The signature to be tested Personalized signatures for each client stored on the server Compare and calculate one by one. and The Hamming distance between them is calculated as follows:

[0053] ,

[0054] in, Indicates the signature to be detected Personalized signatures for all clients The Hamming distance between them was calculated, and the client sequence number with the smallest Hamming distance was identified. And if the minimum Hamming distance is less than a preset similarity threshold =0.9, then the final determination is the client. It is a malicious client that leaks the model.

[0055] This invention also provides a copyright protection system based on feature fusion and federated learning model implemented using the method described above, comprising:

[0056] The watermark processing module is used to receive watermark trigger samples and input the watermark trigger samples into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding the black box watermark inside the convolutional neural network model, the local shared model parameters and local classification layer parameters are obtained.

[0057] The feature extraction module is used to update and aggregate the local shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism to obtain the updated global model shared parameters and personalized classification layer parameters; and to extract watermark features from the updated global model shared parameters based on the feature layer watermark embedding mechanism to obtain the global watermark.

[0058] The signature verification module is used to perform multi-bit quantization of personalized classification layer parameters based on the personalized signature extraction mechanism to obtain a set of binary signatures; to perform copyright verification based on black-box watermarks and global watermarks; and to identify malicious clients based on the set of binary signatures.

[0059] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0060] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0061] Beneficial Effects: This invention constructs a technical defense system through a multi-layered collaborative watermarking framework, injecting core protective capabilities into the federated learning security ecosystem. In the copyright verification dimension, the client-side black-box watermark and the server-side deep feature watermark complement each other. The former achieves lightweight local rights confirmation through locally triggered samples, while the latter constructs a more robust deep watermark identifier based on the feature maps of the model's intermediate activation layers. This dual verification mechanism maintains the robust retention of watermark information while preserving its hidden embedding state within the model structure when combating common attacks such as model pruning and parameter perturbation, providing a technically rigid arbitration basis for copyright ownership involving multiple parties. In the malicious behavior tracing dimension, a personalized signature watermark with a unique mapping relationship is constructed by deeply mining the differentiated features of the client's classification layer parameters. This mechanism breaks through the high-cost tracing mode of traditional solutions that rely on repetitive training. Based on the native parameter interaction process of federated learning, it achieves dynamic binding of watermarks, establishes a precise mapping relationship of client feature fingerprints, and ensures that accountability for violations can be achieved through parameter feature matching in model leakage events. This invention effectively curbs the illegal copying and distribution of models, aiming to strengthen the inherent trust and security foundation of federated learning, and providing key technical support for promoting the healthy and sustainable development of data and model collaboration paradigms. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0063] Figure 2 This is a schematic diagram of the overall framework for federated learning watermark embedding in this invention.

[0064] Figure 3 This is a schematic diagram of the server embedding a global watermark according to the present invention.

[0065] Figure 4 This is a schematic diagram illustrating the extraction of the client's personalized classification layer signature by the server according to the present invention.

[0066] Figure 5 This is a schematic diagram of the overall framework for federated learning copyright verification and tracking of malicious clients in this invention.

[0067] Figure 6 This is a schematic diagram of the system structure of the present invention.

[0068] Figure 7 This is an experimental result of the present invention resisting pruning attacks on the CIFAR10 dataset.

[0069] Figure 8 This is an experimental result figure showing the effectiveness of this invention in resisting fine-tuning attacks on the CIFAR10 dataset. Detailed Implementation

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the above-mentioned contents or other advantages of the present invention will become clearer.

[0071] like Figure 1 As shown, this embodiment provides a copyright protection method for federated learning models based on feature fusion, including the following steps:

[0072] Step 1: Receive watermark trigger samples and input them into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding the black-box watermark inside the convolutional neural network model, the local shared model parameters and local classification layer parameters are obtained.

[0073] The watermark trigger sample is obtained by acquiring the client's normal training sample and the client's corresponding trigger, and combining the client's normal training sample with the client's corresponding trigger. The process includes:

[0074] For each client's normal training samples ,in Represents the original sample. express Corresponding tags This represents normal training samples, for each client. Received a client-specific local trigger from the server. , N represents the total number of clients, i.e., the total number of local triggers; by overlaying client-specific triggers onto the original sample Above, generate watermark trigger samples , This indicates an overlay operation, while simultaneously assigning a watermark target label. Watermark trigger sample used by each client Compared with normal training samples These are used together to train the client model. This step not only perturbs the original samples at the pixel level, but also implants unique response patterns within the feature space. Each client embeds only a very small number (far less than the normal amount of data) of unique watermark-triggered samples into its local model. Compared with its normal training data samples mix.

[0075] The process of inputting watermark-triggered samples into the client model for training, embedding black-box watermarks within the convolutional neural network model, and simultaneously obtaining locally shared model parameters and local classification layer parameters includes:

[0076] The client-side model is trained on both normal training samples and watermark-triggered samples simultaneously, and its total training loss is composed of the main task loss. and watermark loss It is composed of weighted combinations.

[0077] ,

[0078] Where M(⋅) represents the client model, It is the classification loss of the main task. It is the classification loss of the watermarking task. It is a hyperparameter that balances the losses of the main task and the watermarking task. By minimizing While learning the main task, the client model also embeds the feature responses unique to the corresponding watermark-triggered samples.

[0079] Step 2: Based on the personalized parameter distribution mechanism, update and aggregate the local shared model parameters and local classification layer parameters respectively to obtain the updated global model shared parameters and personalized classification layer parameters; based on the feature layer watermark embedding mechanism, extract watermark features from the updated global model shared parameters to obtain the global watermark.

[0080] The process of updating and aggregating the locally shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism to obtain updated global model shared parameters and personalized classification layer parameters includes:

[0081] The server provides data to all participating clients. , Locally shared model parameters reported in each round Perform weighted average aggregation to obtain the globally shared model parameters at the (t+1)th training epoch. :

[0082] ,

[0083] in, This indicates the size of the client's local dataset. Where t represents the training round of federated learning, This indicates the final round of federated learning training for each client. Reported local classification layer parameters Perform federated averaging to obtain the global classification layer parameters at the (t+1)th training epoch. :

[0084] ,

[0085] To address the issue of high heterogeneity in client-side data within federated learning environments, directly using the local classification layer parameters of each client for classification layer signature extraction often results in significant individual shifts in these parameters due to substantial differences in data distribution across clients. This variation significantly weakens the robustness of the classification layer signature. Specifically, during subsequent model fine-tuning or when subjected to attack perturbations, the classification layer parameters of individual clients are prone to drastic changes, leading to unstable extracted classification layer signatures and impacting the reliability of source tracing and the consistency of watermark verification. To resolve this issue, this method first obtains the global classification layer parameters. Calculate the local classification layer parameter vector for each client. With the global classification layer parameter vector Cosine similarity between Then based on cosine similarity Dynamically calculate the weights of the parameters used for client-side personalized classification layer. :

[0086] ,

[0087] ,

[0088] ,

[0089] in, Indicates the parameters of the personalized classification layer. and These are preset hyperparameters. =0.3, ;

[0090] The server packages the aggregated globally shared model parameters and the personalized classification layer parameters generated for each client, and sends them to the corresponding clients. The process of packaging and distributing is as follows:

[0091] ,

[0092] in, This indicates that an operation has been issued.

[0093] The feature-layer watermark embedding mechanism extracts watermark features from the updated global model shared parameters to obtain a global watermark. The process includes:

[0094] The server uses a sample generator. Generate a batch of samples for watermark activation. And the server defines a global trigger. A local trigger is a combination of all positions of a global trigger. By superimposing global triggers onto a portion of the generated samples, an activated sample with a global trigger is produced. Essentially, the role of these activation samples and global triggers is to "awaken" the global model's fusion response to all client-embedded watermark features in the intermediate layers. During the initial training phase, the specific watermark features embedded by each client through local triggers have been effectively fused into the global model parameters, forming a significant distinction from other features in the feature space. This ensures that each client's local features are not only fully preserved within the global model but also exhibit high recognizability and discriminability during subsequent activation feature extraction and watermark verification. These activation samples trigger coupled responses from client features in the intermediate activation layers of the global model, thus forming a globally unique watermark representation in the feature space. Then, the server uses the final round of globally shared model parameters... Extract activation samples separately and generate samples exist The activation features of the layer are as follows:

[0095] ,

[0096] ,

[0097] in, This represents a layer in a convolutional neural network model. and This indicates the feature extraction operation; the feature extraction here is not limited to representing the generated sample itself, but more importantly, it reflects the global fusion effect of the client's local watermark features in the feature space after global aggregation.

[0098] The server will extract the activation features respectively and Input to watermark extraction model Extract the global watermark bit sequence b and the arbitrary random watermark bit sequence respectively. At the same time, training The goal is to minimize the following contrast loss function :

[0099] ,

[0100] Among them, the watermark extraction model It is a model pre-built on the server to train the global watermark. It is a fully connected neural network model, MLP; contrast loss function The constrained watermark extraction model will incorporate global fusion features. Mapped to global watermark bit sequence And simultaneously suppress the activation features generated by the watermark extraction model without a global trigger. Mistakenly judged as a legitimate watermark And map to any random watermark bit sequence. , Represents the binary cross-entropy loss. Watermark extraction model Trainable parameters.

[0101] Step 3: Based on the personalized signature extraction mechanism, multi-bit quantization is performed on the personalized classification layer parameters to obtain a set of binary signatures; copyright verification is performed based on black-box watermark and global watermark; and malicious clients are identified based on the set of binary signatures.

[0102] The personalized signature extraction mechanism involves multi-bit quantization of the personalized classification layer parameters to obtain a set of binary signatures. The process includes:

[0103] In the final round of federated learning, server-side classification layer signature extraction is performed. First, personalized classification layer parameters are obtained from the convolutional neural network model generated by calculating cosine similarity, dynamic weight allocation, and weighted summation. There are a total of classification layers One channel, each channel Each corresponds to a set of parameters Then calculate the L2 norm of each channel j. Meanwhile, during training, the server treats the same channels of each client's local model as a group and saves the minimum norm value of each group. and maximum norm value Then, for each group, use the minimum norm value. and maximum norm value For the channel norm Perform interval normalization:

[0104] ,

[0105] ,

[0106] in, This represents the normalized L2 norm of the j-th channel of the i-th client;

[0107] Normalized L2 norm Perform multi-bit quantization to generate a binary signature, first by quantizing each client... The The norm of each channel Mapped to the corresponding index The corresponding operation is as follows:

[0108] , ,

[0109] The number of quantization intervals K is determined by the bit parameter bits of each preset quantization index mapping: bits = 2 or 4, each quantization index A binary string corresponding to one bit;

[0110] Then, Operation by group each Mapping to the corresponding The following bits are the binary string, and finally each client will have all of them. The binary strings of each channel are concatenated to form a string of length . The complete binary signature, which each client possesses, is represented as follows:

[0111] ,

[0112] in, Sign the final personalized category layer for the client. This is a string concatenation operation. This indicates that the quantization index is encoded as A binary string. Copyright verification is performed based on a combination of black-box watermarks, binary signatures, and a global watermark; malicious clients are identified based on the verification results.

[0113] The copyright verification based on black-box watermarking and global watermarking, and the determination of malicious clients based on a set of binary signatures, include the following process:

[0114] Each client prepares a set containing A stacked dedicated local trigger Black box watermark verification set For each watermark trigger sample in the verification set The input is fed into the suspicious model M to obtain the predicted label. Subsequently through Function to perform model copyright verification:

[0115] ,

[0116] Among them, the suspicious model M is a model defined as one where malicious client copying or abusing the client exists. This represents the nth original sample. , It is an indicator function; if the matching rate is greater than a preset threshold... =0.9, then the function A value of true indicates that the copyright of a suspicious model can be quickly verified; otherwise, a value of false indicates a false value.

[0117] The server performs a higher level of model copyright verification through global watermarking: First, it uses a watermark with a global trigger. Generate samples Extract suspicious models exist Activation features of layers Then, The watermark extraction model is input into the server. The extracted watermark bit sequence is obtained. Finally, calculate Preset global watermark bit sequence The watermark extraction rate between them, through Function to perform model copyright verification:

[0118] ,

[0119] ,

[0120] Where B=256 is the number of bits; This represents the k-th bit of the extracted watermark bit sequence. This represents the k-th bit of the preset watermark bit sequence. If the watermark extraction rate is greater than the threshold =0.95, then the function If the result is true, it means that the copyright of the suspicious model can be quickly verified; otherwise, it is false, indicating a false value.

[0121] After confirming that the template copyright has been stolen, the malicious client is traced and located: the classification layer parameters are extracted from the classification layer of the suspicious model M, and the signature to be detected of the suspicious model is extracted. The signature to be tested Personalized signatures for each client stored on the server Compare and calculate one by one. and The Hamming distance between them is calculated as follows:

[0122] ,

[0123] in, Indicates the signature to be detected Personalized signatures for all clients The Hamming distance between them was calculated, and the client sequence number with the smallest Hamming distance was identified. And if the minimum Hamming distance is less than a preset similarity threshold =0.9, then the final determination is the client. It is a malicious client that leaks the model.

[0124] Specifically, the present invention will be further illustrated below through embodiments:

[0125] To verify the effectiveness of the proposed feature fusion watermark embedding scheme in a federated learning environment, this embodiment designs an image classification task, selecting two different model architectures: ConvNet and ResNet18, respectively trained on the MNIST and CIFAR10 datasets. Taking CIFAR10 and ResNet-18 as examples, ResNet18 is a deep residual network that effectively solves the gradient vanishing problem in deep neural network training by introducing residual connections, and exhibits excellent performance in relatively complex image recognition tasks. The MNIST dataset contains 60,000 handwritten digit images, of which 60,000 are used for training and 10,000 for testing. This dataset has 10 categories, and this example verifies the effectiveness of the method.

[0126] The following is combined Figure 2 A detailed explanation of embedding black-box watermarks within the client-side model:

[0127] The first step is that in each round of training, the client receives the globally shared model parameters sent by the server. and personalized classification layer parameters Meanwhile, the client uses a local trigger issued by the server to construct a watermark trigger sample. .

[0128] The second step involves the client-side model simultaneously training on normal samples. and watermark trigger samples During training, a black-box watermark is embedded within the client model while learning the main task. This local feature watermark embedding ensures that the watermark features of each client are an integral part of the global watermark. After training, the client sends the updated globally shared model parameters and personalized classification layer parameters back to the server.

[0129] The following is combined Figure 2 A detailed explanation of the server's personalized parameter distribution mechanism is provided:

[0130] On the server side, to achieve accurate tracing of maliciously leaking clients in a federated learning environment, this invention introduces a personalized parameter distribution mechanism during the model parameter distribution stage. This mechanism incorporates the personalized classification layer parameter concept from personalized federated learning (such as FedPer and FedRep) and innovatively applies it to the scenario of tracing heterogeneous data in federated learning. Specifically, the server not only obtains globally shared model parameters based on standard federated aggregation, but also performs secondary aggregation and dynamic differentiated distribution of classification layer parameters for each client. Through dynamic weight allocation, customized parameter transmission for each client's personalized features is achieved. This realizes deep coupling between personalized federated learning, model copyright verification, and accurate tracking of malicious clients. While effectively improving the local performance and adaptability of each client's model, it can generate unique and highly discriminative signatures based on the differentiated personalized classification layer parameters, thus providing technical assurance for subsequent model tracing and copyright attribution. The overall process consists of the following two steps:

[0131] The first step is to perform a routine federated aggregation of the shared model parameters reported by all clients, and then update the globally shared model parameters using the aggregation algorithm employed by the server. Secondly, regarding the key steps of personalized model parameters, the server dynamically adjusts the weights of each parameter in the final personalized classification layer by calculating the cosine similarity between the global and each client's classification layer parameters.

[0132] The second step involves using these adjusted weights to generate a completely new set of personalized model parameters for each client. Finally, the globally shared model parameters and the adjusted personalized classification layer parameters are distributed to different clients.

[0133] The following is combined Figure 2 and Figure 3 A detailed explanation of embedding global watermarks on the server:

[0134] The first step, after the entire training process of federated learning is completed, is that the server uses a sample generator. Generate a batch of samples for watermark activation. And the server defines a global trigger. A local trigger is a combination of all positions of a global trigger. By superimposing global triggers onto a portion of the generated samples, an activated sample with a global trigger is produced. The server uses the globally shared model parameters in the final round. Extract activation samples separately and generate samples exist Activation characteristics of the layer.

[0135] In the second step, the server inputs the aforementioned feature maps into the watermark extraction model, optimizes the extraction parameters through backpropagation, and ultimately enables it to accurately project the global activation features into a predefined binary watermark bit string. After training converges, the server has the ability to reliably extract the global watermark from the fused features.

[0136] The following is combined Figure 2 and Figure 4 A detailed explanation of how the server extracts the client's personalized category layer signature:

[0137] The first step is to obtain the personalized classification layer parameters generated in the last round of training by calculating cosine similarity, dynamic weight allocation, and weighted summation. Then calculate the L2 norm of each channel j in the personalized classification layer parameters. Meanwhile, during training, the server treats the same channels from each client as a group and saves the minimum norm value of each group. and maximum norm value Then, for each group, these two values ​​are used to adjust the channel L2 norm. Perform interval normalization to obtain the representation of the first interval. The first client's Normalized parameters for each channel .

[0138] The second step is to normalize the norm of each channel. Perform multi-bit quantization to generate a binary signature, first by quantizing each client... The The norm of each channel Mapped to the corresponding index Below, and through Operation by group each Mapping to the corresponding Below A binary string of bits.

[0139] Step 3: Transfer all data to each client The binary strings of each channel are concatenated to form a string of length . The server obtains the complete binary signature from the client, and finally receives a complete personalized classification layer signature from the client. .

[0140] The following is combined Figure 5 A detailed explanation of the process for federated learning copyright verification and tracking malicious clients:

[0141] The first step is for the client to trigger a sample using a unique watermark. and Perform local verification; if the matching rate is greater than a preset threshold... This enables rapid verification of model attribution.

[0142] The second step involves the server acting as the central coordinator, possessing higher-level model copyright verification permissions. When the server obtains a model suspected of being leaked, it initiates global watermark verification. Specifically, the server extracts the suspicious model. exist Activation features of layers Then, The watermark extraction model is input into the server. The extracted watermark bit sequence is obtained. Subsequently, the verification function was used. Determine if the matching rate is greater than a preset threshold. If this is the case, it indicates that the model may have been tampered with or there is a risk of copyright leakage.

[0143] The third step, after the server's feature-layer watermark verification reveals copyright risks, is to further utilize personalized classification-layer signatures for tracing the source of the leak in order to accurately pinpoint the source. The server extracts the personalized classification-layer signature from suspicious models. Then, using the tracking function The calculation is performed, and if the minimum Hamming distance is less than a preset similarity threshold... Then the client will be definitively determined. It is a malicious client that leaks the model.

[0144] like Figure 2As shown, the watermark embedding process begins with all clients (client 1, client 2... client N) receiving local triggers from the server during each training round and training their models using their local data. While embedding the black-box watermark within the model, they obtain locally shared model parameters and local classification layer parameters, which are then uploaded to the server. Next, the server first updates the aggregated model parameters, then calculates the personalized classification layer parameters, and finally distributes these personalized model parameters, resulting in different model parameters for each client. Finally, the server performs feature layer watermark embedding and personalized signature extraction for subsequent server model copyright verification and tracking of malicious clients.

[0145] like Figure 3 As shown, the feature layer watermark embedding process first superimposes the global trigger onto the generated samples generated by a portion of the generator. Above, generate activation samples with global triggers. The server uses the shared model parameters in the last round. Extract activation samples separately and generate samples exist Activation features of layers , Then the server extracts the activation features respectively. and The input is fed into the watermark extraction model, and the model is trained to extract the global watermark bit sequence. and arbitrary random watermark bit sequences Meanwhile, the parameters of the watermark extraction model are optimized; finally, the watermark extraction model can stably output a global watermark bit sequence based on activation features. .

[0146] like Figure 4 As shown, the personalized signature extraction process first involves the server obtaining personalized classification layer parameters generated by all clients (client 1, client 2, ..., client N) through calculation of cosine similarity, dynamic weight allocation, and weighted summation. Then, the server calculates the normalized L2 norm of each channel j of the personalized classification layer parameters. , , Client 1: , Client 2: , Client N: ... , Then, the normalized norm of each channel is... Binary multi-bit quantization is performed to generate a binary signature (Client 1: 00, 01, Client 2: 01, 11, ..., Client N: 11, 00). Finally, the binary strings of all channels from each client are concatenated to form a complete binary signature. , , ..., Finally, sign the complete personalized category layer for each client.

[0147] like Figure 5 As shown, the process of model copyright verification and tracking malicious clients includes the following steps: First, each client triggers a sample through its own watermark, allowing for independent client verification and the assertion of ownership of the model. Step 1: Each client prepares a set of samples containing... A stacked dedicated local trigger Black box watermark verification set For each watermark trigger sample in the verification set Input into the suspicious model , obtain predicted labels Subsequently through Function to perform model copyright verification:

[0148] ,

[0149] Among them, the suspicious model It is a model designed to prevent malicious client copying or client abuse. Indicates the first One original sample, , It is an indicator function; if the matching rate is greater than a preset threshold... =0.9, then the function A value of true indicates that the copyright of a suspicious model can be quickly verified; otherwise, a value of false indicates a false value.

[0150] Step 2: The server performs a higher level of model copyright verification using a global watermark: First, it uses a system with a global trigger. Generate samples Extract suspicious models exist Activation features of layers Then, The watermark extraction model is input into the server. The extracted watermark bit sequence is obtained. Finally, calculate Preset global watermark bit sequence The watermark extraction rate between them, through Function to perform model copyright verification:

[0151] ,

[0152] ,

[0153] Where B=256 is the number of bits; These represent the k-th bits of the extracted and preset watermark bit sequences, respectively. If the watermark extraction rate is greater than the threshold =0.95, then the function A value of true indicates that the copyright of a suspicious model can be quickly verified; otherwise, a value of false indicates a false value.

[0154] After confirming that the template copyright had been stolen, the malicious client was traced and located: from suspicious models. The classification layer parameters are extracted, and the signature to be detected for the suspicious model is extracted. The signature to be tested Personalized signatures for each client stored on the server Compare and calculate one by one. and The Hamming distance between them is calculated as follows:

[0155] ,

[0156] in, Indicates the signature to be detected Personalized signatures for all clients The Hamming distance between them was calculated, and the client sequence number with the smallest Hamming distance was identified. And if the minimum Hamming distance is less than a preset similarity threshold =0.9, then the final determination is the client. It is a malicious client that leaks the model.

[0157] Experimental Setup: Dataset and Model: This method selects two different model architectures: ConvNet and ResNet18 to train on the MNIST and CIFAR10 datasets, respectively. In simulating a federated learning scenario, the default total number of clients N = 50. In each round, 10% of the clients are randomly selected to participate in local updates. Each client performs 5 local training iterations, with data divided into batches of 256. Each client uses stochastic gradient descent (SGD) for parameter updates, with an initial learning rate of 0.1. To construct a Non-IID data distribution, a Dirichlet distribution (parameter α = 0.1) is used to split the data for each client, ensuring data heterogeneity and enhancing the realism of the experiment.

[0158] Evaluation metrics: This invention uses primary task accuracy to evaluate the model's classification accuracy on the original test dataset; black-box watermark detection rate to evaluate the accuracy of the client-side watermark model on watermark-triggered samples; and white-box watermark detection rate to evaluate the extraction rate of the server-embedded global watermark. Watermark collision rate is used to evaluate the discriminative power of the classification layer signature, and watermark tracking rate is used to evaluate the probability of the classification layer watermark tracking malicious clients.

[0159] Fidelity Experiment: To evaluate the impact of this invention on the primary task performance of the model, a good watermarked model should have similar primary task accuracy to the watermark-free model. Table 1 shows the primary task accuracy of this invention compared to the watermark-free method under different numbers of clients and datasets. On both datasets, the primary task accuracy of this invention is consistently close to or slightly higher than that of the watermark-free method under different numbers of clients. This result indicates that this invention not only works effectively without significantly affecting the primary task accuracy, but can also improve the primary task accuracy of the model under certain settings.

[0160] Table 1

[0161]

[0162] Non-independent and identically distributed (ICD) experiments: To evaluate the performance of this invention under different non-independent and identically distributed (ICD) scenarios, this invention simulated four different Dirichlet distribution parameters to verify the main task accuracy, black-box watermark detection rate, white-box watermark extraction rate, watermark collision rate, and watermark tracking rate. At different α values, the unevenness of the data increases as the α value decreases, which directly affects the model's performance in federated learning. As α increases, the main task accuracy, black-box watermark detection rate, and white-box watermark extraction rate are all controlled within a relatively small range, with overall fluctuations within a small interval. Specifically, the main task accuracy fluctuates from 99.34% to 97.95%, with an overall fluctuation of 1.39%. The black-box watermark detection rate fluctuates from 99.94% to 97.55%, with an overall fluctuation of 2.39%; and the white-box watermark extraction rate fluctuates from 96.88% to 96.65%, with an overall fluctuation of 0.23%.

[0163] However, the watermark collision rate remained 0 for all α values, indicating that the classification layer signatures maintained high distinctiveness across different α values, even under conditions of more non-independent and identically distributed data, the classification layer signatures remained highly dissimilar. The watermark tracking rate demonstrated superior tracking ability at α=0.05 and α=0.1, reaching 99% and 98.58% respectively, representing good tracking ability even under highly uneven data distribution, as shown in Table 2.

[0164] Table 2

[0165]

[0166] Robustness Experiment: To resist pruning and fine-tuning attacks on the watermark model, which could lead to watermark removal, this invention evaluates the robustness of the watermark model against both pruning and fine-tuning attacks. Figure 7 As shown, pruning experiments on the CIFAR10 dataset demonstrate that as the pruning rate increases, the various indicators remain relatively stable. When the pruning rate increases to 0.5, the black-box watermark detection rate is above 90%, the white-box watermark extraction rate remains above 95%, and the watermark tracking rate remains essentially unchanged. Figure 8 As shown in the fine-tuning experiments on the CIFAR10 dataset, the fluctuations in the main task accuracy and white-box watermark extraction rate are within a very small range. When the number of fine-tuning rounds is increased to 50, the black-box watermark detection rate and watermark tracking rate are still greater than 95%. These results demonstrate that although pruning and fine-tuning affect the watermark model's metrics to some extent, neither can completely remove the watermark from the model, showcasing the robustness of this invention under both attacks.

[0167] like Figure 6 As shown, based on the aforementioned method, this embodiment also provides a copyright protection system for a federated learning model based on feature fusion, including:

[0168] The watermark processing module is used to receive watermark trigger samples and input the watermark trigger samples into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding the black box watermark inside the convolutional neural network model, the local shared model parameters and local classification layer parameters are obtained.

[0169] The feature extraction module is used to update and aggregate the local shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism to obtain the updated global model shared parameters and personalized classification layer parameters; and to extract watermark features from the updated global model shared parameters based on the feature layer watermark embedding mechanism to obtain the global watermark.

[0170] The signature verification module is used to perform multi-bit quantization of personalized classification layer parameters based on the personalized signature extraction mechanism to obtain a set of binary signatures; to perform copyright verification based on black-box watermarks and global watermarks; and to identify malicious clients based on the set of binary signatures.

[0171] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0172] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0173] This invention provides a copyright protection method and system for a federated learning model based on feature fusion. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A copyright protection method for a federated learning model based on feature fusion, characterized in that, Includes the following steps: Step 1: Receive watermark trigger samples and input them into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding the black-box watermark inside the convolutional neural network model, the local shared model parameters and local classification layer parameters are obtained. Step 2: Based on the personalized parameter distribution mechanism, update and aggregate the local shared model parameters and local classification layer parameters respectively to obtain the updated global model shared parameters and personalized classification layer parameters. The global watermark is obtained by extracting watermark features from the updated global model shared parameters based on the feature layer watermark embedding mechanism. Step 3: Based on the personalized signature extraction mechanism, perform multi-bit quantization on the personalized classification layer parameters to obtain a set of binary signatures; Copyright verification is performed based on black-box watermarking and global watermarking, and malicious clients are identified based on a set of binary signatures. In step 1, normal training samples and corresponding triggers for the client are obtained. The normal training samples are combined with the corresponding triggers to obtain watermark trigger samples, specifically including: For each client's normal training samples ,in Represents the original sample. express Corresponding tags This represents normal training samples, for each client. Received a client-specific local trigger from the server. , N represents the total number of clients, i.e., the total number of local triggers; by overlaying client-specific triggers onto the original sample Above, generate watermark trigger samples , This indicates an overlay operation, while simultaneously assigning a watermark target label. Watermark trigger sample used by each client Compared with normal training samples Together they are used to train the client-side model; Step 2 includes: the server processing all participating clients. , Locally shared model parameters reported in each round Perform weighted average aggregation to obtain the globally shared model parameters at the (t+1)th training epoch. : ， in, This indicates the size of the client's local dataset. Where t represents the training round of federated learning, This indicates the final round of federated learning training for each client. Reported local classification layer parameters Perform federated averaging to obtain the global classification layer parameters at the (t+1)th training epoch. : ， Calculate the local classification layer parameter vector for each client. With the global classification layer parameter vector Cosine similarity between Then based on cosine similarity Dynamically calculate the weights of the parameters used for client-side personalized classification layer. : ， ， ， in, Indicates the parameters of the personalized classification layer. and These are preset hyperparameters; The server packages the aggregated globally shared model parameters and the personalized classification layer parameters generated for each client, and sends them to the corresponding clients. The process of packaging and distributing is as follows: ， in, This indicates that an operation has been issued.

2. The method according to claim 1, characterized in that, In step 1, the client model is trained simultaneously on both normal training samples and watermark-triggered samples, with a total training loss of The calculation formula is: ， Where M(⋅) represents the client model, It is the classification loss of the main task. It is the classification loss of the watermarking task. It is a hyperparameter that balances the losses of the main task and the watermarking task. By minimizing While learning the main task, the client model also embeds the feature responses unique to the corresponding watermark-triggered samples.

3. The method according to claim 2, characterized in that, Step 2 also includes: the server using a sample generator Generate a batch of samples for watermark activation. And the server defines a global trigger. A local trigger is a combination of all positions of a global trigger. By superimposing global triggers onto the generated sample, an activated sample with global triggers is generated. The server uses the globally shared model parameters in the final round. Extract activation samples separately exist Activation features of layers and generate samples exist Activation features of layers : ， ， in, This represents a layer in a convolutional neural network model. and This indicates the operation of extracting features; The server will extract the activation features respectively and Input to watermark extraction model Extract the global watermark bit sequence respectively and arbitrary random watermark bit sequence Meanwhile, a watermark extraction model was trained. The goal is to minimize the following contrast loss function : ， Among them, the watermark extraction model It is a model pre-built on the server to train the global watermark. It is a fully connected neural network model, MLP; contrast loss function The constrained watermark extraction model will incorporate global fusion features. Mapped to global watermark bit sequence And simultaneously suppress the activation features generated by the watermark extraction model without a global trigger. Mistakenly judged as a legitimate watermark And map to any random watermark bit sequence. , Represents the binary cross-entropy loss. Watermark extraction model Trainable parameters.

4. The method according to claim 3, characterized in that, Step 3 includes: Obtain personalized classification layer parameters from a convolutional neural network model generated by calculating cosine similarity, dynamic weight allocation, and weighted summation. There are a total of classification layers One channel, each channel Each corresponds to a set of parameters Then calculate the L2 norm of each channel j. Meanwhile, during training, the server treats the same channels of each client's local model as a group and saves the minimum norm value of each group. and maximum norm value Then, for each group, use the minimum norm value. and maximum norm value For the channel norm Perform interval normalization: ， ， in, This represents the normalized L2 norm of the j-th channel of the i-th client; Normalized L2 norm Perform multi-bit quantization to generate a binary signature, first by quantizing each client... The The norm of each channel Mapped to the corresponding index The corresponding operation is as follows: ， ， The number of quantization intervals K is determined by the bit parameter bits of each preset quantization index mapping: bits = 2 or 4, each quantization index A binary string corresponding to one bit; Then, Operation by group each Mapping to the corresponding The following bits are the binary string, and finally each client will have all of them. The binary strings of each channel are concatenated to form a string of length . The complete binary signature, which each client possesses, is represented as follows: ， in, Sign the final personalized category layer for the client. This is a string concatenation operation. This indicates that the quantization index is encoded as A binary string.

5. The method according to claim 4, characterized in that, Step 3 also includes: Each client prepares a set containing A stacked dedicated local trigger Black box watermark verification set For each watermark trigger sample in the verification set The input is fed into the suspicious model M to obtain the predicted label. Subsequently through Function to verify model copyright: ， Among them, the suspicious model M is a model defined as one where malicious client copying or abusing the client exists. This represents the nth original sample. , It is an indicator function; if the matching rate is greater than a preset threshold... =0.9, then the function A value of true indicates that the copyright of a suspicious model can be quickly verified; otherwise, a value of false indicates a false value. The server performs a higher level of model copyright verification through global watermarking: First, it uses a watermark with a global trigger. Generate samples Extract suspicious models exist Activation features of layers Then, The watermark extraction model is input into the server. The extracted watermark bit sequence is obtained. Finally, calculate Preset global watermark bit sequence The watermark extraction rate between them, through Function to verify model copyright: ， ， Where B is the number of bits; This represents the k-th bit of the extracted watermark bit sequence. This represents the k-th bit of the preset watermark bit sequence. If the watermark extraction rate is greater than the threshold Then the function If the result is true, it means that the copyright of the suspicious model can be quickly verified; otherwise, it is false, indicating a false value. After confirming that the template copyright has been stolen, the malicious client is traced and located: the classification layer parameters are extracted from the classification layer of the suspicious model M, and the signature to be detected of the suspicious model is extracted. The signature to be tested Personalized signatures for each client stored on the server Compare and calculate one by one. and The Hamming distance between them is calculated as follows: ， in, Indicates the signature to be detected Personalized signatures for all clients The Hamming distance between them was calculated, and the client sequence number with the smallest Hamming distance was identified. And if the minimum Hamming distance is less than a preset similarity threshold Then the final judgment is made by the client. It is a malicious client that leaks the model.

6. A copyright protection system based on a federated learning model using feature fusion, implemented by the method described in any one of claims 1 to 5, characterized in that, include: The watermark processing module is used to receive watermark trigger samples and input the watermark trigger samples into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding the black box watermark inside the convolutional neural network model, the local shared model parameters and local classification layer parameters are obtained. The feature extraction module is used to update and aggregate the local shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism, so as to obtain the updated global model shared parameters and personalized classification layer parameters. The global watermark is obtained by extracting watermark features from the updated global model shared parameters based on the feature layer watermark embedding mechanism. The signature verification module is used to perform multi-bit quantization on the personalized classification layer parameters based on the personalized signature extraction mechanism to obtain a set of binary signatures. Copyright verification is performed based on black-box watermarking and global watermarking, and malicious clients are identified based on a set of binary signatures.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 5.

Images