Federal learning model copyright protection method and system based on feature fusion

Abstract

The invention provides a federal learning model copyright protection method and system based on feature fusion, and the method comprises the steps: 1, receiving a watermark triggering sample, and inputting the watermark triggering sample into a pre-established client model for training; 2, updating and aggregating the local shared model parameters and the local classification layer parameters to obtain updated global model shared parameters and personalized classification layer parameters; performing watermark feature extraction on the updated global model sharing parameters based on a feature layer watermark embedding mechanism to obtain a global watermark; 3, performing multi-bit quantization on the personalized classification layer parameters to obtain a set of binary signatures; and performing copyright verification based on the black box watermark and the global watermark, and determining a malicious client based on a set of binary signatures. According to the method, illegal copying and distribution of the model can be effectively restrained, and a key technical support is provided for promoting healthy and sustainable development of a data and model cooperation normal form.

Description

Technical Field

[0001] The present invention relates to the field of federated learning security technology, and in particular to a method and system for copyright protection of a federated learning model based on feature fusion. Background Art

[0002] Federated learning, an emerging distributed machine learning paradigm, allows multiple participants to collaboratively train a shared deep neural network model without directly sharing local raw data. In this framework, each participant trains the model locally, and a central server then aggregates and updates the model parameters uploaded by each participant. This mechanism, which only uploads parameters or gradients rather than raw data, fundamentally avoids the risk of raw data leakage and thus has inherent advantages in protecting data privacy. Consequently, it has been widely used in data-sensitive fields such as healthcare, finance, and the military. However, the distributed training nature of federated learning gives each participant access to and control over their local model, which also introduces new security risks: malicious actors may steal, copy, or illegally distribute the model, posing a significant challenge to the intellectual property protection of the model owner.

[0003] To address the aforementioned risks of model copyright protection and malicious activity tracing, federated learning model watermarking technology has emerged. This technology embeds verifiable watermark information into the model, aiming to verify model copyright and track malicious actors without compromising the model's original performance. Currently, federated learning model watermarking technologies are primarily categorized into client-side model watermarking and server-side model watermarking. Specifically, client-side watermarking schemes leverage the client's contribution to federated learning training to achieve verification. However, when malicious models are stolen by the client, such schemes struggle to track the specific responsible party, potentially allowing the leaker to escape accountability. Server-side watermarking schemes, on the other hand, typically require generating and embedding a unique, personalized watermark for each client to accurately trace the source of the leak. However, this process often requires the server to repeatedly train or fine-tune the model for different clients to inject the watermark, which incurs significant and unnecessary computational overhead and severely limits the system's scalability and the total number of clients it can support. An ideal federated learning copyright protection framework should recognize the dual ownership of both the server and the client as co-participants. On this basis, the server acting as the central coordinator should be granted a higher level of verification authority to achieve authoritative arbitration and accurate tracing of malicious behavior. However, existing solutions generally grant verification rights to a single party and fail to establish an effective multi-party collaborative right confirmation and accountability mechanism. In addition, due to the inherent data heterogeneity problem in the federated learning environment, the data of each client is usually non-independent and identically distributed (Non-IID). As a result, the existing model watermarking technology, in its pursuit of global robustness and accuracy of watermark information, is in fundamental conflict with the goals of federated learning. The global model needs to take into account the adaptability of each client's local data, while the watermark embedding operation often forces the model to sacrifice the fitting accuracy of specific local data. This contradiction directly undermines the fairness principle of federated learning and reduces the performance of the global model for some clients. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the existing technology and provide a copyright protection method and system for a federated learning model based on feature fusion.

[0005] The method comprises the following steps: Step 1: Receive a watermark trigger sample and input the watermark trigger sample into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding a black box watermark inside the convolutional neural network model, local shared model parameters and local classification layer parameters are obtained. Step 2: Based on the personalized parameter delivery mechanism, the local shared model parameters and local classification layer parameters are updated and aggregated respectively to obtain updated global model shared parameters and personalized classification layer parameters; based on the feature layer watermark embedding mechanism, watermark features are extracted from the updated global model shared parameters to obtain a global watermark; In step 3, based on the personalized signature extraction mechanism, the personalized classification layer parameters are quantized into multiple bits to obtain a set of binary signatures. Copyright verification is performed based on the black box watermark and the global watermark, and malicious clients are identified based on the set of binary signatures.

[0006] In step 1, the client's normal training samples and the client's corresponding trigger are obtained, and the client's normal training samples are combined with the client's corresponding trigger to obtain a watermark trigger sample, which specifically includes: For each client's normal training sample ,in represents the original sample, express Corresponding label, Represents normal training samples, each client Receive client-specific local triggers sent by the server , , N represents the total number of clients, that is, the total number of local triggers; by superimposing the client-specific triggers on the original sample Generate watermark trigger samples , Indicates the overlay operation and gives the watermark target label , the watermark trigger sample used by each client With normal training samples Used together to train client models.

[0007] In step 1, the client model is trained on both normal training samples and watermark-triggered samples, and the total training loss is The calculation formula is: , Among them, M(⋅) represents the client model, is the classification loss of the main task, is the classification loss of the watermarking task, is a hyperparameter that balances the loss of the main task and the watermark task, , by minimizing ,While learning the main task, the client model also embeds the ,characteristic responses unique to the corresponding watermark-triggered samples.

[0008] Step 2 includes: the server sends the following to all participating clients: , Local shared model parameters reported in each round Perform weighted average aggregation to obtain the global shared model parameters at the t+1th training round : , in, Indicates the size of the client's local dataset. , where t represents the training round of federated learning, Indicates the last round of federated learning training, for each client Reported local classification layer parameters Perform federated averaging to obtain the global classification layer parameters at the t+1th training round : , Calculate the local classification layer parameter vector for each client and the global classification layer parameter vector The cosine similarity between , and then according to the cosine similarity , dynamically calculate the weights of the client-side personalized classification layer parameters : , , , in, represents the personalized classification layer parameters, and is the preset hyperparameter, =0.3, ; The server packages the aggregated global shared model parameters and the personalized classification layer parameters generated for each client and sends them to the corresponding client. The process of packaging and issuing is as follows: , in, Indicates a delivery operation.

[0009] Step 2 also includes: the server generates a sample Generate a batch of generated samples for watermark activation ; and the server defines a global trigger ,The local trigger is a combination of all positions of the global trigger.,By superimposing the global trigger on the 640 generated samples,,the activation samples with the global trigger are generated. , the server passes the last round of global shared model parameters Extract activation samples separately exist Activation features of the layer and generate samples exist Activation features of the layer : , , in, Represents a layer in the convolutional neural network model, and Represents the operation of extracting features; The server extracts the activation features and Input to watermark extraction model , respectively extract the global watermark bit sequence and any random watermark bit sequence ,At the same time, train the watermark extraction model The goal is to minimize the following contrast loss function : , Among them, the watermark extraction model It is a model pre-established by the server for training global watermarks. It is a fully connected neural network model MLP; contrast loss function The constrained watermark extraction model will carry the global fusion feature Mapping to the global watermark bit sequence , and at the same time suppress the activation features generated by the watermark extraction model without global triggers Mistakenly identified as a legitimate watermark , and mapped to any random watermark bit sequence , represents the binary cross entropy loss, Watermark extraction model The trainable parameters of .

[0010] Step 3 includes: Obtain the personalized classification layer parameters in the convolutional neural network model generated by calculating cosine similarity, dynamic weight allocation, and weighted summation , the classification layer has channels, each channel Each corresponds to a set of parameters , and then calculate the L2 norm of each channel j At the same time, during the training process, the server treats the same channel of each client's local model as a group and saves the minimum norm value of each group. and the maximum norm value , and then use the minimum norm value for each group and the maximum norm value Channel norm Perform interval normalization: , , in, represents the normalized L2 norm of the jth channel of the i-th client; The normalized L2 norm Perform multi-bit quantization to generate a binary signature. First, each client No. The norm of the channels Mapped to the corresponding index The corresponding operations are: , , The number of quantization intervals K is determined by the preset bit parameter bits of each quantization index mapping: , bits=2 or 4, each quantization index A binary string corresponding to a bits position; Then, The operation groups each Map to corresponding The binary string of the bits under The binary strings of channels are concatenated to form a length of The complete binary signature of each client is as follows: , in, Complete personalized classification layer signature for the final client, For string concatenation operation, Indicates encoding the quantization index as bit binary string.

[0011] Step 3 also includes: Each client prepares a set of Dedicated local triggers are superimposed Black box watermark verification set , for each watermark trigger sample in the validation set , input to the suspicious model M, and get the predicted label , and then through Function to verify model copyright: , Among them, the suspicious model M is a model that is set to have malicious clients copying or abusing clients. represents the nth original sample, , Is an indicator function, if the matching rate is greater than the pre-set threshold =0.9, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. The server performs higher-level model copyright verification through global watermarks: First, use a global trigger with The generated samples , extract suspicious models exist Activation features of the layer , then, Input to the watermark extraction model trained on the server , get the extracted watermark bit sequence Finally, calculate Preset global watermark bit sequence The watermark extraction rate between Function to verify model copyright: , , Among them, B=256 is the number of bits; represents the kth bit of the extracted watermark bit sequence, Represents the kth bit of the preset watermark bit sequence, , if the watermark extraction rate is greater than the threshold =0.95, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. After confirming that the template copyright has been stolen, the malicious client is retroactively located: the classification layer parameters are extracted from the classification layer of the suspicious model M, and the signature to be detected of the suspicious model is extracted. , the signature to be detected Personalized signature of each client saved with the server Compare one by one and calculate and The Hamming distance between them is as follows: , in, Indicates the signature to be detected Personalized signatures with all clients The Hamming distance between them is calculated and identified by the client serial number with the smallest Hamming distance. , and if the minimum Hamming distance is less than the preset similarity threshold =0.9, then the client is finally determined to be It is a malicious client that leaks the model.

[0012] The present invention also provides a copyright protection system for a federated learning model based on feature fusion implemented by the method, comprising: A watermark processing module is used to receive watermark trigger samples and input the watermark trigger samples into a pre-established client model for training. The client model adopts a convolutional neural network model, embeds a black box watermark inside the convolutional neural network model, and obtains local shared model parameters and local classification layer parameters; The feature extraction module is used to update and aggregate the local shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism to obtain updated global model shared parameters and personalized classification layer parameters; based on the feature layer watermark embedding mechanism, watermark features are extracted from the updated global model shared parameters to obtain a global watermark; The signature verification module is used to perform multi-bit quantization on the personalized classification layer parameters based on the personalized signature extraction mechanism to obtain a set of binary signatures; perform copyright verification based on black box watermarks and global watermarks, and identify malicious clients based on the set of binary signatures.

[0013] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method.

[0014] The present invention also provides a storage medium storing a computer program or instruction, which executes the steps of the method when the computer program or instruction is run on a computer.

[0015] Beneficial effects: The present invention constructs a technical defense system through a multi-layer collaborative watermark framework, injecting core protection capabilities into the federated learning security ecosystem: in the copyright verification dimension, the black box watermark on the client side and the deep feature watermark on the server side form a complementary verification dimension. The former realizes lightweight local rights confirmation through local triggering samples, and the latter constructs a deep watermark identification with stronger anti-interference ability based on the feature map of the intermediate activation layer of the model. This dual verification mechanism not only maintains the robustness of watermark information when combating common attacks such as model pruning and parameter perturbations, but also maintains its hidden embedded state in the model structure, providing a technically rigid arbitration basis for copyright ownership involving multiple subjects. In the dimension of malicious behavior tracing, a personalized signature watermark with a unique mapping relationship is constructed by deeply mining the differentiated features of the client classification layer parameters. This mechanism breaks through the high-cost tracing model of traditional solutions that rely on repeated training, realizes dynamic binding of watermarks based on the native parameter interaction process of federated learning, establishes a precise mapping relationship of client feature fingerprints, and ensures that in the event of a model leak, accountability for violations can be achieved through parameter feature matching. This invention effectively curbs the illegal copying and distribution of models, aims to strengthen the inherent trust and security foundation of federated learning, and provide key technical support for promoting the healthy and sustainable development of the data and model collaboration paradigm. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flow chart of the method of the present invention.

[0017] Figure 2 This is a schematic diagram of the overall framework of the federated learning watermark embedding of the present invention.

[0018] Figure 3 This is a schematic diagram of the server embedding a global watermark in the present invention.

[0019] Figure 4 This is a schematic diagram of the server extracting the client's personalized classification layer signature.

[0020] Figure 5 This is a schematic diagram of the overall framework of the federated learning copyright verification and tracking of malicious clients in the present invention.

[0021] Figure 6 It is a schematic diagram of the system structure of the present invention.

[0022] Figure 7 This is a graph showing the experimental results of the present invention in resisting pruning attacks on the CIFAR10 dataset.

[0023] Figure 8 This is the experimental result of the present invention against fine-tuning attacks on the CIFAR10 dataset. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above content or other advantages of the present invention will become more clear.

[0025] like Figure 1 As shown, this embodiment provides a copyright protection method for a federated learning model based on feature fusion, including the following steps: Step 1: Receive a watermark trigger sample and input the watermark trigger sample into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding a black box watermark inside the convolutional neural network model, local shared model parameters and local classification layer parameters are obtained. The watermark trigger sample is obtained by obtaining a normal training sample of the client and a trigger corresponding to the client, and combining the normal training sample of the client with the trigger corresponding to the client. The process includes: For each client's normal training sample ,in represents the original sample, express Corresponding label, Represents normal training samples, each client Receive client-specific local triggers sent by the server , , N represents the total number of clients, that is, the total number of local triggers; by superimposing the client-specific triggers on the original sample Generate watermark trigger samples , Indicates the overlay operation and gives the watermark target label , the watermark trigger sample used by each client With normal training samples This step not only perturbs the original sample at the pixel level, but also embeds a unique response pattern in the feature space. Each client only embeds a very small amount (far less than the normal amount of data) and exclusive watermark trigger samples in its local model. Its normal training data samples mix.

[0026] The watermark trigger sample is input into the client model for training, and the local shared model parameters and local classification layer parameters are obtained while embedding the black box watermark inside the convolutional neural network model. The process includes: The client model is trained on both normal training samples and watermark-triggered samples, and its total training loss is composed of the main task loss and watermark loss Weighted combination.

[0027] , Among them, M(⋅) represents the client model, is the classification loss of the main task, is the classification loss of the watermarking task, is a hyperparameter that balances the loss of the main task and the watermark task, , by minimizing ,While learning the main task, the client model also embeds the ,characteristic responses unique to the corresponding watermark-triggered samples.

[0028] Step 2: Based on the personalized parameter delivery mechanism, the local shared model parameters and local classification layer parameters are updated and aggregated to obtain updated global model shared parameters and personalized classification layer parameters; based on the feature layer watermark embedding mechanism, watermark features are extracted from the updated global model shared parameters to obtain a global watermark; The personalized parameter delivery mechanism is used to update and aggregate the local shared model parameters and the local classification layer parameters to obtain updated global model shared parameters and personalized classification layer parameters. The process includes: The server sends a message to all participating clients. , Local shared model parameters reported in each round Perform weighted average aggregation to obtain the global shared model parameters at the t+1th training round : , in, Indicates the size of the client's local dataset. , where t represents the training round of federated learning, Indicates the last round of federated learning training, for each client Reported local classification layer parameters Perform federated averaging to obtain the global classification layer parameters at the t+1th training round : , In order to solve the problem of high heterogeneity of client data in the federated learning environment, if the local classification layer parameters of each client are directly used to extract the classification layer signature, the classification layer parameters often show large individual deviations due to the significant differences in data distribution of each client. This difference will significantly weaken the robustness of the classification layer signature. Specifically, when the model is fine-tuned or subjected to attack disturbances, the classification layer parameters of individual clients are prone to drastic changes, resulting in unstable extracted classification layer signatures, thereby affecting the reliability of traceability judgment and the consistency of watermark verification. To solve the above problems, this method first obtains the global classification layer parameters. , calculate the local classification layer parameter vector of each client and the global classification layer parameter vector The cosine similarity between , and then according to the cosine similarity , dynamically calculate the weights of the client-side personalized classification layer parameters : , , , in, represents the personalized classification layer parameters, and is the preset hyperparameter, =0.3, ; The server packages the aggregated global shared model parameters and the personalized classification layer parameters generated for each client and sends them to the corresponding client. The process of packaging and issuing is as follows: , in, Indicates a delivery operation.

[0029] The feature layer watermark embedding mechanism is used to extract watermark features from the updated global model shared parameters to obtain a global watermark. The process includes: The server passes a sample generator Generate a batch of generated samples for watermark activation ; and the server defines a global trigger ,The local trigger is a combination of all positions of the global trigger. By superimposing the global trigger on the partial generated samples, the activation sample with the global trigger is generated. . In essence, the role of these activation samples and global triggers is to "evoke" the global model's fusion response to the watermark features embedded by all clients in the middle layer. Because in the early training process, the specific watermark features embedded by each client through local triggers have been effectively integrated into the global model parameters, and have formed a significant distinction with other features in the feature space. This not only allows the local features of each client to be completely retained within the global model, but also enables them to be highly identifiable and distinguishable during subsequent activation feature extraction and watermark verification. These activation samples will trigger the coupled response of each client's features in the middle activation layer of the global model, thereby forming a globally unique watermark expression in the feature space. Then, the server uses the last round of global shared model parameters Extract activation samples separately and generate samples exist The activation features of the layer are: , , in, Represents a layer in the convolutional neural network model, and Represents the operation of extracting features; the feature extraction here is not limited to characterizing the generated sample itself, but more importantly reflects the global fusion effect of the client's local watermark features in the feature space after global aggregation.

[0030] The server extracts the activation features and Input to watermark extraction model , respectively extract the global watermark bit sequence b and any random watermark bit sequence , while training The goal is to minimize the following contrast loss function : , Among them, the watermark extraction model It is a model pre-established by the server for training global watermarks. It is a fully connected neural network model MLP; contrast loss function The constrained watermark extraction model will carry the global fusion feature Mapping to the global watermark bit sequence , and at the same time suppress the activation features generated by the watermark extraction model without global triggers Mistakenly identified as a legitimate watermark , and mapped to any random watermark bit sequence , represents the binary cross entropy loss, Watermark extraction model The trainable parameters of .

[0031] Step 3: Based on the personalized signature extraction mechanism, the personalized classification layer parameters are quantized into multiple bits to obtain a set of binary signatures. Copyright verification is performed based on the black box watermark and the global watermark, and malicious clients are identified based on the set of binary signatures. The personalized signature extraction mechanism is based on which the personalized classification layer parameters are quantized into multiple bits to obtain a set of binary signatures. The process includes: In the last round of federated learning, the server classification layer signature extraction is performed. First, the personalized classification layer parameters in the convolutional neural network model generated by calculating cosine similarity, dynamic weight allocation and weighted summation are obtained. , the classification layer has channels, each channel Each corresponds to a set of parameters , and then calculate the L2 norm of each channel j At the same time, during the training process, the server treats the same channel of each client's local model as a group and saves the minimum norm value of each group. and the maximum norm value , and then use the minimum norm value for each group and the maximum norm value Channel norm Perform interval normalization: , , in, represents the normalized L2 norm of the jth channel of the i-th client; The normalized L2 norm Perform multi-bit quantization to generate a binary signature. First, each client No. The norm of the channels Mapped to the corresponding index The corresponding operations are: , , The number of quantization intervals K is determined by the preset bit parameter bits of each quantization index mapping: , bits=2 or 4, each quantization index A binary string corresponding to a bits position; Then, The operation groups each Map to corresponding The binary string of the bits under The binary strings of channels are concatenated to form a length of The complete binary signature of each client is as follows: , in, Complete personalized classification layer signature for the final client, For string concatenation operation, Indicates encoding the quantization index as The copyright verification is performed based on the black box watermark, the set of binary signatures and the global watermark, and the malicious client is determined based on the verification result.

[0032] The copyright verification based on black box watermarks and global watermarks and the identification of malicious clients based on a set of binary signatures include the following process: Each client prepares a set of Dedicated local triggers are superimposed Black box watermark verification set , for each watermark trigger sample in the validation set , input to the suspicious model M, and get the predicted label , and then through Function to verify model copyright: , Among them, the suspicious model M is a model that is set to have malicious clients copying or abusing clients. represents the nth original sample, , Is an indicator function, if the matching rate is greater than the pre-set threshold =0.9, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. The server performs higher-level model copyright verification through global watermarks: First, use a global trigger with The generated samples , extract suspicious models exist Activation features of the layer , then, Input to the watermark extraction model trained on the server , get the extracted watermark bit sequence Finally, calculate Preset global watermark bit sequence The watermark extraction rate between Function to verify model copyright: , , Among them, B=256 is the number of bits; represents the kth bit of the extracted watermark bit sequence, Represents the kth bit of the preset watermark bit sequence, , if the watermark extraction rate is greater than the threshold =0.95, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. After confirming that the template copyright has been stolen, the malicious client is retroactively located: the classification layer parameters are extracted from the classification layer of the suspicious model M, and the signature to be detected of the suspicious model is extracted. , the signature to be detected Personalized signature of each client saved with the server Compare one by one and calculate and The Hamming distance between them is as follows: , in, Indicates the signature to be detected Personalized signatures with all clients The Hamming distance between them is calculated and identified by the client serial number with the smallest Hamming distance. , and if the minimum Hamming distance is less than the preset similarity threshold =0.9, then the client is finally determined to be It is a malicious client that leaks the model.

[0033] Specifically, the present invention will be further described below through examples: To verify the effectiveness of the proposed feature fusion watermark embedding scheme in a federated learning environment, this example designed an image classification task and selected two different model architectures: ConvNet and ResNet18 to train the MNIST and CIFAR10 datasets, respectively. Taking CIFAR10 and ResNet-18 as examples, ResNet18 is a deep residual network that effectively solves the gradient vanishing problem in deep neural network training by introducing residual connections, and demonstrates excellent performance in more complex image recognition tasks. The MNIST dataset contains 60,000 images of handwritten digits, of which 60,000 are used for training and 10,000 for testing. This dataset has 10 categories, and the effectiveness of the method was verified through this example.

[0034] The following combination Figure 2 A detailed description of the black box watermark embedded in the client model: The first step is that in each round of training, the client receives the global shared model parameters sent by the server. and personalized classification layer parameters At the same time, the client uses the local trigger sent by the server to build a watermark trigger sample .

[0035] In the second step, the client model is also training samples normally. and watermark trigger samples During training, while learning the main task, a black-box watermark is embedded within the client model. This local feature watermark embedding ensures that each client's watermark features are an integral component of the global watermark. After training, the client transmits the updated global shared model parameters and personalized classification layer parameters back to the server.

[0036] The following combination Figure 2 The server's personalized parameter delivery mechanism is described in detail: On the server side, in order to achieve accurate tracing of malicious leaking clients in a federated learning environment, the present invention introduces a personalized parameter distribution mechanism in the model parameter distribution stage. This mechanism introduces the idea of ​​personalized classification layer parameters in personalized federated learning (such as FedPer and FedRep), and innovatively applies it to the federated learning tracing scenario of heterogeneous data. Specifically, the server not only obtains the global shared model parameters based on the standard federated aggregation, but also performs secondary aggregation and dynamic differentiated distribution of the classification layer parameters of each client. Through dynamic weight allocation, customized parameter delivery for the personalized characteristics of each client is achieved. Deep coupling of personalized federated learning with model copyright verification and accurate tracking of malicious clients is achieved. While effectively improving the local performance and adaptability of each client model, it can generate a unique and highly discriminative signature based on the differentiated distributed personalized classification layer parameters, thereby providing technical guarantees for subsequent model tracing and copyright ownership. The overall process is divided into the following two steps: The first step is to perform regular federation aggregation on the shared model parameters reported by all clients. The aggregation algorithm used by the server updates the global shared model parameters. Secondly, for the key step of personalizing model parameters, the server dynamically adjusts the weight of each in the final personalized classification layer parameters by calculating the cosine similarity of the global and each client's classification layer parameters.

[0037] In the second step, using these adjusted weights, the server generates a new set of personalized model parameters for each client. Finally, the global shared model parameters and the adjusted personalized classification layer parameters are sent to different clients.

[0038] The following combination Figure 2 and Figure 3 A detailed description of server-embedded global watermark: In the first step, after the entire training process of federated learning is completed, the server uses a sample generator Generate a batch of generated samples for watermark activation , and the server defines a global trigger ,The local trigger is a combination of all positions of the global trigger. By superimposing the global trigger on the partial generated samples, the activation sample with the global trigger is generated. , the server passes the last round of global shared model parameters Extract activation samples separately and generate samples exist The activation features of the layer.

[0039] In the second step, the server inputs the above feature maps into the watermark extraction model, and optimizes the extraction parameters through back propagation, so that it can accurately project the global activation features into the predefined binary watermark bit string. After training convergence, the server has the ability to reliably extract the global watermark from the fused features.

[0040] The following combination Figure 2 and Figure 4 A detailed description of how the server extracts the client's personalized classification layer signature: The first step is to obtain the personalized classification layer parameters generated by calculating cosine similarity, dynamic weight allocation and weighted summation in the last round of training. , and then calculate the L2 norm of each channel j of the personalized classification layer parameter At the same time, during the training process, the server treats the same channel of each client as a group and saves the minimum norm value of each group and the maximum norm value , and then each group uses these two values ​​to calculate the channel L2 norm Perform interval normalization to get Client's Channel normalized parameters .

[0041] The second step is to normalize the norm of each channel Perform multi-bit quantization to generate a binary signature. First, each client No. The norm of the channels Mapped to the corresponding index Down and through The operation groups each Map to corresponding Next A binary string of bits.

[0042] Step 3: Each client will The binary strings of channels are concatenated to form a length of The complete binary signature of the client, and finally the server obtains a complete personalized classification layer signature of the client .

[0043] The following combination Figure 5 The process of verifying copyright and tracking malicious clients in federated learning is described in detail: The first step is that the client triggers the sample through a dedicated watermark and Perform local verification. If the matching rate is greater than the pre-set threshold , then rapid verification of model attribution is achieved.

[0044] In the second step, the server acts as a central coordinator and has a higher level of model copyright verification authority. When the server obtains a suspected leaked model, it will start global watermark verification. Specifically, the server will extract the suspicious model exist Activation features of the layer , then, Input to the watermark extraction model trained on the server , get the extracted watermark bit sequence . Then, through the verification function Determine whether the matching rate is greater than the pre-set threshold , it indicates that the model may have been tampered with or there is a risk of copyright leakage.

[0045] In the third step, after the server’s feature layer watermark verification reveals the copyright risk, in order to accurately lock the source of the leak, the server will further use the personalized classification layer signature to trace the source. The server extracts the classification layer personalized signature from the suspicious model. , then, using the tracking function Calculate, if the minimum Hamming distance is less than the preset similarity threshold , then the client is finally determined It is a malicious client that leaks the model.

[0046] like Figure 2 As shown in the figure, the watermark embedding process begins with all clients (client 1, client 2…client N) receiving local triggers from the server during each round of training, and training the model based on their own local data. While embedding the black box watermark inside the model, the local shared model parameters and local classification layer parameters are obtained and uploaded to the server. Secondly, the server first updates the aggregate model parameters, calculates the personalized parameters of the classification layer in the second step, and sends the personalized model parameters in the third step, so that the client obtains different model parameters. Finally, the server performs feature layer watermark embedding and personalized signature extraction for subsequent server model copyright verification and tracking of malicious clients.

[0047] like Figure 3 As shown in the figure, the feature layer watermark embedding process first superimposes the global trigger to the generated sample generated by the partial generator Generate activation samples with global triggers , the server passes the last round of shared model parameters Extract activation samples separately and generate samples exist Activation features of the layer , ;Then the server extracts the activation features and Input to the watermark extraction model, train the watermark extraction model to extract the global watermark bit sequence and any random watermark bit sequence , while optimizing the parameters of the watermark extraction model; finally, the watermark extraction model can stably output the global watermark bit sequence according to the activation features .

[0048] like Figure 4 As shown in the figure, the process of personalized signature extraction is that the server first obtains the personalized classification layer parameters generated by all clients (client 1, client 2...client N) by calculating cosine similarity, dynamic weight allocation and weighted summation, and then calculates the L2 norm of the personalized classification layer parameters after normalization for each channel j. , , , Client 1: , , Client 2: , ,…, client N: , ; Then, the normalized norm of each channel Perform binary multi-bit quantization to generate a binary signature (client 1: 00, 01, client 2: 01, 11, ..., client N: 11, 00). Finally, concatenate the binary strings of all channels of each client to form a complete binary signature, where: , ,…, Finally, a complete personalized classification layer signature is provided for each client.

[0049] like Figure 5 As shown in the figure, the process of model copyright verification and tracking malicious clients includes the following steps: First, each client triggers a sample through a unique watermark, and can independently perform client verification and declare the corresponding ownership of the model. Step 1: Each client prepares a set of Dedicated local triggers are superimposed Black box watermark verification set , for each watermark trigger sample in the validation set , input to the suspicious model , get the predicted label , and then through Function to verify model copyright: , Among them, the suspicious model It is a model that is designed to allow malicious clients to copy or abuse the client. Indicates the The original samples, , Is an indicator function, if the matching rate is greater than the pre-set threshold =0.9, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. Step 2: The server performs higher-level model copyright verification through global watermarks: First, use a global trigger The generated samples , extract suspicious models exist Activation features of the layer , then, Input to the watermark extraction model trained on the server , get the extracted watermark bit sequence Finally, calculate Preset global watermark bit sequence The watermark extraction rate between Function to verify model copyright: , , Among them, B=256 is the number of bits; Represent the kth bit of the extracted and preset watermark bit sequence, , if the watermark extraction rate is greater than the threshold =0.95, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. After confirming that the template copyright has been stolen, trace back to locate the malicious client: Extract the classification layer parameters from the classification layer and extract the signature to be detected of the suspicious model , the signature to be detected Personalized signature of each client saved with the server Compare one by one and calculate and The Hamming distance between them is as follows: , in, Indicates the signature to be detected Personalized signatures with all clients The Hamming distance between them is calculated and identified by the client serial number with the smallest Hamming distance. , and if the minimum Hamming distance is less than the preset similarity threshold =0.9, then the client is finally determined to be It is a malicious client that leaks the model.

[0050] Experimental Setup: Datasets and Models: This method selects two different model architectures: ConvNet and ResNet18, trained on the MNIST and CIFAR10 datasets, respectively. When simulating a federated learning scenario, the default number of clients, N, is set to 50. 10% of the clients are randomly selected each round to participate in local updates. Each client performs five local training iterations, using a batch size of 256. Each client uses stochastic gradient descent (SGD) for parameter updates, with an initial learning rate of 0.1. To construct a non-IID data distribution, a Dirichlet distribution (parameter α = 0.1) is used to split the data across clients to ensure data heterogeneity and enhance the real-world relevance of the experiment.

[0051] Evaluation Metrics: This paper uses the main task accuracy to evaluate the classification accuracy of the model on the original test dataset; the black-box watermark detection rate to evaluate the accuracy of the client-side watermark model on watermark trigger samples; and the white-box watermark detection rate to evaluate the extraction rate of the server-embedded global watermark. The watermark collision rate is used to evaluate the discriminability of the classification layer signature, and the watermark tracking rate is used to evaluate the probability of the classification layer watermark tracking a malicious client.

[0052] Fidelity Experiment: To evaluate the impact of our invention on the model's primary task performance, a good watermark model should achieve similar primary task accuracy to the unwatermarked model. Table 1 shows the primary task accuracy of our invention compared to the unwatermarked method under different client numbers and datasets. On both datasets, the primary task accuracy of our invention under different client numbers is consistently close to or slightly higher than the unwatermarked method. This result demonstrates that our invention not only works effectively without significantly affecting the primary task accuracy, but can also improve the model's primary task accuracy under certain settings.

[0053] Table 1

[0054]

[0055] Non-IID experiment: In order to evaluate the performance of the present invention in different non-IID scenarios, the present invention simulates four different Dirichlet distribution parameters to verify the main task accuracy, black box watermark detection rate, white box watermark extraction rate, watermark collision rate and watermark tracking rate of the present invention. Under different α values, the unevenness of the data increases as the α value decreases, which directly affects the performance of the model in federated learning. As α increases, the main task accuracy, black box watermark detection rate and white box watermark extraction rate are all controlled within a relatively small range, and the overall fluctuation is within a small interval. Specifically, the fluctuation range of the main task accuracy is 99.34% to 97.95%, with an overall fluctuation of 1.39%. The black box watermark detection rate fluctuates from 99.94% to 97.55%, with an overall fluctuation of 2.39%; the white box watermark extraction rate fluctuates from 96.88% to 96.65%, with an overall fluctuation of 0.23%.

[0056] However, the watermark collision rate remains zero for all values ​​of α, indicating that the classification-level signatures maintain high distinctiveness across different values ​​of α. Even when the data distribution is more non-IID, the classification-level signatures remain very dissimilar. The watermark tracking rate exhibits excellent tracking capabilities at α = 0.05 and α = 0.1, reaching 99% and 98.58%, respectively. This demonstrates that the classification-level signatures maintain good tracking capabilities even in cases of highly uneven data distribution, as shown in Table 2.

[0057] Table 2

[0058]

[0059] Robustness experiment: In order to resist the attacker's pruning and fine-tuning attacks on the watermark model, so that the watermark in the model is removed, this paper evaluates the robustness of the watermark model under the two attacks of pruning and fine-tuning. Figure 7 As shown in Figure 2, the pruning experiment on the CIFAR10 dataset shows that as the pruning rate increases, the various indicators are basically stable. When the pruning rate increases to 0.5, the black box watermark detection rate is above 90%, the white box watermark extraction rate is still greater than 95%, and the watermark tracking rate remains basically unchanged. Figure 8 As shown in the figure, fine-tuning experiments conducted on the CIFAR10 dataset show that the fluctuations in the main task accuracy and the white-box watermark extraction rate are both within a very small range. When the number of fine-tuning rounds is increased to 50, the black-box watermark detection rate and watermark tracking rate are still greater than 95%. These results show that although pruning and fine-tuning will affect the watermark model's performance to a certain extent, neither can completely remove the watermark from the model, demonstrating the robustness of the present invention under both attacks.

[0060] like Figure 6 As shown, based on the above method, this embodiment also provides a copyright protection system for a federated learning model based on feature fusion, including: A watermark processing module is used to receive watermark trigger samples and input the watermark trigger samples into a pre-established client model for training. The client model adopts a convolutional neural network model, embeds a black box watermark inside the convolutional neural network model, and obtains local shared model parameters and local classification layer parameters; The feature extraction module is used to update and aggregate the local shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism to obtain updated global model shared parameters and personalized classification layer parameters; based on the feature layer watermark embedding mechanism, watermark features are extracted from the updated global model shared parameters to obtain a global watermark; The signature verification module is used to perform multi-bit quantization on the personalized classification layer parameters based on the personalized signature extraction mechanism to obtain a set of binary signatures; perform copyright verification based on black box watermarks and global watermarks, and identify malicious clients based on the set of binary signatures.

[0061] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0062] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the above-described method. The storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0063] This invention provides a method and system for copyright protection in a federated learning model based on feature fusion. There are numerous methods and approaches for implementing this technical solution. The foregoing description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the invention, and such improvements and modifications are also within the scope of protection of the invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A copyright protection method for a federated learning model based on feature fusion, characterized in that: The following steps are involved: Step 1: Receive a watermark trigger sample and input the watermark trigger sample into a pre-established client model for training. The client model adopts a convolutional neural network model. While embedding a black box watermark inside the convolutional neural network model, local shared model parameters and local classification layer parameters are obtained. Step 2: Based on the personalized parameter delivery mechanism, the local shared model parameters and local classification layer parameters are updated and aggregated to obtain updated global model shared parameters and personalized classification layer parameters; Based on the feature layer watermark embedding mechanism, watermark features are extracted from the updated global model shared parameters to obtain the global watermark; Step 3: Based on the personalized signature extraction mechanism, the personalized classification layer parameters are quantized into multiple bits to obtain a set of binary signatures; Copyright verification is performed based on black-box watermarks and global watermarks, and malicious clients are identified based on a collection of binary signatures.

2. The method according to claim 1, characterized in that In step 1, the client's normal training samples and the client's corresponding trigger are obtained, and the client's normal training samples are combined with the client's corresponding trigger to obtain a watermark trigger sample, which specifically includes: For each client's normal training sample ,in represents the original sample, express Corresponding label, Represents normal training samples, each client Receive client-specific local triggers sent by the server , , N represents the total number of clients, that is, the total number of local triggers; by superimposing the client-specific triggers on the original sample Generate watermark trigger samples , Indicates the overlay operation and gives the watermark target label , the watermark trigger sample used by each client With normal training samples Used together to train client models.

3. The method according to claim 2, characterized in that In step 1, the client model is trained on both normal training samples and watermark-triggered samples, and the total training loss is The calculation formula is: ， Among them, M(⋅) represents the client model, is the classification loss of the main task, is the classification loss of the watermarking task, is a hyperparameter that balances the loss of the main task and the watermark task, , by minimizing ,While learning the main task, the client model also embeds the ,characteristic responses unique to the corresponding watermark-triggered samples.

4. The method according to claim 3, characterized in that Step 2 includes: the server sends the following to all participating clients: , Local shared model parameters reported in each round Perform weighted average aggregation to obtain the global shared model parameters at the t+1th training round : ， in, Indicates the size of the client's local dataset. , where t represents the training round of federated learning, Indicates the last round of federated learning training, for each client Reported local classification layer parameters Perform federated averaging to obtain the global classification layer parameters at the t+1th training round : ， Calculate the local classification layer parameter vector for each client and the global classification layer parameter vector The cosine similarity between , and then according to the cosine similarity , dynamically calculate the weights of the client-side personalized classification layer parameters : ， ， ， in, represents the personalized classification layer parameters, and is the preset hyperparameter; The server packages the aggregated global shared model parameters and the personalized classification layer parameters generated for each client and sends them to the corresponding client. The process of packaging and issuing is as follows: ， in, Indicates a delivery operation.

5. The method according to claim 4, characterized in that Step 2 also includes: the server generates a sample Generate a batch of generated samples for watermark activation ; and the server defines a global trigger ,The local trigger is a combination of all positions of the global trigger. By superimposing the global trigger on the generated sample, an activation sample with the global trigger is generated. , the server passes the last round of global shared model parameters Extract activation samples separately exist Activation features of the layer and generate samples exist Activation features of the layer : ， ， in, Represents a layer in the convolutional neural network model, and Represents the operation of extracting features; The server extracts the activation features and Input to watermark extraction model , respectively extract the global watermark bit sequence and any random watermark bit sequence ,At the same time, train the watermark extraction model The goal is to minimize the following contrast loss function : ， Among them, the watermark extraction model It is a model pre-established by the server for training global watermarks. It is a fully connected neural network model MLP; contrast loss function The constrained watermark extraction model will carry the global fusion feature Mapping to the global watermark bit sequence , and at the same time suppress the activation features generated by the watermark extraction model without global triggers Mistakenly identified as a legitimate watermark , and mapped to any random watermark bit sequence , represents the binary cross entropy loss, Watermark extraction model The trainable parameters of .

6. The method according to claim 5, characterized in that Step 3 includes: Obtain the personalized classification layer parameters in the convolutional neural network model generated by calculating cosine similarity, dynamic weight allocation, and weighted summation , the classification layer has channels, each channel Each corresponds to a set of parameters , and then calculate the L2 norm of each channel j At the same time, during the training process, the server treats the same channel of each client's local model as a group and saves the minimum norm value of each group. and the maximum norm value , and then use the minimum norm value for each group and the maximum norm value Channel norm Perform interval normalization: ， ， in, represents the normalized L2 norm of the jth channel of the i-th client; The normalized L2 norm Perform multi-bit quantization to generate a binary signature. First, each client No. The norm of the channels Mapped to the corresponding index The corresponding operations are: ， ， The number of quantization intervals K is determined by the preset bit parameter bits of each quantization index mapping: , bits=2 or 4, each quantization index A binary string corresponding to a bits position; Then, The operation groups each Map to corresponding The binary string of the bits under The binary strings of channels are concatenated to form a length of The complete binary signature of each client is as follows: ， in, Complete personalized classification layer signature for the final client, For string concatenation operation, Indicates encoding the quantization index as bit binary string.

7. The method according to claim 6, characterized in that Step 3 also includes: Each client prepares a set of Dedicated local triggers are superimposed Black box watermark verification set , for each watermark trigger sample in the validation set , input to the suspicious model M, and get the predicted label , and then through Function to verify model copyright: ， Among them, the suspicious model M is a model that is set to have malicious clients copying or abusing clients. represents the nth original sample, , Is an indicator function, if the matching rate is greater than the pre-set threshold =0.9, then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. The server performs higher-level model copyright verification through global watermarks: First, use a global trigger with The generated samples , extract suspicious models exist Activation features of the layer , then, Input to the watermark extraction model trained on the server , get the extracted watermark bit sequence Finally, calculate Preset global watermark bit sequence The watermark extraction rate between Function to verify model copyright: ， ， Where B is the number of bits; represents the kth bit of the extracted watermark bit sequence, Represents the kth bit of the preset watermark bit sequence, , if the watermark extraction rate is greater than the threshold , then the function If the judgment is true, it means that the copyright of the suspicious model is quickly verified, otherwise it is false. After confirming that the template copyright has been stolen, the malicious client is retroactively located: the classification layer parameters are extracted from the classification layer of the suspicious model M, and the signature to be detected of the suspicious model is extracted. , the signature to be detected Personalized signature of each client saved with the server Compare one by one and calculate and The Hamming distance between them is as follows: ， in, Indicates the signature to be detected Personalized signatures with all clients The Hamming distance between them is calculated and identified by the client serial number with the smallest Hamming distance. , and if the minimum Hamming distance is less than the preset similarity threshold , then the client is finally determined It is a malicious client that leaks the model.

8. A copyright protection system for a federated learning model based on feature fusion implemented by the method according to any one of claims 1 to 7, characterized in that: include: A watermark processing module is used to receive watermark trigger samples and input the watermark trigger samples into a pre-established client model for training. The client model adopts a convolutional neural network model, embeds a black box watermark inside the convolutional neural network model, and obtains local shared model parameters and local classification layer parameters; The feature extraction module is used to update and aggregate the local shared model parameters and local classification layer parameters based on the personalized parameter distribution mechanism to obtain updated global model shared parameters and personalized classification layer parameters; Based on the feature layer watermark embedding mechanism, watermark features are extracted from the updated global model shared parameters to obtain the global watermark; A signature verification module is used to perform multi-bit quantization on the personalized classification layer parameters based on the personalized signature extraction mechanism to obtain a set of binary signatures; Copyright verification is performed based on black-box watermarks and global watermarks, and malicious clients are identified based on a collection of binary signatures.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 7 are executed.

Images