A federated learning copyright protection method and system based on mutual information

Abstract

The present invention discloses a mutual information-based federated learning copyright protection method and system, which belongs to the field of federated learning security. The method includes: sampling noise from a latent space and inputting it into a generator to generate a real sample dataset #imgabs0#; generating a watermark trigger sample set #imgabs3# based on the real sample dataset #imgabs1# using a trigger generation network #imgabs2#; embedding a dormant watermark by maximizing the mutual information between the gradients of the global model #imgabs4# in the real data sample set #imgabs5# and the watermark trigger sample set #imgabs6#; fine-tuning the global model #imgabs9# using the real sample dataset #imgabs7# and the watermark trigger sample set #imgabs8# to extract labels; and comparing the extracted labels with the target labels; if the two are consistent, the ownership of the model is verified. By embedding the dormant watermark directly into the model, rather than the traditional explicit model watermark, it can effectively prevent malicious users from detecting and deleting the watermark before stealing the model. Furthermore, the watermark can be activated after the user fine-tunes the model.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning security, and specifically relates to a federated learning copyright protection method and system based on mutual information. Background Art

[0002] With the continuous advancement of science and technology, data privacy protection has become a focus of attention in today's digital age. Federated learning, an innovative distributed learning framework, has been widely used in multiple key areas due to its unique privacy protection features. For example, in the banking loan business, different financial institutions can use federated learning to jointly train more accurate credit assessment models without sharing sensitive raw customer data, thereby improving the accuracy and efficiency of loan approvals. In the field of medical image analysis, medical institutions can jointly conduct model training to improve the accuracy of disease diagnosis while protecting patient privacy. In object detection tasks, different companies or research teams can also achieve collaborative data utilization and enhance model performance through federated learning.

[0003] Federated learning cleverly resolves the conflict between data privacy and model training by allowing participants to jointly train models without sharing original data. This promotes in-depth collaboration between different organizations and entities, improves the comprehensiveness and robustness of models, and effectively protects user privacy. However, in the distributed environment of federated learning, malicious clients may compromise system security through various means, leading to model leaks and threatening the rights and interests of model owners, system stability, and user privacy. Model theft attacks are a serious security threat. Attackers may illegally obtain copies of models and use them for commercial competition or other malicious purposes, thereby infringing on model ownership. To address this issue, model watermarking technology has been proposed and has become an effective solution.

[0004] The core idea of model watermarking technology is to embed an imperceptible marker into the model, similar to adding a watermark to a file or image. When a model is illegally copied and used, researchers can verify the original ownership of the model by detecting this watermark. Based on the location and timing of watermark embedding, watermarking technology can generally be categorized into client-side watermarking and server-side watermarking. Client-side watermarking embeds the watermark during the client-side training process, marking the model early in training and enhancing the robustness of the watermark. Server-side watermarking, on the other hand, is added by a central server after model aggregation, ensuring the consistency and authority of the watermark. In the complex collaborative environment of federated learning, servers often require clients to provide partial data to enhance the robustness of the watermark. However, this practice conflicts with the privacy protection principles of federated learning. Furthermore, model collaborators often fine-tune the trained model to better suit specific application requirements, which can weaken the watermark's characteristics. Furthermore, malicious collaborators can use specific means to detect whether a stolen model contains a watermark and, if so, attempt to remove it. Since federated learning involves multiple collaborators and there is a risk of malicious users destroying the watermark through collusion attacks, which may significantly reduce the watermark extraction rate. Therefore, a federated learning copyright protection method and system based on mutual information is proposed. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a federated learning copyright protection method and system based on mutual information, which solves the problems in the existing technology.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A mutual information-based federated learning copyright protection method includes the following steps:

[0008] Sample noise from the latent space and input it into the generator to generate a real sample dataset ;

[0009] Based on the real sample dataset , using triggers to generate networks , generate watermark trigger sample set ;

[0010] By maximizing the global model In the real data sample set and watermark-triggered sample set The mutual information between gradients is used to complete the embedding of dormant watermarks;

[0011] Using real sample data sets and watermark trigger sample set For the global model Fine-tune and extract labels; compare the extracted labels with the target labels; if the two are consistent, the ownership of the model is proved.

[0012] Furthermore, a real sample dataset is generated The process is:

[0013] S11, sampling noise from the potential space that obeys the normal distribution to obtain the input vector;

[0014] S12, the generator maps the input vector to a specific space and finally generates a real sample data set ;

[0015] S13, update the generator through backpropagation .

[0016] Furthermore, the generator is composed of a three-layer fully connected neural network.

[0017] Furthermore, a watermark trigger sample set is generated The process is:

[0018] S21, define the trigger generation network and initialize the trigger characteristics , train the trigger generation network , designed to optimize trigger characteristics ,Mining the features of target tags, the target tags are tags pre-defined and selected by the server;

[0019] S22, the real sample data set Input trigger generation network , output results and hyperparameters Multiply together to get the optimized trigger characteristics And the watermark trigger sample set is ;in is a data sample in the input real sample data set, is the target label.

[0020] Furthermore, the global model The server aggregates the local models uploaded by each client to obtain:

[0021]

[0022] in, Is the client Model, Is the client data, is the collection of all client data; N is the total number of clients.

[0023] Furthermore, the dormant watermark embedding step is:

[0024] S31, using a real sample dataset with correct labels For the global model Implement fine-tuning processing;

[0025] S32, based on the fine-tuned global model , a joint update operation is performed on the current embedding parameters, aiming to maximize the global model In the real data sample set and watermark-triggered sample set Mutual information between gradients;

[0026] S33, using watermark to trigger sample set For the global model Make fine adjustments.

[0027] A mutual information-based federated learning copyright protection system, comprising:

[0028] Real sample data generation module: samples noise from the potential space and inputs it into the generator to generate a real sample data set ;

[0029] Watermark trigger sample generation module: based on the real sample data set , using triggers to generate networks , generate watermark trigger sample set ;

[0030] Watermark embedding module: by maximizing the global model In the real data sample set and watermark-triggered sample set The mutual information between gradients is used to complete the embedding of dormant watermarks;

[0031] And, verification module: using real sample data sets and watermark trigger sample set For the global model Fine-tune and extract labels; compare the extracted labels with the target labels; if the two are consistent, the ownership of the model is proved.

[0032] A computer storage medium stores a readable program, which, when executed by a processor, can execute the above-mentioned federated learning copyright protection method based on mutual information.

[0033] An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0034] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned mutual information-based federated learning copyright protection method.

[0035] A computer program product includes computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the above-mentioned mutual information-based federated learning copyright protection method.

[0036] Beneficial effects of the present invention:

[0037] 1. This invention embeds a dormant watermark directly into the model, rather than the traditional explicit model watermark. This effectively prevents malicious users from detecting and removing the watermark before stealing the model, while also allowing the watermark to be activated after the user fine-tunes the model.

[0038] 2. The embedding of the watermark of the present invention is divided into three steps to minimize the impact on model performance. To achieve this goal, the server uses two generators: one is used to generate a real data sample set that complies with the privacy protection principle of federated learning, and the other is used to generate a watermark trigger sample set, thereby ensuring that the impact of the watermark trigger sample set on model performance is minimized.

[0039] 3. In the watermark embedding process, the concealment of the watermark is enhanced by maximizing the mutual information, and a multi-layer perceptron (MLP) network is adopted to improve the robustness of watermark embedding.

[0040] 4. The method of the present invention significantly enhances the model's defense capabilities against various watermark attacks, including model fine-tuning attacks, pruning attacks, watermark coverage attacks, and model merging attacks; and the present invention provides an effective way to prove the ownership of the model, further enhancing the security and reliability of the federated learning system; through this mutual information-based federated learning copyright protection method, the watermark's concealment and the overall system performance are optimized, while enhancing the model's practicality and security in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 It is a schematic diagram of the process of generating a real sample data set of the present invention;

[0043] Figure 2 It is a schematic diagram of the watermark trigger sample set generation process of the present invention;

[0044] Figure 3 This is a flow chart of the copyright protection method of federated learning based on mutual information of the present invention;

[0045] Figure 4 It is a schematic diagram of the model copyright verification of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] like Figure 3 As shown, a mutual information-based federated learning copyright protection method includes the following steps:

[0048] S1, samples noise from the latent space and inputs it into the generator to generate a real sample dataset ;

[0049] like Figure 1 As shown, generate a real sample data set The process is:

[0050] S11, from the normal distribution The noise is sampled in the latent space of , ,in represents the mean, represents the standard deviation; another input label of the generator is ;

[0051] S12, the generator consists of a three-layer fully connected neural network; the generator takes the input vector Map the output to a specific space. The specific mapping process is as follows:

[0052]

[0053] in, 、 、 They are the three weight matrices of the fully connected layer, 、 、 is the corresponding bias term, Represents the result after applying the Relu activation function, Represents the output of the generator , output and input Composing a real sample data set During training It is input into the classification model and the output result of the model output layer is calculated. , and use the mean square error (MSE) to measure the difference between the output of the model output layer and the target one-hot encoding; the loss function Expressed as:

[0054]

[0055] in, represents the one-hot encoding of the label, is the loss of the generated data sample labels, Represents the output of the i-th data, and n represents the total number of data.

[0056] S13, update the generator through backpropagation :

[0057]

[0058] in, represents the learning rate, Represents the loss function Gradient with respect to the parameters.

[0059] S2, a real sample dataset generated based on S1 , using triggers to generate networks , generate watermark trigger sample set ;

[0060] like Figure 2 As shown, generate watermark trigger sample set The process is:

[0061] S21, define the trigger generation network , and initialize the trigger feature , train the trigger generation network Designed to optimize trigger characteristics , mining watermark trigger sample set labels Features, are tags predefined and selected by the server;

[0062] The features contained in the target label are mined by solving the following optimization tasks:

[0063]

[0064] in, is a hyperparameter, is the global model, To minimize the real sample data set losses, is the loss value of all samples, is the loss of a single sample, are tags predefined and selected by the server.

[0065] Given the obtained It increases the probability that the model will classify any data as the target label, so it can be regarded as the feature of the target label learned by the model; In addition, in order to adjust right The impact of ,in is a hyperparameter;

[0066] S22, the real sample data set Input trigger generation network , output results and hyperparameters Multiply, that is ; The watermark trigger sample set is ;

[0067] In this embodiment, the trigger generates a network It consists of an encoder and a decoder. The encoder contains four (convolution layer + normalization layer + Relu) blocks. The decoder contains three (deconvolution layer + normalization layer + Relu) blocks, and finally a (deconvolution layer + normalization layer + Tanh) block. In order to control the trigger generation network The output strength of , using Tanh as the activation function of the last layer;

[0068] S3, by maximizing the global model In the real data sample set and watermark-triggered sample set The mutual information between gradients is used to complete the embedding of dormant watermarks;

[0069] like Figure 4 As shown, in the federated learning scenario, the server aggregates the local models uploaded by each client to obtain a global model. :

[0070]

[0071] in, is the global model, Is the client Model, Is the client data, is the collection of all client data; N is the total number of clients.

[0072] The steps of dormant watermark embedding are:

[0073] S31, obtain the global model , using a dataset of real samples with correct labels For the global model Implement fine-tuning processing;

[0074] S32, based on the fine-tuned global model , for the global model that needs to embed watermark and The joint update operation is performed to maximize the mutual information expressed by the following formula:

[0075]

[0076] in, is a real sample dataset, yes The distribution of is the distribution of random sampling; yes A batch of samples, It is from Sampling and independent of Sample. is the watermark trigger sample set, yes A batch sample of It is a smooth approximation of the Relu function; Is the optimization goal, which aims to make the real sample data set Gradient and watermark trigger sample set A global model that maximizes mutual information between the gradients of The parameter distribution of . is a discriminant function used to identify the distribution of model parameters and is defined as follows:

[0077]

[0078] in, Is a watermark embedding function that combines gradient information; in this invention, this information is measured using the L2 norm and the distance between them; is the input sample; stands for Multilayer Perceptron (MLP); represents the corresponding malicious gradient; Represents the global model; Represents a composite function, Passed as input to middle.

[0079] S33, using watermark to trigger sample set For the global model Make fine adjustments.

[0080] These samples use the original labels, and the sample set is recorded as This operation essentially strengthens the dormant watermark and creates a base model for the next generation of iterations. .

[0081] Specifically, to perform image classification tasks, this example carefully selected the ResNet18 model as the global model in the federated learning environment, and used the CIFAR10 dataset as the client training set. CIFAR10 is a widely used dataset in image research, containing 60,000 color images, with a training set consisting of 50,000 images and a test set consisting of 10,000 images. In each category, the training set contains 5,000 images and the test set contains 1,000 images. These images are evenly distributed across ten categories, fully ensuring data diversity and representativeness, providing rich and balanced samples for model training.

[0082] In addition, this example introduces the German Traffic Sign Recognition Benchmark (GTSRB) dataset. This dataset is highly authoritative and widely used in traffic sign image research. It contains a large number of traffic sign images of different types, covering common categories such as prohibition signs, instruction signs, and warning signs. These images are collected from real traffic scenes, with rich background variations, varying lighting conditions, and multiple shooting angles. They highly reproduce the complex situations encountered in real applications, providing valuable samples for research on traffic-related image classification tasks and helping models learn comprehensive and accurate traffic sign features.

[0083] In terms of model selection, in addition to the ResNet18 model, the VGG16 model can also be used. VGG16 is a classic deep convolutional neural network whose structure contains 16 weight layers. The model is mainly composed of multiple convolutional layers and pooling layers stacked alternately. The convolutional layer is used to extract image features, while the pooling layer is responsible for reducing the dimension of the feature map and reducing the amount of calculation. Through a series of convolution and pooling operations, VGG16 can effectively extract deep features in the image. Similar to ResNet18, when using the VGG16 model, choose to extract activation maps in its penultimate convolutional layer. The key visual information contained in these activation maps will lay a solid foundation for subsequent learning and classification tasks.

[0084] The ResNet18 model has a unique architecture, with 18 layers comprising convolutional layers, batch normalization layers, ReLU activation functions, and residual blocks, which play a key role in feature enhancement. This unique structural design enables ResNet18 to effectively learn rich features when processing complex images, demonstrating excellent performance in image classification tasks. We chose to extract activation maps from the penultimate convolutional layer because they provide highly abstract and effective visual information, making them ideal for subsequent learning and classification.

[0085] In the federated learning framework, the server first receives model updates from each client. These model updates contain the model parameters obtained after training the client based on its local dataset. Client local datasets are typically unique and diverse, reflecting the characteristics of the client's environment or data source.

[0086] After receiving a model update, the server rigorously verifies the model's security. This verification process is crucial for ensuring the safe and stable operation of the entire federated learning system, preventing damage to the model from malicious attacks, data contamination, or other security threats. The server employs a series of security checks, such as verifying the source of model updates and checking the legitimacy of model parameters, to ensure the model's security and reliability.

[0087] After confirming the model's security, the server aggregates the models from different clients using the FedAvg algorithm (Federated Averaging Algorithm). FedAvg is a widely used model aggregation method in federated learning. It generates a global model by calculating the weighted average of the model parameters submitted by all participating clients. Specifically, the algorithm assigns weights to each client based on factors such as the size and quality of its dataset. It then takes the weighted sum of the model parameters from each client, ultimately generating a global model that integrates information from all clients. This weighted average aggregation approach leverages the strengths of each client, improving both the performance and generalization of the global model.

[0088] S4, using real sample data sets and watermark trigger sample set For the global model Fine-tune and extract labels ; and extract the tags With target label Compare; if and If they are consistent, the ownership of the model can be proved;

[0089] like Figure 4 As shown in the figure, when the server suspects a model leak, it will adopt the following rigorous verification process:

[0090] S41, first, the server will select a pre-defined real sample data set , and fine-tune the model. These real sample datasets They have been rigorously screened, have clear features and annotations, and can affect model parameters in a stable and predictable manner during fine-tuning.

[0091] S42, after fine-tuning is completed, the server will trigger the watermark sample set obtained through the predefined process The watermark trigger set samples are cleverly designed to activate watermark-related features that may be hidden in the model.

[0092] S43, then the server will carefully check the output labels of the model for the watermark trigger set samples, focusing on determining whether the output labels are completely consistent with the predefined target labels;

[0093] If the output label is the same as the predefined label, it provides strong evidence that the model is at risk of leakage, because only a model that has undergone a specific watermark embedding process can output the corresponding predefined label when faced with such specially designed input samples.

[0094] Based on a similar inventive concept, an embodiment of the present invention further provides a computer storage medium storing a readable program, which, when executed by a processor, can execute the above-mentioned mutual information-based federated learning copyright protection method.

[0095] Based on similar inventive concepts, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0096] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned mutual information-based federated learning copyright protection method.

[0097] Based on similar inventive concepts, an embodiment of the present invention further provides a computer program product, including computer instructions, which instruct a computing device to perform operations corresponding to the above-mentioned mutual information-based federated learning copyright protection method.

[0098] Example 2

[0099] In this embodiment, a federated learning copyright protection system based on mutual information is proposed, including:

[0100] Real sample data generation module: Sample noise from the potential space and input it into the generator to generate a real sample data set ;

[0101] Watermark trigger sample generation module: based on the real sample data set , using triggers to generate networks , generate watermark trigger sample set ;

[0102] Watermark embedding module: by maximizing the global model In the real data sample set and watermark-triggered sample set The mutual information between gradients is used to complete the embedding of dormant watermarks;

[0103] And, verification module: using real sample data sets and watermark trigger sample set For the global model Fine-tune and extract labels ; and extract the tags With target label Compare; if and If they are consistent, the ownership of the model can be proved.

[0104] The method of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It will be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0105] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A mutual information-based federated learning copyright protection method, characterized in that: The following steps are involved: Sample noise from the latent space and input it into the generator to generate a real sample dataset ; Based on the real sample dataset , using triggers to generate networks , generate watermark trigger sample set ; By maximizing the global model In the real data sample set and watermark-triggered sample set The mutual information between gradients is used to complete the embedding of dormant watermarks; Using real sample data sets and watermark trigger sample set For the global model Fine-tune and extract labels; compare the extracted labels with the target labels; if the two are consistent, the ownership of the model is proved; Generate watermark trigger sample set The process is: S21, define the trigger generation network and initialize the trigger characteristics , train the trigger generation network , designed to optimize trigger characteristics ,Mining the features of target tags, the target tags are tags pre-defined and selected by the server; S22, the real sample data set Input trigger generation network , output results and hyperparameters Multiply together to get the optimized trigger characteristics And the watermark trigger sample set is ;in is a data sample in the input real sample data set, is the target label; The global model The server aggregates the local models uploaded by each client to obtain: in, Is the client Model, Is the client data, is the collection of all client data; N is the total number of clients; The dormant watermark embedding steps are: S31, using a real sample dataset with correct labels For the global model Implement fine-tuning processing; S32, based on the fine-tuned global model , a joint update operation is performed on the current embedding parameters, aiming to maximize the global model In the real data sample set and watermark-triggered sample set Mutual information between gradients; S33, using watermark to trigger sample set For the global model Make fine adjustments.

2. A mutual information-based federated learning copyright protection method according to claim 1, characterized in that: Generate real sample dataset The process is: S11, sampling noise from the potential space that obeys the normal distribution to obtain the input vector; S12, the generator maps the input vector to a specific space and finally generates a real sample data set ; S13, update the generator through backpropagation .

3. The mutual information-based federated learning copyright protection method according to claim 1, characterized in that: The generator consists of a three-layer fully connected neural network.

4. A federated learning copyright protection system based on mutual information, characterized in that: include: Real sample data generation module: samples noise from the potential space and inputs it into the generator to generate a real sample data set ; Watermark trigger sample generation module: based on the real sample data set , using triggers to generate networks , generate watermark trigger sample set ; Watermark embedding module: by maximizing the global model In the real data sample set and watermark-triggered sample set The mutual information between gradients is used to complete the embedding of dormant watermarks; And, verification module: using real sample data sets and watermark trigger sample set For the global model Fine-tune and extract labels; compare the extracted labels with the target labels; if the two are consistent, the ownership of the model is proved; Generate watermark trigger sample set The process is: S21, define the trigger generation network and initialize the trigger characteristics , train the trigger generation network , designed to optimize trigger characteristics ,Mining the features of target tags, the target tags are tags pre-defined and selected by the server; S22, the real sample data set Input trigger generation network , output results and hyperparameters Multiply together to get the optimized trigger characteristics And the watermark trigger sample set is ;in is a data sample in the input real sample data set, is the target label; The global model The server aggregates the local models uploaded by each client to obtain: in, Is the client Model, Is the client data, is the collection of all client data; N is the total number of clients; The dormant watermark embedding steps are: S31, using a real sample dataset with correct labels For the global model Implement fine-tuning processing; S32, based on the fine-tuned global model , a joint update operation is performed on the current embedding parameters, aiming to maximize the global model In the real data sample set and watermark-triggered sample set Mutual information between gradients; S33, using watermark to trigger sample set For the global model Make fine adjustments.

5. A computer storage medium storing a readable program, characterized in that: When the program is executed by a processor, it can execute the mutual information-based federated learning copyright protection method described in any one of claims 1 to 3.

6. An electronic device, characterized in that: include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the mutual information-based federated learning copyright protection method according to any one of claims 1 to 3.

7. A computer program product comprising computer instructions, characterized in that The computer instructions instruct the computing device to perform operations corresponding to the mutual information-based federated learning copyright protection method as described in any one of claims 1 to 3.

Images