Efficient and verifiable federated learning sentiment classification method for public opinion scene

Abstract

The application discloses a kind of high-efficiency verifiable federated learning sentiment classification methods for public opinion scene, comprising: initialization stage is initialized to scheme;Each participant trains large language model, obtains local gradient update;Participating party uses convolution gradient compression to compress local gradient, obtains gradient update, and is uploaded to server;After server receives the compressed gradient of all participants, carries out security aggregation, obtains global gradient update;Server according to probability gradient sampling strategy, randomly sample part gradient dimension, send to participant verification;Gradient dimension is decompressed by participant, and compared with local gradient, to verify the correctness of server aggregation;Server extracts part of layer parameters according to random layer parameter verification mechanism, and requires participants to provide corresponding local parameters;The local parameters of random layer are downloaded by server, and are verified by participants;The application aims at realizing efficient, safe and verifiable federated learning.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and data privacy protection, and in particular to an efficient and verifiable federated learning sentiment classification method for public opinion scenarios. Background Technology

[0002] In recent years, Large Language Models (LLMs) have garnered widespread attention due to their outstanding performance in natural language processing. These models typically contain hundreds of millions of parameters and require training on massive datasets. However, centralized training faces numerous challenges due to data privacy and security constraints. Federated Learning (FL), as a distributed machine learning paradigm, offers a potential solution to this problem. In federated learning, each client trains the model locally and uploads gradient updates, rather than the original data, to a central server for aggregation. Nevertheless, existing federated learning schemes still face many difficulties when applied to large language models.

[0003] Due to the massive number of parameters in large language models, gradient updates have extremely high dimensionality. Directly transmitting and validating these high-dimensional gradients not only incurs huge communication overhead but also increases the computational burden on the validation process. Furthermore, malicious parties may submit biased or random gradient updates, affecting the performance of the global model. Therefore, ensuring aggregation efficiency while validating the accuracy and reliability of gradient updates has become a key challenge in the field of federated learning. Some existing works have attempted to address these issues through secure multi-party computation and homomorphic encryption. However, these methods often introduce high computational and communication overhead, making them difficult to apply directly to federated learning of large language models. Therefore, an efficient, secure, and verifiable federated learning scheme is needed to support the training and deployment of large language models in distributed scenarios. This will not only promote the widespread application of artificial intelligence technology but also greatly contribute to the development of data privacy protection. Summary of the Invention

[0004] To address the problems existing in the prior art, the purpose of this invention is to provide an efficient and verifiable federated learning sentiment classification method for public opinion scenarios. This invention is applicable to the training and deployment of large language models in distributed scenarios, aiming to achieve efficient, secure and verifiable federated learning.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: an efficient and verifiable federated learning sentiment classification method for public opinion scenarios, comprising the following steps:

[0006] Step 1: Initialization Phase - Initialize the scheme;

[0007] Step 2: Each participant, i.e., each client, trains the large language model locally using its own data to obtain local gradient updates; the participants use Convolutional Gradient Compression (CGC) to compress the local gradients to obtain compressed gradient updates, and upload them to the server; after receiving the compressed gradients from all participants, the server first performs secure aggregation to obtain global gradient updates.

[0008] Step 3: The server randomly samples a portion of the gradient dimensions according to the probabilistic gradient sampling strategy (PGS) and sends them to the participants for verification. The participants decompress the received gradient dimensions and compare them with their local gradients to verify the correctness of the server's aggregation.

[0009] Step 4: The server extracts parameters from a portion of the layers based on the random layer parameter verification mechanism and requests the participants to provide the corresponding local parameters. The server then downloads the local parameters of the random layers for the participants to verify, ensuring the consistency of the global parameters.

[0010] Step 5: After successful verification, the participants apply the global gradient update to the local model and begin a new round of federated learning.

[0011] As a further improvement to the present invention, step 1 is specifically as follows:

[0012] Input gradient matrix Where m and n are the number of rows and columns of the matrix, respectively. Represent the real number field; define the maximum dimension d = max(m,n); then initialize two sets of matrix sequences: Here, the subscripts t and s represent the subscripts used during the training and random layer selection phases, respectively. The elements in the two sets of matrix sequences represent the matrices used by each client to perturb the gradient, ensuring the security of the encryption process. There are a total of N clients. Then, N-1 random matrices of dimension d are generated. Calculate the last matrix Ensure that the sum modulo N equals a predefined invertible matrix of dimension d. Right now Finally, two different random convolution kernels of dimension M are generated. and generation The inverse matrix of each client i

[0013] As a further improvement of the present invention, step 2 is specifically as follows:

[0014] During training and updating, the gradient matrix of the l-th layer of client i is first input. And the matrix generated during the initialization phase; convolution is performed on the gradient matrix of each client i. Obtain the convolution value W l,i;W l,i [x,y] represents the convolution result at position (x,y), (u,v) represents the offset of the convolution kernel, and M represents the size of the convolution kernel; and its homomorphic hash value h is calculated. l,i =H(W l,i The algorithm is then broadcast to other clients, where H() represents the hash function; subsequently, each client i encrypts the gradient of the l-th layer. Uploaded to the server; the server aggregates the encryption gradient to obtain... Then for N of the randomly selected server model L -l layers {l1,l2,…,l NL-l The gradient matrix G in} l′ , where N L This represents the total number of layers in the server model, where l' represents a randomly selected layer; convolution is performed on the selected gradient matrix. Obtain the convolution value W l′ ; Calculate the homomorphic hash value h l′ =H(W l′ ), and a random matrix for each client i Encrypt the gradient of each layer l' Finally, collect the hash value sequence h of all selected layers for each client i. l′ and encrypted sequence S l′,k This is broadcast by the server to other clients.

[0015] As a further improvement to the present invention, step 3 is specifically as follows:

[0016] Each client uses convolution kernel K t For the aggregation gradient E l Convolution Obtain the convolution value E′ l ; Calculate the homomorphic hash value h of the aggregation gradient. l =H(E′) l ), and h for each client i l,i Homomorphic multiplication of hash values If H l ≠h l If so, the verification fails.

[0017] As a further improvement to the present invention, step 4 is specifically as follows:

[0018] For each selected layer l', client i decrypts it using the corresponding inverse matrix to obtain... And use convolution kernel K s Convolution Obtain the convolution value K; calculate the homomorphic hash value h. K =H(K); if h K ≠h l′If so, the verification fails.

[0019] As a further improvement of the present invention, in step 3, the probability gradient sampling strategy PGS is specifically as follows:

[0020] Let the gradient matrix of client i in the l-th layer be given by the i-th iteration. Where d represents the gradient dimension; the probabilistic gradient sampling strategy PGS starts from... Randomly select a gradient matrix of dimension a (a < d) Verification was conducted, among which The probability of each element in the array being selected is...

[0021] This invention presents a novel layer-by-layer verification method that, combined with Convolution Gradient Compression (CGC) technology, significantly reduces the dimensionality of gradient updates, thereby reducing communication and computational overhead. Furthermore, this method achieves efficient gradient verification while maintaining model performance.

[0022] This invention introduces a random layer parameter verification mechanism, further enhancing the security of the scheme. By randomly selecting parameters from a subset of layers for verification, gradient tampering by malicious actors can be effectively prevented, while simultaneously reducing verification costs.

[0023] This invention proposes a Probabilistic Gradient Sampling (PGS) strategy, which dynamically adjusts the gradient sampling dimension during the validation process based on the training epochs. This adaptive sampling method minimizes communication and computational overhead while ensuring comprehensive validation.

[0024] This invention, based on a secure aggregation protocol, guarantees the privacy of gradient updates by participating parties. Even in the presence of malicious participants, the scheme ensures the correctness and immutability of the aggregation results.

[0025] The beneficial effects of this invention are:

[0026] 1. This invention designs a novel layer-by-layer verification method that, combined with Convolution Gradient Compression (CGC) technology, can significantly reduce the dimensionality of gradient updates, thereby reducing communication and computational overhead. Simultaneously, this method can achieve efficient gradient verification while maintaining model performance.

[0027] 2. This invention introduces a random layer parameter verification mechanism, further enhancing the security of the scheme. By randomly selecting parameters from a subset of layers for verification, gradient tampering by malicious parties can be effectively prevented, while simultaneously reducing verification costs.

[0028] 3. This invention proposes a Probabilistic Gradient Sampling (PGS) strategy, which dynamically adjusts the gradient sampling dimension during the verification process based on the training epochs. This adaptive sampling method minimizes communication and computational overhead while ensuring comprehensive verification.

[0029] 4. This invention is based on a secure aggregation protocol, ensuring the privacy of gradient updates by participating parties. Even in the presence of malicious participants, the scheme can ensure the correctness and immutability of the aggregation results.

[0030] 5. The efficient and verifiable federated learning sentiment classification method (ConPro-FL) for public opinion scenarios, based on convolutional gradient compression and probabilistic sampling, proposed in this invention significantly improves the efficiency and security of federated learning of large language models, while protecting the data privacy of participating parties. This solution provides an efficient, secure, and verifiable solution for the training and deployment of large language models in distributed scenarios, and is expected to promote the widespread application of artificial intelligence technology and the development of data privacy protection. Attached Figure Description

[0031] Figure 1 This is a system framework diagram of an embodiment of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] Example

[0034] like Figure 1 As shown, an efficient and verifiable federated learning sentiment classification method for public opinion scenarios includes the following steps:

[0035] (1) Each participant uses its own data to train the large language model locally and obtains local gradient updates;

[0036] (2) The participants use CGC technology to compress the local gradients, obtain the compressed gradient update, and upload it to the central server;

[0037] (3) After receiving the compressed gradients from all participants, the server first performs secure aggregation to obtain the global gradient update.

[0038] (4) The server randomly samples a portion of the gradient dimensions according to the PGS strategy and sends them to the participants for verification.

[0039] (5) The participants decompress the received gradient dimension and compare it with the local gradient to verify the correctness of the server aggregation.

[0040] (6) The server extracts parameters from some layers based on the random layer parameter verification mechanism and requires the participants to provide the corresponding local parameters.

[0041] (7) The server downloads local parameters of the random layer, which are verified by the participants to ensure the consistency of the global parameters;

[0042] (8) After the verification is successful, the participants will apply the global gradient update to the local model and start a new round of federated learning.

[0043] Specifically, it is divided into three stages: the initialization stage, the training and update stage, and the verification stage.

[0044] Initialization phase: First, the scheme is initialized; the gradient matrix is ​​input. Where m and n are the number of rows and columns of the matrix, respectively. Represent the real number field; define the maximum dimension d = max(m,n); then initialize two sets of matrix sequences: The subscripts t and s indicate the use during the training and random layer selection phases, respectively. The elements in these two sets of sequences represent the matrices used by each client to perturb the gradients, ensuring the security of the encryption process. There are a total of N clients. Then, N-1 random matrices of dimension d are generated. Calculate the last matrix Ensure that their sum modulo N equals a predefined invertible matrix of dimension d. Right now Finally, two different random convolution kernels of dimension M are generated. and generation The inverse matrix of each client i

[0045] Training and Update Phase: During model training and update, the gradient matrix of the l-th layer of client i is first input. And the matrix generated during the initialization phase; convolution is performed on the gradient matrix of each client i. Obtain the convolution value W l,i ;W l,i [x,y] represents the convolution result at position (x,y), (u,v) represents the offset of the convolution kernel, and M represents the size of the convolution kernel; and its homomorphic hash value h is calculated. l,i =H(W l,i The algorithm is broadcast to other clients, where H() represents a hash function; then each client i encrypts the gradient of the l-th layer. Uploaded to the server; the server aggregates the encryption gradient to obtain... Then for N of the randomly selected server model L -l layers {l1,l2,…,lNL-l The gradient matrix G in} l′ , where N L This represents the total number of layers in the server model, where l' represents a randomly selected layer; convolution is performed on the selected gradient matrix. Obtain the convolution value W l′ ; Calculate the homomorphic hash value h l′ =H(W l′ ), and a random matrix for each client i Encrypt the gradient of each layer l' Finally, collect the hash value sequence h of all selected layers for each client i. l′ and encrypted sequence S l′,k This is broadcast by the server to other clients.

[0046] Validation Phase: During the validation process of the aggregation results, each client uses a convolution kernel K. t For the aggregation gradient E l

[0047] Convolution Obtain the convolution value E′ l ; Calculate the homomorphic hash value h of the aggregation gradient. l =H(E′) l ), and h for each client i l,i Homomorphic multiplication of hash values If H l ≠h l If the verification fails, then the verification fails. During the random layer parameter verification process, each client i decrypts each selected layer l' using the corresponding inverse matrix to obtain the result. And use convolution kernel K s Convolution Obtain the convolution value K; calculate the homomorphic hash value h. K =H(K); if h K ≠h l′ If so, the verification fails.

[0048] Furthermore, this embodiment also introduces a probabilistic gradient sampling (PGS) strategy: let the gradient matrix of client i in the l-th layer during the e-th iteration be... Where d represents the gradient dimension; the PGS policy starts from... Randomly select a gradient matrix of dimension a (a < d) Verification was conducted, among which The probability of each element in the array being selected is... By randomly selecting a smaller-dimensional gradient matrix from the original gradient matrix for verification, the PGS strategy can further reduce computational overhead.

[0049] By employing the aforementioned innovative methods, the ConPro-FL scheme significantly improves the efficiency and security of federated learning for large language models, providing strong support for its application in distributed scenarios. This embodiment combines layer-by-layer verification with convolutional gradient compression, significantly reducing the dimensionality of gradient updates and thus reducing overhead. This method achieves efficient gradient verification while maintaining model performance. This embodiment introduces a random layer parameter verification mechanism to further enhance security. By randomly sampling parameters from some layers for verification, gradient tampering by malicious parties can be effectively prevented, while simultaneously reducing verification costs. This embodiment proposes a PGS strategy, dynamically adjusting the gradient sampling dimension during the verification process based on the training epochs. This sampling method minimizes communication and computational overhead while ensuring comprehensive verification. This embodiment, based on a secure aggregation protocol, guarantees the privacy of gradient updates by participating parties, ensuring the correctness and immutability of the aggregation results even in the presence of malicious parties.

[0050] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An efficient and verifiable federated learning sentiment classification method for public opinion scenarios, characterized in that, Includes the following steps: Step 1: Initialization Phase - Initialize the scheme; Step 2: Each participant, i.e., each client, trains the large language model locally using its own data to obtain local gradient updates; the participants use Convolutional Gradient Compression (CGC) to compress the local gradients to obtain compressed gradient updates, and upload them to the server; after receiving the compressed gradients from all participants, the server first performs secure aggregation to obtain global gradient updates. Step 2 is described in detail below: During training and updating, the gradient matrix of the l-th layer of client i is first input. And the matrix generated during the initialization phase; convolution is performed on the gradient matrix of each client i. Obtain the convolution value W l,i ;W l,i [x,y] represents the convolution result at position (x,y), [u,v] represents the offset of the convolution kernel, and M represents the size of the convolution kernel; and its homomorphic hash value is calculated. And broadcast to other clients, among which This represents the hash function; then each client i encrypts the gradient of the l-th layer. Uploaded to the server; the server aggregates the encryption gradient to obtain... Then for N of the randomly selected server model L -l layer The gradient matrix G in l′ , where N L This represents the total number of layers in the server model, where l' represents a randomly selected layer; convolution is performed on the selected gradient matrix. Obtain the convolution value W l′ Calculate the homomorphic hash value And use a random matrix for each client i Encrypt the gradient of each layer l' Finally, collect the hash sequence h of all selected layers for each client i. l′ and encrypted sequence S l′,k The server broadcasts this information to other clients. Step 3: The server randomly samples a portion of the gradient dimensions according to the probabilistic gradient sampling strategy (PGS) and sends them to the participants for verification. The participants decompress the received gradient dimensions and compare them with their local gradients to verify the correctness of the server's aggregation. Step 3 is as follows: Each client uses convolution kernel K t For the aggregation gradient E l Perform convolution Obtain the convolution value E l ; Calculate the homomorphic hash value of the aggregate gradient. and h for each client i l,i Homomorphic multiplication of hash values If H l ≠h l If so, the verification fails; In step 3, the probability gradient sampling strategy PGS is as follows: Let the gradient matrix of client i in the l-th layer be given by the i-th iteration. Where d represents the gradient dimension; the probabilistic gradient sampling strategy PGS starts from... A gradient matrix of dimension a is randomly selected from the given data. Verify that a < d, where The probability of each element being selected is Step 4: The server extracts parameters from a portion of the layers based on the random layer parameter verification mechanism and requests the participants to provide the corresponding local parameters. The server then downloads the local parameters of the random layers for the participants to verify, ensuring the consistency of the global parameters. Step 5: After successful verification, the participants apply the global gradient update to the local model and begin a new round of federated learning.

2. The efficient and verifiable federated learning sentiment classification method for public opinion scenarios according to claim 1, characterized in that, Step 1 is described in detail as follows: Input gradient matrix Where m and n are the number of rows and columns of the matrix, respectively. Represents the real number field; Define the maximum dimension d = max(m,n); then initialize two sets of matrix sequences: Here, the subscripts t and s represent the subscripts used during the training and random layer selection phases, respectively. The elements in the two sets of matrix sequences represent the matrices used by each client to perturb the gradient, ensuring the security of the encryption process. There are a total of N clients. Then, N-1 random matrices of dimension d are generated. Calculate the last matrix Ensure and model It equals a predefined invertible matrix of dimension d. Right now Finally, two different random convolution kernels of dimension M are generated. and generation The inverse matrix of each client i 3. The efficient and verifiable federated learning sentiment classification method for public opinion scenarios according to claim 1, characterized in that, Step 4 is described in detail below: For each selected layer l', client i decrypts it using the corresponding inverse matrix to obtain... And use convolution kernel K s Convolution Obtain convolution values Calculate homomorphic hash value if The verification will fail.

Images