Public opinion scene-oriented efficient verifiable federal learning sentiment classification method

Abstract

The invention discloses a public opinion scene-oriented efficient verifiable federal learning sentiment classification method. The method comprises the following steps of: initializing a scheme in an initialization stage; each participant trains the large language model to obtain local gradient update; the participant compresses the local gradient by using convolution gradient compression to obtain gradient update, and uploads the gradient update to a server; after the server receives the compression gradients of all the participants, performing security aggregation to obtain global gradient update; the server randomly samples partial gradient dimensions according to a probability gradient sampling strategy and sends the partial gradient dimensions to the participant for verification; the participant decompresses the gradient dimension, compares the gradient dimension with a local gradient, and verifies the correctness of server aggregation; the server extracts partial layer parameters according to a random layer parameter verification mechanism, and requires a participant to provide corresponding local parameters; the server downloads local parameters of the random layer, and the local parameters are verified by a participant; the invention aims to realize efficient, safe and verifiable federated learning.

Description

Technical Field

[0001] The present invention relates to the technical fields of machine learning and data privacy protection, and particularly to an efficient verifiable federated learning sentiment classification method for public opinion scenarios. Background Art

[0002] In recent years, large language models (LLMs) have received extensive attention due to their excellent performance in the field of natural language processing. These models usually contain hundreds of millions of parameters and need to be trained on large-scale datasets. However, due to data privacy and security restrictions, centralized training faces many challenges. Federated Learning (FL), as a distributed machine learning paradigm, provides a possibility to solve this problem. In federated learning, each client trains the model locally and uploads the gradient updates instead of the original data to the central server for aggregation. Nevertheless, existing federated learning solutions still face many difficulties when applied to large language models.

[0003] Due to the huge number of parameters in large language models, the dimension of gradient updates is extremely high. Directly transmitting and verifying these high-dimensional gradients not only brings huge communication overhead but also increases the computational burden on the verification process. Additionally, malicious parties may submit biased or random gradient updates, affecting the performance of the global model. Therefore, how to verify the accuracy and reliability of gradient updates while ensuring aggregation efficiency has become a key challenge in the field of federated learning. Some existing works attempt to solve the above problems through technologies such as secure multi-party computation and homomorphic encryption. However, these methods often introduce high computational and communication overhead and are difficult to be directly applied to the federated learning of large language models. Therefore, an efficient, secure, and verifiable federated learning solution is needed to support the training and deployment of large language models in distributed scenarios. This will not only promote the popularization of artificial intelligence technology but also greatly facilitate the development of the data privacy protection cause. Summary of the Invention

[0004] To solve the problems existing in the prior art, the purpose of the present invention is to provide an efficient verifiable federated learning sentiment classification method for public opinion scenarios. The present invention is applicable to the training and deployment of large language models in distributed scenarios, aiming to achieve efficient, secure, and verifiable federated learning.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: an efficient verifiable federated learning sentiment classification method for public opinion scenarios, including the following steps:

[0006] Step 1, initialize the solution in the initialization phase;

[0007] Step 2: Each participant, i.e., each client, trains the large language model locally using its own data to obtain local gradient updates; the participant uses Convolutional Gradient Compression (CGC) to compress the local gradients to obtain compressed gradient updates, and uploads them to the server; after the server receives the compressed gradients from all participants, it first performs secure aggregation to obtain global gradient updates;

[0008] Step 3: The server randomly samples some gradient dimensions according to the Probabilistic Gradient Sampling Strategy (PGS) and sends them to the participants for verification; the participant decompresses the received gradient dimensions and compares them with the local gradients to verify the correctness of the server aggregation;

[0009] Step 4: The server extracts the parameters of some layers according to the random layer parameter verification mechanism and requests the participants to provide the corresponding local parameters; the server downloads the local parameters of the random layer for the participants to verify to ensure the consistency of the global parameters;

[0010] Step 5: After the verification passes, the participant applies the global gradient update to the local model and starts a new round of federated learning.

[0011] As a further improvement of the present invention, Step 1 is specifically as follows:

[0012] Input gradient matrix where m and n are the number of rows and columns of the matrix respectively, represents the real number field; define the maximum dimension d = max(m, n); then initialize two groups of matrix sequences: where the subscripts t and s represent the use in the training stage and the random layer selection stage respectively, and the elements in the two groups of matrix sequences represent the matrices used by each client to perturb the gradients to ensure the security of the encryption process. There are N clients in total; then generate N - 1 random matrices of dimension d Calculate the last matrix Ensure that the sum modulo N is equal to a predefined invertible matrix of dimension d That is Finally, generate two different random convolution kernels of dimension M And generate the inverse matrix of each client i in

[0013] As a further improvement of the present invention, Step 2 is specifically as follows:

[0014] During the training and update process, first input the gradient matrix of the l-th layer of client i and the matrix generated in the initialization stage; perform convolution on the gradient matrix of each client i to obtain the convolution value W l,i;W l,i [x,y] represents the convolution result at position (x,y), (u,v) represents the offset of the convolution kernel, and M represents the size of the convolution kernel; and calculate its homomorphic hash value h l,i =H(W l,i ), and broadcast it to other clients, where H() represents the hash function; then each client i encrypts the gradient of layer l Upload to the server; the server aggregates the encrypted gradient to obtain Then for the random selection server model N L -l layers {l1,l2,…,l NL-l The gradient matrix G in l′ , where N L Represents the total number of layers of the server model, l' represents a randomly selected layer; convolution of the selected gradient matrix Get the convolution value W l′ ; Calculate the homomorphic hash value h l′ =H(W l′ ), and use a random matrix for each client i Encrypt the gradient of each layer l' Finally, the hash value sequence h of all selected layers of each client i is collected. l′ and the encryption sequence S l′,k , broadcast by the server to other clients.

[0015] As a further improvement of the present invention, the step 3 is specifically as follows:

[0016] Each client uses convolution kernel K t For the polymerization gradient E l Perform convolution Get the convolution value E′ l ; Calculate the homomorphic hash value h of the aggregated gradient l =H(E′ l ), and h for each client i l,i Hash value homomorphic multiplication If H l ≠h l , the verification fails.

[0017] As a further improvement of the present invention, the step 4 is specifically as follows:

[0018] Each client i decrypts each selected layer l' using the corresponding inverse matrix to obtain And use the convolution kernel K s Perform convolution Get the convolution value K; calculate the homomorphic hash value h K =H(K); if h K ≠h l′, the verification fails.

[0019] As a further improvement of the present invention, in step 3, the probability gradient sampling strategy PGS is specifically as follows:

[0020] Let the gradient matrix of client i at the l-th layer in the e-th round of iteration be where d represents the gradient dimension; the probability gradient sampling strategy PGS randomly selects a gradient matrix of a dimension a (a < d) from for verification, where the probability of each element in being selected is

[0021] The present invention designs a new layer-by-layer verification method, which combines the Convolution Gradient Compression (CGC) technology, can significantly reduce the dimension of gradient update, thereby reducing communication and computational overhead. At the same time, this method can achieve efficient gradient verification while maintaining the model performance.

[0022] The present invention introduces a random layer parameter verification mechanism, which further enhances the security of the scheme. By randomly selecting the parameters of some layers for verification, it can effectively prevent the gradient tampering of malicious participants and reduce the verification cost at the same time.

[0023] The present invention proposes a Probabilistic Gradient Sampling (PGS) strategy, which dynamically adjusts the gradient sampling dimension in the verification process according to the training rounds. This adaptive sampling method minimizes communication and computational overhead while ensuring the comprehensiveness of verification.

[0024] Based on the secure aggregation protocol, the present invention guarantees the privacy of the gradient update of participants. Even in the presence of malicious participants, the scheme can ensure the correctness and immutability of the aggregation result.

[0025] The beneficial effects of the present invention are:

[0026] 1. The present invention designs a new layer-by-layer verification method, which combines the Convolution Gradient Compression (CGC) technology, can significantly reduce the dimension of gradient update, thereby reducing communication and computational overhead. At the same time, this method can achieve efficient gradient verification while maintaining the model performance.

[0027] 2. The present invention introduces a random layer parameter verification mechanism, which further enhances the security of the scheme. By randomly selecting the parameters of some layers for verification, it can effectively prevent the gradient tampering of malicious participants and reduce the verification cost at the same time.

[0028] 3. The present invention proposes a Probabilistic Gradient Sampling (PGS) strategy to dynamically adjust the gradient sampling dimension during the verification process according to the training rounds. This adaptive sampling method minimizes communication and computational overhead while ensuring the comprehensiveness of verification.

[0029] 4. Based on the secure aggregation protocol, the present invention ensures the privacy of the gradient updates of the participants. Even in the presence of malicious participants, the scheme can ensure the correctness and immutability of the aggregated results.

[0030] 5. The proposed Convolutional Gradient Compression and Probabilistic Sampling-based Efficient Verifiable Federated Learning Sentiment Classification Method for Public Opinion Scenarios (ConPro-FL) significantly improves the efficiency and security of federated learning for large language models while protecting the data privacy of the participants. This scheme provides an efficient, secure, and verifiable solution for the training and deployment of large language models in distributed scenarios, and is expected to promote the popularization of artificial intelligence technology and the development of data privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a system framework diagram of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Embodiment

[0034] As Figure 1 shown, an efficient verifiable federated learning sentiment classification method for public opinion scenarios includes the following steps:

[0035] (1) Each participant trains the large language model using its own data locally to obtain local gradient updates;

[0036] (2) The participant uses the CGC technology to compress the local gradient to obtain the compressed gradient update and uploads it to the central server;

[0037] (3) After the server receives the compressed gradients from all participants, it first performs secure aggregation to obtain the global gradient update;

[0038] (4) The server randomly samples some gradient dimensions according to the PGS strategy and sends them to the participants for verification;

[0039] (5) The participant decompresses the received gradient dimension and compares it with the local gradient to verify the correctness of the server aggregation;

[0040] (6) The server extracts the parameters of some layers according to the random layer parameter verification mechanism and requests the participating parties to provide the corresponding local parameters;

[0041] (7) The server downloads the local parameters of the random layer, which are verified by the participating parties to ensure the consistency of the global parameters;

[0042] (8) After passing the verification, the participating parties apply the global gradient update to the local model and start a new round of federated learning.

[0043] Specifically, it is divided into three stages, namely the initialization stage, the training and update stage, and the verification stage;

[0044] Initialization stage: First, initialize the scheme; input the gradient matrix where m and n are the number of rows and columns of the matrix respectively, represents the real number field; define the maximum dimension d = max(m, n); then initialize two groups of matrix sequences: where the subscripts t and s represent the use in the training stage and the random layer selection stage respectively, and the elements in these two sequences represent the matrices used by each client to perturb the gradient to ensure the security of the encryption process. There are N clients in total; then generate N - 1 random matrices of dimension d Calculate the last matrix Ensure that their sum modulo N is equal to a predefined invertible matrix of dimension d That is Finally, generate two different random convolution kernels of dimension M And generate the inverse matrix of each client i in

[0045] Training and update stage: During the model training and update process, first input the gradient matrix of the l-th layer of client i and the matrices generated in the initialization stage; perform convolution on the gradient matrix of each client i to obtain the convolution value W l,i ; where W l,i [x, y] represents the convolution result at position (x, y), (u, v) represents the offset of the convolution kernel, and M represents the size of the convolution kernel; and calculate its homomorphic hash value h l,i = H(W l,i ) and broadcast it to other clients, where H() represents the hash function; then each client i encrypts the gradient of the l-th layer and uploads it to the server; the server aggregates the encrypted gradients to obtain Then for the N L -l layers randomly selected by the server model {l1, l2,..., lNL-l Gradient matrix G in l′ , where N L represents the total number of layers of the server model, and l' represents a randomly selected layer; convolve the selected gradient matrix to obtain the convolution value W l′ ; calculate the homomorphic hash value h l′ = H(W l′ ), and use the random matrix of each client i to encrypt the gradient of each layer l' Finally, collect the hash value sequence h l′ and the encrypted sequence S l′,k of all selected layers of each client i, and broadcast them from the server to other clients.

[0046] Verification phase: In the process of verifying the aggregation result, each client uses the convolution kernel K t to convolve the aggregated gradient E l

[0047] to perform convolution to obtain the convolution value E' l ; calculate the homomorphic hash value h l = H(E' l ), and perform homomorphic multiplication on the h l,i hash values of each client i If H l ≠ h l , the verification fails; in the process of verifying the random layer parameters, for each selected layer l' of each client i, decrypt it with the corresponding inverse matrix to obtain and use the convolution kernel K s to perform convolution to obtain the convolution value K; calculate the homomorphic hash value h K = H(K); if h K ≠ h l′ , the verification fails.

[0048] In addition, this embodiment also introduces a Probabilistic Gradient Sampling (PGS) strategy: Let the gradient matrix of client i at layer l in the e-th iteration be where d represents the gradient dimension; the PGS strategy randomly selects a gradient matrix of dimension a (a < d) from (66) for verification, where the probability of each element in being selected is By randomly selecting a gradient matrix of a smaller dimension from the original gradient matrix for verification, the PGS strategy can further reduce the computational overhead.

[0049] By adopting the above innovative method, the ConPro-FL solution can significantly improve the efficiency and security of federated learning of large language models, providing strong support for its application in distributed scenarios; in this embodiment, layer-by-layer verification combined with convolutional gradient compression significantly reduces the dimension of gradient updates, thereby reducing overhead. This method can achieve efficient gradient verification while maintaining the performance of the model. This embodiment introduces a random layer parameter verification mechanism, further enhancing security. By randomly selecting the parameters of some layers for verification, it can effectively prevent gradient tampering by malicious participants while reducing the verification cost. This embodiment proposes a PGS strategy to dynamically adjust the gradient sampling dimension during verification according to the training rounds. This sampling method minimizes communication and computational overhead while ensuring the comprehensiveness of verification. This embodiment is based on a secure aggregation protocol, ensuring the privacy of the gradient updates of the participants. Even in the presence of malicious participants, it can ensure the correctness and immutability of the aggregation result.

[0050] The above embodiments only represent the specific implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. An efficient verifiable federated learning sentiment classification method for public opinion scenarios, characterized in that It includes the following steps: Step 1: Initialize the solution during the initialization phase; Step 2: Each participating party, i.e., each client, trains the large language model locally using its own data to obtain local gradient updates; the participating party uses Convolutional Gradient Compression (CGC) to compress the local gradients to obtain compressed gradient updates, and uploads them to the server; after the server receives the compressed gradients from all participating parties, it first performs secure aggregation to obtain global gradient updates; Step 3: The server randomly samples some gradient dimensions according to the Probabilistic Gradient Sampling Strategy (PGS) and sends them to the participating parties for verification; the participating parties decompress the received gradient dimensions and compare them with the local gradients to verify the correctness of the server aggregation; Step 4: The server extracts the parameters of some layers according to the random layer parameter verification mechanism and requests the participating parties to provide the corresponding local parameters; the server downloads the local parameters of the random layer for verification by the participating parties to ensure the consistency of the global parameters; Step 5: After the verification passes, the participating parties apply the global gradient updates to the local model and start a new round of federated learning.

2. The efficient verifiable federated learning sentiment classification method for public opinion scenarios according to claim 1, characterized in that The specific content of Step 1 is as follows: Input gradient matrix where m and n are the number of rows and columns of the matrix respectively, represents the real number field; Define the maximum dimension \(d = \max(m,n)\); then initialize two sets of matrix sequences: Among them, the subscripts \(t\) and \(s\) represent the use in the training stage and the random layer selection stage respectively. The elements in the two sets of matrix sequences represent the matrices used by each client to perturb the gradient to ensure the security of the encryption process. There are a total of \(N\) clients; then generate \(N - 1\) random matrices of dimension \(d\). Calculate the last matrix Ensure that the sum modulo \(N\) is equal to a predefined invertible matrix of dimension \(d\). That is Finally, generate two different random convolution kernels of dimension \(M\). And generate the inverse matrix of each client \(i\) in 3. The efficient verifiable federated learning sentiment classification method for public opinion scenarios according to claim 2, wherein The specific content of Step 2 is as follows: During the training and updating process, first input the gradient matrix of the l-th layer of client i and the matrix generated in the initialization stage; perform convolution on the gradient matrix of each client i to obtain the convolution value W l,i ; where W l,i [x, y] represents the convolution result at position (x, y), (u, v) represents the offset of the convolution kernel, and M represents the size of the convolution kernel; and calculate its homomorphic hash value h l,i =H(W l,i ), and broadcast it to other clients, where H() represents the hash function; then each client i encrypts the gradient of the l-th layer and uploads it to the server; the server aggregates the encrypted gradients to obtain Then, for the gradient matrix G L of the N -l layer l′ randomly selected from the server model, where N L represents the total number of layers of the server model, and l' represents the randomly selected layer; perform convolution on the selected gradient matrix to obtain the convolution value W l′ ; calculate the homomorphic hash value h l′ =H(W l′ ), and encrypt the gradient of each layer l' with the random matrix of each client i Finally, collect the hash value sequence h l′ and the encrypted sequence S l′,k of all selected layers of each client i, and broadcast them from the server to other clients.

4. The efficient verifiable federated learning sentiment classification method for public opinion scenarios according to claim 3, characterized in that, The specific content of Step 3 is as follows: Each client convolves with the convolutional kernel K t on the aggregated gradient E l to perform convolution ′ Obtain the convolution value E l ; Calculate the homomorphic hash value h of the aggregated gradient l = H(E′ l ), and perform homomorphic multiplication on the h l,i hash values of each client i If H l ≠ h l , the verification fails.

5. The efficient verifiable federated learning sentiment classification method for public opinion scenarios according to claim 4, characterized in that, The specific content of Step 4 is as follows: For each selected layer l' by each client i, decrypt it using the corresponding inverse matrix to obtain and perform convolution using the convolution kernel K s to get the convolution value K; calculate the homomorphic hash value h = H(K); if h K ≠ h K ≠ h l′ , the verification fails.

6. The efficient verifiable federated learning sentiment classification method for public opinion scenarios according to claim 5, wherein In Step 3, the Probabilistic Gradient Sampling Strategy (PGS) is specifically as follows: Let the gradient matrix of client i at the l-th layer in the e-th round of iteration be where d represents the gradient dimension; the probability gradient sampling strategy PGS randomly selects a gradient matrix of dimension a (a < d) from for verification, where The probability that each element in is selected is ​

Images