Wireless traffic prediction method based on dynamic gradient sparse federated learning

Abstract

The invention relates to a wireless traffic prediction method based on dynamic gradient sparse federated learning. Comprising the following steps: a training process: a central server broadcasts a global model to each local base station device participating in local training; training and updating the received global model on the local data set by the local base station equipment; then, gradient information screened out according to the dynamic gradient rarefaction rule is uploaded to a central server, and other gradient information is discarded in situ and does not execute a communicated task; after the central server receives the gradient information uploaded by the local base station equipment, federal aggregation is carried out to generate a new global model; broadcasting the obtained new global model to the local base station equipment again; and the prediction process comprises the step of realizing wireless traffic prediction through the new global model. According to the invention, the dynamic gradient sparsification strategy is added before the communication between the central server and the local base station equipment, so that the phenomenon of high communication overhead caused by frequent communication can be reduced.

Description

Technical Field

[0001] The present invention relates to the fields of wireless communication and artificial intelligence technology, and in particular to a wireless traffic prediction method based on dynamic sparse federated learning. Background Art

[0002] Wireless traffic forecasting is a key task to accurately predict the traffic in the future time period in wireless communication networks. Accurate traffic forecasting can help network operators optimize network resource allocation, improve network performance, and meet user needs.

[0003] However, traditional wireless traffic prediction methods have some limitations, which restrict their effectiveness and scalability in practical applications. Traditional wireless traffic prediction methods usually require centralized data collection, facing the problems of uneven data distribution and large-scale data processing. Centralized methods involve privacy and security risks and are not effective for large-scale networks. Non-independent and identically distributed data is difficult to process, and frequent model updates are also a challenge.

[0004] To overcome these problems, federated learning is applied to wireless traffic prediction as a special distributed machine learning method. It allows participants to perform local model training while protecting data privacy, and achieve collaborative learning and parameter updates through global model aggregation. This method improves prediction accuracy and scalability while protecting data privacy. At the same time, it also significantly reduces communication overhead compared to centralized methods.

[0005] However, federated learning still requires a high communication frequency in wireless traffic prediction, so reducing communication overhead and improving communication efficiency are still one of the main tasks today. Therefore, it is urgent to develop a wireless network prediction method that takes into account both prediction performance and reducing communication overhead. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a wireless traffic prediction method based on dynamic gradient sparse federated learning;

[0007] The present invention aims to provide a solution that can accurately predict wireless traffic while protecting user privacy and reducing communication overhead. It is used to solve the challenges of traditional traffic prediction methods in protecting user privacy and reducing data transmission volume. It can be directly used in network resource allocation, optimizing network decisions, etc.

[0008] The embodiment of the present invention analyzes the challenges faced by wireless traffic services and the advantages of the federated learning framework, and concludes that it is feasible to achieve wireless traffic prediction under the federated learning framework. Under the framework of federated learning, based on the conclusion that most of the information transmission is redundant in each communication process between the central server and the local base station device, the present invention proposes a dynamic gradient sparsification strategy. This strategy is not only intended to improve the prediction performance, but also to reduce the amount of communication between the central server and the local base station device each time, thereby achieving the purpose of reducing communication overhead.

[0009] Terminology explanation:

[0010] 1. Stochastic gradient descent is the most commonly used optimization algorithm for solving machine learning algorithms. For the objective function to be optimized, the stochastic gradient descent algorithm calculates the gradient information of the objective function by cyclically sampling part of the data samples, and uses this gradient information to update the model parameters. After reaching the maximum number of cycles, the cycle ends, and stochastic gradient descent will return the final model parameter information.

[0011] 2. Adaptive moment estimation introduces high-order gradient information and momentum terms on the basis of stochastic gradient descent algorithm. Adaptive moment estimation uses the first-order and second-order information of gradient to dynamically adjust the learning rate of each parameter. The main advantage of adaptive moment estimation is that after the bias correction of gradient, the learning rate of each iteration has a certain range, making the parameters relatively stable.

[0012] 3. Momentum method is an extension of the traditional gradient descent algorithm. It is more efficient than the stochastic gradient descent algorithm. Momentum method is also called momentum-based gradient descent method. It is a method that accelerates the change of gradient vector in the relevant direction and finally achieves accelerated convergence. Momentum method is a very popular optimization algorithm and is used in the optimization of many current deep learning models.

[0013] The technical solution of the present invention is:

[0014] A wireless traffic prediction method based on dynamic gradient sparse federated learning, comprising:

[0015] The training process includes:

[0016] The central server broadcasts the global model to each local base station device participating in local training;

[0017] The local base station equipment trains and updates the received global model on the local data set; then, the gradient information selected according to the dynamic gradient sparsification rule is uploaded to the central server, and the remaining gradient information is discarded on the spot and no longer performs the communication task;

[0018] After receiving the gradient information uploaded by the local base station device, the central server performs federation aggregation to generate a new global model; the obtained new global model is broadcasted to the local base station device again;

[0019] Repeat the training process until the training end condition is met;

[0020] The prediction process includes:

[0021] Wireless traffic prediction is achieved through a new global model.

[0022] Preferably, according to the present invention, the wireless traffic prediction method is run on a wireless traffic prediction system; the wireless traffic prediction system includes a plurality of local base station devices, namely base stations, and a central server.

[0023] Preferably according to the present invention, the global model is a linear regression model or a neural network model.

[0024] Preferably, according to the present invention, the local base station device trains the received global model on the local data set; comprising:

[0025] Local data set preprocessing stage: data cleaning, removal of invalid or missing data, processing of outliers or noise data; then, standardization or normalization of the data;

[0026] For the preprocessed local data set, divide it into a training data set and a test data set;

[0027] A sliding window setting is performed on both the training dataset and the test dataset.

[0028] Preferably, according to the present invention, the local base station device updates the received global model on the local data set; comprising:

[0029] A. Determine the optimization method;

[0030] B. Select the batch size and gradient calculation algorithm based on the size of the training set and test set and the hardware conditions;

[0031] C. Update the global model using the calculated gradient information;

[0032] D. Repeat steps B and C according to the training settings until the maximum number of training cycles is reached and the relevant parameters are updated.

[0033] Preferably, according to the present invention, the gradient information is screened out according to the dynamic gradient sparsification rule, as shown in formula (1):

[0034]

[0035] In formula (1), represents the gradient information filtered out after gradient sparsification; g(·) is the generalized representation of the gradient sparsification strategy; ρ represents the gradient sparsification ratio; ρ satisfies the condition: 0≤ρ≤1, and is adjusted according to the training round.

[0036] Preferably, according to the present invention, after receiving the gradient information uploaded by the local base station device, the central server performs federation aggregation to generate a new global model; as shown below:

[0037]

[0038] Among them, w represents the global model and ∈ represents the learning rate of the optimization algorithm.

[0039] Preferably, according to the present invention, wireless traffic prediction is achieved through a new global model; including:

[0040] The local base station device uses the final prediction model, i.e., the new global model, to perform wireless traffic prediction on the test dataset.

[0041] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of a wireless traffic prediction method based on dynamic gradient sparse federated learning.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a wireless traffic prediction method based on dynamic gradient sparse federated learning.

[0043] The beneficial effects of the present invention are:

[0044] The embodiment of the present invention implements wireless traffic prediction on the federated learning framework and proposes a dynamic gradient sparsification federated learning algorithm. After receiving the locally uploaded gradient information, the central server performs federated aggregation to generate a new global model. Compared with traditional federated learning, the example method of the present invention incorporates a dynamic gradient sparsification strategy before the central server and the local base station device communicate, which can reduce the high communication overhead caused by frequent communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of a wireless traffic prediction system of the present invention;

[0046] Figure 2 It is a schematic diagram of the training process of the prediction model of the present invention;

[0047] Figure 3 is a schematic diagram comparing the prediction performance of the algorithm proposed in the embodiment of the present invention with that of the traditional algorithm on two data sets;

[0048] Figure 4 It is a schematic diagram of comparison between a predicted value and a true value of a certain local base station device and error analysis according to an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of communication overhead analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0051] Example 1

[0052] A wireless traffic prediction method based on dynamic gradient sparse federated learning, comprising:

[0053] The training process includes:

[0054] The central server broadcasts the global model to each local base station device participating in local training;

[0055] The local base station equipment trains and updates the received global model on the local data set; then, according to the dynamic gradient sparsification rule, a small part of the important gradient information is screened out and uploaded to the central server, and the rest of the gradient information is discarded on the spot and no longer performs the communication task;

[0056] After receiving the gradient information uploaded by the local base station device, the central server performs federation aggregation to generate a new global model; the obtained new global model is broadcasted to the local base station device again;

[0057] Repeat the training process until the training end condition is met;

[0058] The prediction process includes:

[0059] Wireless traffic prediction is achieved through a new global model.

[0060] Compared with traditional federated learning, the example method of the present invention takes into account the dynamic gradient sparsification strategy before the central server and the local base station device communicate, which can reduce the high communication overhead caused by frequent communication and improve the overall communication efficiency of wireless traffic prediction work.

[0061] Example 2

[0062] The wireless traffic prediction method based on dynamic gradient sparse federated learning described in Example 1 is different in that:

[0063] The wireless traffic prediction method runs on the wireless traffic prediction system; e.g. Figure 1As shown in Figure 1, the wireless traffic prediction system includes multiple (K) local base station devices, i.e., base stations, and a central server. At the same time, each base station has q wireless traffic data and a local prediction model w from the central server. k .

[0064] A wireless traffic prediction method based on dynamic gradient sparsification federated learning is used to solve the problems of excessive communication overhead and inability to protect data privacy in traditional centralized methods. This method achieves the wireless traffic prediction task by combining the ideas of gradient sparsification and federated learning. Figure 1 The system model is shown. Figure 2 The core flow chart of the system is shown. It contains K base stations, K = 100. As shown below:

[0065] (1) Initializing all local base station devices, assuming that there are K local base station devices. The initialization process includes: numbering the local base station devices as 1; ...; k; ...; K; numbering all local data sets as d k ;…;d k ;…;d K ;

[0066] (2) The central server initializes a random neural network model w for training, namely the global model w, and broadcasts it to the learning unit of each local base station device;

[0067] (3) After receiving the global model w from the central server, the local base station device k is renumbered as w k , and use the local dataset to train the local model w k Train and update the local model w k Parameters, after t local training rounds, the new local model w is obtained k ;

[0068] (4) Each local base station device filters the updated local model according to a specific gradient sparsification strategy, and finally sends the compressed gradient information of the local model to the central server;

[0069] (5) After receiving the gradient information from the local base station device, the central server obtains a new global model w through the federated aggregation method and broadcasts the obtained new global model to the local base station device;

[0070] (6) Repeat steps (3) to (5) until the training end condition is met;

[0071] When the central server broadcasts the global model to the local base station device for the first time, after the local base station device receives the global model, the local base station device needs to determine the prediction model type and initialize the local data set. The specific steps include:

[0072] A. Regarding the determination of the prediction model type: Since wireless traffic data has a strong time dependency, this patent chooses to use a long short-term memory network that can handle long-term dependencies; since edge base station devices collect a large amount of data, in order to lower the threshold of edge device computing performance, this patent chooses to use a lightweight long short-term memory network with only two layers; since wireless traffic data has long-term trends and periodicity, the model input includes two parts, which are used to represent trend characteristics and periodic characteristics respectively;

[0073] B. For the initialization of local data sets, firstly, data cleaning is performed to remove invalid or missing data and process outliers or noise data. Then, the data is standardized;

[0074] C. For the test set and training set, this patent chooses to set the last week as the test set and the rest of the data as the training set. This patent adopts the sliding window mechanism, the sliding window size is set to p = 3, and the sliding step size is set to z period =24, z close = 1. A training data set of 1104 items and a test data set of 168 items are generated.

[0075] The global model is a linear regression model or a neural network model. For example: linear regression method, multi-layer perceptron, long-term and short-term neural network model, etc.

[0076] The local base station device trains the received global model on the local data set; including:

[0077] Local data set preprocessing stage: data cleaning, removal of invalid or missing data, processing of outliers or noise data; then, standardization or normalization of the data;

[0078] For the preprocessed local data set, divide it into training data set and test data set according to a certain ratio;

[0079] Sliding windows are set for both the training dataset and the test dataset. The sliding window size is set to p and the sliding step size is set to z.

[0080] The local base station device updates the received global model on the local data set; including:

[0081] A. Determine the optimization method; including but not limited to stochastic gradient descent, adaptive moment estimation, momentum method;

[0082] B. Select an appropriate batch size and gradient calculation algorithm based on the size of the training set and test set and the hardware conditions; the actual training is performed on the base station side. The training can be based on the size of the training set and test set (if the training and test samples are small and can be put into the video memory at one time, a large batch is selected to save processing time; otherwise, a small batch must be selected to ensure successful training) and the current load of the base station (idle state, few service users, more computing resources available, and low CPU temperature; busy state, many service users, fewer computing resources available, and high CPU temperature) to determine the batch size (the batch size indicates how many samples are sampled at a time for calculation. The more samples in the batch, the higher the computing resource requirements) and the gradient calculation method (the traditional stochastic gradient descent only needs to calculate the gradient information, with less calculation and low complexity; the advanced gradient calculation method has more calculation and high complexity).

[0083] C. Update the global model using the calculated gradient information;

[0084] Gradient information is the gradient value obtained by derivation of the objective function with respect to the model parameters. From a mathematical point of view, gradient information can be expressed as Where f represents the loss function corresponding to the model, w represents the model parameters, Represents the derivative operation of the objective function f with respect to w.

[0085] D. Repeat steps B and C according to the training settings until the maximum number of training cycles is reached and the relevant parameters are updated.

[0086] According to the dynamic gradient sparsification rule, a small part of important gradient information is screened out, and the local gradient information to be uploaded is calculated according to formula (1) And discard other redundant information:

[0087]

[0088] In formula (1), represents the gradient information filtered after gradient sparsification; g(·) is a generalized representation of the gradient sparsification strategy; g(·) represents a strategy for sparsifying the gradient vector; in actual execution, given a gradient vector Sort its values; when a ρ value is given, for example, ρ = 0.2, the largest ρn values ​​and their corresponding positions are selected, and the gradient values ​​at other positions are filtered out and only these ρn gradient values ​​are retained. ρ represents the gradient sparsification ratio; ρ satisfies the condition: 0≤ρ≤1, and ρ is not fixed and is adjusted according to the training round.

[0089] After receiving the gradient information uploaded by the local base station device, the central server performs federation aggregation to generate a new global model; as shown below:

[0090]

[0091] Among them, w represents the global model and ∈ represents the learning rate of the optimization algorithm.

[0092] Wireless traffic prediction is achieved through a new global model; including:

[0093] The local base station device uses the final prediction model, i.e., the new global model, to perform wireless traffic prediction on the test dataset.

[0094] By comparing the predicted value with the true value, the relevant evaluation indicators are calculated to evaluate the prediction performance;

[0095] Calculate the total compression ratio of all local base station devices and evaluate the total communication overhead;

[0096] After the evaluation is completed, the local base station equipment continues to collect local data to enrich the database.

[0097] When using the wireless traffic prediction method based on dynamic gradient sparse federated learning of the present invention, the following changes will occur over time: the traffic pattern near the base station may change; the wireless traffic data collected by the local base station continues to accumulate; the working status of the local base station equipment may change. This will cause the performance of the original prediction model to continue to decline, and the overhead of retraining the prediction model will continue to increase. To solve this problem, the wireless traffic prediction method based on dynamic gradient sparse federated learning of the present invention also includes the following specific steps:

[0098] According to the working conditions of the base stations, new prediction models are trained regularly. Each time the prediction model is trained, the data set size is fixed and the data set is updated in real time. According to the working status of the local base station equipment, whether to participate in the prediction model training task is reasonably adjusted. If the number of base stations participating in the training does not meet the set threshold during a certain update, the update is skipped and the next update request is waited for.

[0099] After achieving the training goal, the performance of the prediction model is evaluated, including:

[0100] A. The local base station device uses the final prediction model to predict wireless traffic on the test data set;

[0101] B. Compare the predicted value with the true value and calculate the relevant evaluation indicators to evaluate the prediction performance. Since it is time series data, the accuracy cannot be compared, so this patent evaluates the model prediction performance by the mean absolute error and mean square error;

[0102] C. Calculate the total compression ratio of all local base station devices and evaluate the total communication overhead;

[0103] D. After the evaluation is completed, the local base station equipment continues to collect local data to enrich the database.

[0104] The results of wireless service traffic prediction based on dynamic gradient sparse federated learning proposed in this embodiment are as follows: Figure 3 , Figure 4 , Figure 5 shown. Figure 3 The figure is a schematic diagram comparing the prediction performance of the algorithm proposed in this embodiment with the traditional algorithm on two data sets; the two data sets are the Milan data set and the Trento data set. The traditional federated type indicates an algorithm that does not compress gradient information; the traditional centralized type indicates that the same prediction algorithm is used as the federated type, but with a different training strategy. For a prediction task, there is a data set and a prediction model. If the model is trained directly on the data set, then this category is centralized learning; if the data set is distributed on K local devices instead of in a data center, and these K devices train the model in a collaborative manner, then it is the traditional federated type; if the data is distributed on K local devices, and during collaborative training, each local device first compresses the gradient information and then sends it to the server for aggregation, then it is the algorithm proposed in this embodiment.

[0105] Figure 4 This is a schematic diagram of the comparison between the predicted value and the true value of a local base station device and the error analysis according to an embodiment of the present invention; the closer the predicted value is to the true value, the more accurate the prediction is. Figure 4 It can be seen from the above that the algorithm proposed in this embodiment is more consistent with the true value. Figure 4 The following shows the cumulative probability distribution of the forecast absolute error (CDR). If the CDR curve of the forecast absolute error quickly reaches probability 1, it means that the forecast is good. Figure 4 It can also be concluded that the algorithm proposed in this embodiment is better than the traditional federal and centralized algorithms.

[0106] from Figure 3 and Figure 4 It can be observed that the use of dynamic gradient sparsification strategy improves the prediction model performance. Therefore, the wireless service flow prediction solution proposed in the present invention can effectively improve the prediction performance.

[0107] Figure 5 The figure is a schematic diagram of the communication overhead analysis of the embodiment of the present invention. The traditional federation algorithm transmits all the gradient information, so its communication overhead is 100%; while the algorithm proposed in this embodiment greatly compresses the gradient information to be transmitted by the local base station, and the communication overhead is only 1.929% of the traditional algorithm. Figure 5 It can be concluded from the results that the use of dynamic gradient sparsification strategy can greatly reduce the communication burden and improve communication efficiency.

[0108] Example 3

[0109] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a wireless traffic prediction method based on dynamic gradient sparse federated learning described in embodiment 1 or 2 are implemented.

[0110] Example 4

[0111] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a wireless traffic prediction method based on dynamic gradient sparse federated learning described in embodiment 1 or 2 are implemented.

Claims

1. A wireless traffic prediction method based on dynamic gradient sparsification federated learning, characterized in that: include: The training process includes: The central server broadcasts the global model to each local base station device participating in local training; The local base station equipment trains and updates the received global model on the local data set; then, the gradient information selected according to the dynamic gradient sparsification rule is uploaded to the central server, and the remaining gradient information is discarded on the spot and no longer performs the communication task; After receiving the gradient information uploaded by the local base station device, the central server performs federation aggregation to generate a new global model; the obtained new global model is broadcasted to the local base station device again; Repeat the training process until the training end condition is met; The prediction process includes: Wireless traffic prediction is achieved through a new global model.

2. According to claim 1, a wireless traffic prediction method based on dynamic gradient sparsification federated learning is characterized in that: The wireless traffic prediction method is run on a wireless traffic prediction system; the wireless traffic prediction system includes a plurality of local base station devices, namely base stations, and a central server.

3. According to claim 1, a wireless traffic prediction method based on dynamic gradient sparse federated learning is characterized in that: The global model is a linear regression model or a neural network model.

4. According to claim 1, a wireless traffic prediction method based on dynamic gradient sparse federated learning is characterized in that: The local base station device trains the received global model on the local data set; including: Local data set preprocessing stage: data cleaning, removal of invalid or missing data, processing of outliers or noise data; then, standardization or normalization of the data; For the preprocessed local data set, divide it into a training data set and a test data set; A sliding window setting is performed on both the training dataset and the test dataset.

5. According to claim 1, a wireless traffic prediction method based on dynamic gradient sparsification federated learning is characterized in that: The local base station device updates the received global model on the local data set; including: A. Determine the optimization method; B. Select the batch size and gradient calculation algorithm based on the size of the training set and test set and the hardware conditions; C. Update the global model using the calculated gradient information; D. Repeat steps B and C according to the training settings until the maximum number of training cycles is reached and the relevant parameters are updated.

6. According to claim 1, a wireless traffic prediction method based on dynamic gradient sparsification federated learning is characterized in that: The gradient information is filtered out according to the dynamic gradient sparsification rule, as shown in formula (1): In formula (1), represents the gradient information filtered out after gradient sparsification; g(·) is the generalized representation of the gradient sparsification strategy; ρ represents the gradient sparsification ratio; ρ satisfies the condition: 0≤ρ≤1, and is adjusted according to the training round.

7. According to claim 1, a wireless traffic prediction method based on dynamic gradient sparsification federated learning is characterized in that: After receiving the gradient information uploaded by the local base station device, the central server performs federation aggregation to generate a new global model; as shown below: Among them, w represents the global model and ∈ represents the learning rate of the optimization algorithm.

8. A wireless traffic prediction method based on dynamic gradient sparse federated learning according to any one of claims 1 to 7, characterized in that: Wireless traffic prediction is achieved through a new global model; including: The local base station device uses the final prediction model, i.e., the new global model, to perform wireless traffic prediction on the test dataset.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of any one of claims 1-8 of the wireless traffic prediction method based on dynamic gradient sparse federated learning are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of any one of claims 1-8 of a wireless traffic prediction method based on dynamic gradient sparse federated learning are implemented.

Images