Human activity recognition method and system based on federated learning

Abstract

The present invention provides a method and system for human activity recognition based on federated learning. The method comprises: step 1: the server determines the training task and the corresponding activity data features, generates initial model parameters and sends them to each client; step 2: each client performs local training based on the received initial model parameters or new global model parameters, combines its own local activity data, generates local update parameters, and uploads the local update parameters to the server; step 3: the server performs weighted average aggregation on the received local update parameters of each client to generate new global model parameters, and sends the new global model parameters to each client; step 4: repeating steps 2 to 3 until the training stop condition is reached to obtain the final global model parameters, and the server sends the final global model parameters to each client. The present invention improves the accuracy of HAR recognition while protecting user privacy.

Description

Technical Field

[0001] The present invention relates to the technical field of human activity recognition, and in particular to a human activity recognition method and system based on federated learning. Background Art

[0002] With the rapid aging of the population, the research on human activity recognition (HAR) has become particularly important. People can detect daily activities through wearable devices such as smartphones, wristbands and smart glasses. In recent years, machine learning has been increasingly applied to the field of human activity recognition. Various methods based on heuristic manual feature extraction and automatic feature extraction have been applied to human activity recognition for identification and classification. At present, existing human activity recognition work focuses on centralized learning methods to train models on central servers, which requires collecting all activity data from the user end for centralized training. However, centralized collection of sensor data will cause the risk of sensitive information being leaked, and users' privacy vital sign data is difficult to be effectively protected. In addition, when running machine learning models on mobile devices, the labeled data on each client is usually not enough to train a good model. Moreover, due to individual differences among different users, external environment, and sensor bias, different user data often show high heterogeneity and randomness, resulting in poor model accuracy.

[0003] In order to protect the privacy of user data, some scholars have applied federated learning to HAR. Federated learning (FL) is a distributed machine learning method. Different from the centralized training method of machine learning, its training data does not need to be centrally collected and stored in the cloud. Instead, it is based on the local data of each data source for model training. The data does not need to be moved, which reduces the external transmission of data and reduces the risk of privacy data leakage. However, the existing federated learning method learns the global model for the user by averaging the models of each user end. Studies have shown that although this method can handle non-independent and identically distributed data to a certain extent, the decrease in data accuracy is inevitable. Summary of the invention

[0004] In view of the problem that current human activity recognition methods cannot protect the privacy of user data or the recognition results are not accurate enough, the present invention provides a human activity recognition method and system based on federated learning, which can improve the accuracy of human recognition while protecting user privacy.

[0005] On the one hand, the present invention provides a method for human activity recognition based on federated learning, wherein multiple clients establish a connection and communicate with a server via Ethernet, and the method comprises:

[0006] Step 1: The server determines the training task and the corresponding activity data features, generates initial model parameters, and sends the initial model parameters to each client;

[0007] Step 2: Each client performs local training based on the received initial model parameters or new global model parameters in combination with its own local activity data, generates local update parameters, and uploads the local update parameters to the server;

[0008] Step 3: The server performs weighted average aggregation on the local update parameters received from each client to generate new global model parameters, and sends the new global model parameters to each client;

[0009] Step 4: Repeat steps 2 to 3 until the training stop condition is reached to obtain the final global model parameters. The server sends the final global model parameters to each client.

[0010] Furthermore, step 2 specifically includes:

[0011] Each client performs a local single or multiple iterations on the initial model parameters or new global model parameters according to the set mini-batch value, number of iterations, optimization algorithm and learning rate, generates local update parameters, and then encrypts and transmits the local update parameters to the server.

[0012] Furthermore, the optimization algorithm adopts the gradient descent method and calculates according to formula (1): Then according to Generate local update parameters according to formula (2)

[0013]

[0014]

[0015] in, represents the gradient of the k-th client after the t-th iteration update, represents the gradient operator, w t represents the global model parameters after the tth iteration update, f k (*) represents the objective function on a given k-th client, η is the learning rate, represents the local update parameter of the kth client, and t represents the number of iterations.

[0016] Furthermore, step 3 specifically includes:

[0017] Determine a value of a hyperparameter p, and use the hyperparameter p to adjust the weight of the local update parameters of each client;

[0018] The server performs weighted average aggregation on the local update parameters received from each client according to formula (3) to generate a new global model parameter w t :

[0019]

[0020] in, K represents the number of clients, n k represents the sample size of the kth client, represents the weight of the local update parameter of the kth client, represents the local update parameter of the kth client, and t represents the number of iterations.

[0021] Furthermore, during the training process, the function shown in formula (4) is used as the objective function:

[0022]

[0023] Where w represents the model parameter, f(w) represents the objective function, and F k (w) represents the average loss function of the kth client.

[0024] On the other hand, the present invention also provides a human activity recognition system based on federated learning, comprising: a server and a plurality of clients; the plurality of clients are connected to and communicate with the server via Ethernet;

[0025] The server is used to determine the training task and the corresponding activity data features, generate initial model parameters, and send the initial model parameters to each client; and perform weighted average aggregation on the received local update parameters of each client to generate new global model parameters, and send the new global model parameters to each client;

[0026] Each of the clients is used to perform local training based on the received initial model parameters or new global model parameters in combination with its own local activity data, generate local update parameters, and upload the local update parameters to the server.

[0027] Beneficial effects of the present invention:

[0028] The human activity recognition method and system based on federated learning provided by the embodiment of the present invention is a new horizontal federation (HFL) framework. Each client does not share data, which solves the problem of privacy protection in the HAR recognition process and realizes the protection of user vital sign data in HAR recognition. In addition, by introducing a hyperparameter p, by changing the weight of the client's local update parameter, weighted aggregation is used to optimize the model aggregation effect, which improves the accuracy of human activity recognition to a certain extent, solves the problem of low aggregation accuracy, and improves the recognition accuracy of the federated learning algorithm on HAR. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the interaction process between multiple clients and servers provided in an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a process flow of a human activity recognition method based on federated learning provided in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of selecting the optimal value of the hyperparameter p provided in an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of Accuracy at different learning rates provided by an embodiment of the present invention;

[0033] Figure 5 The following is a schematic diagram of the accuracy of the method of the present invention and the existing FEDAVG algorithm provided in the embodiments of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for human activity recognition based on federated learning, wherein multiple clients establish a connection and communicate with a server via Ethernet, and the method comprises the following steps:

[0036] S101: The server determines the training task and the corresponding activity data features, generates initial model parameters, and sends the initial model parameters to each client;

[0037] S102: Each client performs local training based on the received initial model parameters or new global model parameters in combination with its own local activity data to generate local update parameters, and uploads the local update parameters to the server;

[0038] Specifically, each client performs a single or multiple iterations of the initial model parameters or the new global model parameters locally according to the set mini-batch value, number of iterations, optimization algorithm and learning rate, generates local update parameters, and then encrypts and transmits the local update parameters to the server. The Paillier homomorphic encryption algorithm is selected for encryption.

[0039] In practical applications, considering that the size of the data sample affects the accuracy of the generated local update parameters to a certain extent, conditions can be set on whether the client is allowed to participate in this training task. For example, clients with a local activity data sample size greater than 10 are allowed to participate in this training task.

[0040] S103: The server performs weighted average aggregation on the local update parameters received from each client to generate new global model parameters, and sends the new global model parameters to each client;

[0041] S104: Repeat steps S102 to S103 until the training stop condition is reached to obtain the final global model parameters, and the server sends the final global model parameters to each client.

[0042] The human activity recognition method based on federated learning provided in the embodiment of the present invention is a new horizontal federation (HFL) framework. Each client does not share data, which solves the problem of privacy protection in the HAR recognition process and realizes the protection of user vital sign data in HAR recognition. In addition, a weighted aggregation method is adopted to solve the problem of low aggregation accuracy and improve the recognition accuracy of the federated learning algorithm on HAR.

[0043] Example 2

[0044] Based on the above embodiment, the embodiment of the present invention further provides a human activity recognition method based on federated learning (called P-FEDAVG), which includes the following steps:

[0045] S201: The server determines the training task and the corresponding activity data features, generates initial model parameters, and sends the initial model parameters to each client;

[0046] S202: Each client performs local training based on the received initial model parameters or new global model parameters in combination with its own local activity data to generate local update parameters, and uploads the local update parameters to the server;

[0047] When combining local activity data and current model parameters for training, the multi-classification cross entropy loss function categorical_crossentropy is used and softmax is introduced as a classifier to identify activities.

[0048] As an implementation method, each client uses the gradient descent method to calculate according to formula (1): Then according to Generate local update parameters according to formula (2)

[0049]

[0050]

[0051] in, represents the gradient of the k-th client after the t-th iteration update, represents the gradient operator, w t represents the global model parameters after the tth iteration update, f k (*) represents the objective function on a given k-th client, η is the learning rate, represents the local update parameter of the kth client, and t represents the number of iterations.

[0052] S203: The server performs weighted average aggregation on the local update parameters received from each client to generate new global model parameters, and sends the new global model parameters to each client;

[0053] It can be understood that the amount of data participating in local training on each client is not the same. Considering that small samples reflect obvious individual characteristics, large samples can better reflect common characteristics. Therefore, in this embodiment, in order to effectively adjust the weights of the local update parameters of each client, a hyperparameter p is introduced to achieve a relatively reasonable distribution of weights, thereby improving the accuracy of the entire global model parameters.

[0054] It should be noted that when p>1, the weight of the local update parameters of the client with a large sample size will be increased, and the weight of the local update parameters of the client with a small sample size will be weakened; when p<1, the weight of the local update parameters of the client with a small sample size will be increased, and the weight of the local update parameters of the client with a large sample size will be weakened. By determining the optimal p value, the accuracy of HAR activity recognition can be improved.

[0055] As an implementation method, the server performs weighted average aggregation on the local update parameters received from each client according to formula (3) to generate a new global model parameter w t :

[0056]

[0057] in, K represents the number of clients, n k represents the sample size of the kth client, represents the weight of the local update parameter of the kth client, represents the local update parameter of the kth client, and t represents the number of iterations.

[0058] S204: Repeat steps S202 to S203 until the training stop condition is reached to obtain the final global model parameters, and the server sends the final global model parameters to each client.

[0059] Specifically, during the training process, the function shown in formula (4) is used as the objective function:

[0060]

[0061] Where w represents the model parameter, f(w) represents the objective function, and F k (w) represents the average loss function of the kth client.

[0062] The training stop condition can be repeated iterations until the model converges to minimize the accuracy loss of global model parameters.

[0063] The training data of federated learning comes from different data sources, and the sample data of different clients are not independent and identically distributed. The training data obviously cannot meet the two conditions that can affect federated learning: independent distribution and consistency of training data. If the training data distribution of the data source is different, then integrating multiple sub-models will become a huge problem. In the prior art, there are methods that use neural networks to integrate multiple sub-models, but the neural network model generally behaves as a non-convex function, and it is difficult to optimize the model loss function after parameter averaging. This is indeed the case. Some scholars have conducted research on independent and identically distributed data and non-independent and identically distributed data, and found that the final integrated model of non-independent distribution cannot meet the expected requirements. In addition, since the sample data of different clients are not independent and identically distributed, when the amount of data on the client is very different, directly using the sample size for first-order modeling will cause drift between the aggregation result and the true result. In order to solve the above problem, the human activity recognition method based on federated learning provided in the embodiment of the present invention introduces a hyperparameter p, and optimizes the model aggregation effect by weighted aggregation by changing the weight of the client's local update parameter, thereby improving the accuracy of human activity recognition to a certain extent.

[0064] Example 3

[0065] Combination Figure 1 As shown, an embodiment of the present invention further provides a human activity recognition system based on federated learning, including: a server and multiple clients; the multiple clients are connected to and communicate with the server via Ethernet;

[0066] The server is used to determine the training task and the corresponding activity data features, generate initial model parameters, and send the initial model parameters to each client; and perform weighted average aggregation on the received local update parameters of each client to generate new global model parameters, and send the new global model parameters to each client;

[0067] Each of the clients is used to perform local training based on the received initial model parameters or new global model parameters in combination with its own local activity data, generate local update parameters, and upload the local update parameters to the server.

[0068] It should be noted that the human activity recognition system based on federated learning provided in the embodiment of the present invention is to implement the above method embodiment. Its specific functions can be referred to the above method embodiment and will not be repeated here.

[0069] In order to verify the effectiveness of the human activity recognition method and system provided by the present invention, the present invention also provides the following experimental data.

[0070] Dataset: The public dataset UCI-HAR (Human Activity Recognition Using Smartphones Data Set) is used to predict the effect of activity recognition. The UCI-HAR dataset is completed by 30 volunteers aged between 19 and 48. Each person wears a smartphone (Samsung Galaxy S) around his waist and then completes 6 activities (walking, climbing stairs, descending stairs, sitting, standing, and lying down). The three-axis linear acceleration and three-axis angular velocity data are obtained through the built-in accelerometer and gyroscope of the mobile phone. The sampling frequency of the data is 50Hz, and it contains 561 feature vectors of time domain and frequency domain variables. Table 1 is a detailed description of the dataset. On the server and client, a DNN neural network is used for training and prediction. The network is a 6-layer fully connected network structure. The stochastic gradient descent method is used for optimization. During the training process, 70% of the data is used for model training, and the remaining 30% is used for model evaluation.

[0071] Table 1 Description of UCI-HAR dataset

[0072] Activity Type illustrate Sample size walk Participants walk straight on a level surface 1722 Go up the stairs Participants walk up the stairs 1544 Go down the stairs Participants walk down the stairs 1406 Meditation Participants sit on chairs 1777 Standing Participants stand still 1906 Lie down Participant sleeping or lying down 1904

[0073] Experimental results: Following the homomorphic encryption in federated learning, we select appropriate hyperparameters p, learning rate, and number of neural network layers to ensure the convergence and aggregation effect of the model. For the hyperparameter p, we select the optimal p value by comparing the convergence of the model loss under different p values. Figure 3 As shown in the figure, the change of model loss value at different Epochs under different p values ​​is used to show the dynamic change of p value. It can be seen that when p = 1.3, the model loss reaches the minimum, indicating that when the hyperparameter p = 1.3 in this data set, the sample weight can be better fitted, reducing the drift distance between the model aggregation and the true value.

[0074] In addition, the changes in recognition accuracy under different learning rates are compared when the optimal hyperparameter p=1.3 to improve the convergence speed of the model and the recognition accuracy. Figure 4 As shown in the figure, the horizontal axis represents the number of iterations Epoch, and the vertical axis represents the recognition accuracy Accuracy. When the learning rate is 0.008, the Accuracy shows a downward trend as the number of Epochs increases. When the learning rate is 0.002, the convergence speed of Accuracy is slow; when the learning rate is 0.006, the convergence speed of Accuracy is fast. When the number of iterations Epoch reaches 100, the model's Loss value and recognition accuracy Accuracy tend to be stable, and the highest Accuracy value reaches 99.41%. Therefore, this paper selects a learning rate of 0.006.

[0075] Finally, under the optimal p value, the changes in the recognition accuracy Accuracy of the method of the present invention under different neural network layers are compared, and the number of layers of the neural network model most suitable for training is selected to improve the convergence speed of the model. As shown in Table 2, the effect of the recognition accuracy Accuracy under different neural network layers is presented as a quadratic function similar to an opening downward. The Accuracy value first increases, then slowly flattens and slightly decreases. The reason is that as the number of layers increases, the expression ability and fitting ability of the model are improved, and the effect of the model is also improved. However, when the number of neural network layers reaches 6 layers, the model effect remains the same or decreases, mainly because the model has overfitting as the number of layers increases. After experiments, we selected the most suitable network model structure to achieve better training effects and improve the accuracy of activity recognition. In addition, hyperparameters are used in the training process of the model. Hyperparameters are crucial for deep learning methods because they directly affect the effect of the training algorithm and have a crucial impact on the performance of the deep learning model. Table 3 is a summary of the hyperparameter tuning of the model in the experiment.

[0076] Table 2 Comparison of network layers

[0077]

[0078] Table 3 Summary of model hyperparameters

[0079]

[0080] In order to verify the effectiveness of the method of the present invention, the changes in the loss value and recognition accuracy of the model under the existing algorithm and the P-FedAvg algorithm are compared. Figure 5 As shown. Through experiments, it is found that the effect of the model trained by the federated learning (FedAvg and P-FedAvg) method increases with the increase of the number of iterations Epoch. When the number of iterations Epoch reaches 100, the Loss value and recognition accuracy of the model tend to be stable. By comparing the FedAvg and P-FedAvg algorithms, it can be found that the recognition accuracy of the P-FedAvg algorithm is better than that of the FedAvg algorithm as a whole, indicating that the P-FedAvg algorithm proposed in the present invention has obvious effects.

[0081] Table 4 shows the accuracy comparison between the proposed method and other network models on the same dataset. This further verifies that the proposed method model has positively improved the overall accuracy while ensuring the privacy of user vital sign data.

[0082] Table 4 Performance results comparison

[0083]

[0084] In Table 4, Lee [1] refers to Lee S M, Yoon S M, Cho H. Human activity recognition from accelerometer data using Convolutional Neural Network [C] / / 2017 ieee international conference on big data and smart computing (bigcomp). IEEE, 2017:131-134; Hernandez [2] refers to Hernández F, Suárez L F, Villamizar J, et al. Human activity recognition on smartphones using a bidirectional lstm network [C] / / 2019 XXII symposium on image, signal processing and artificial vision (STSIVA). IEEE, 2019:1-5; Mutegeki [3] refers to Mutegeki R, Han D S. A CNN-LSTM approach to human activity recognition [C] / / 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC). IEEE, 2020:362-366; Ni [4] refers to Ni Q, Fan Z, Zhang L, et al. Leveraging wearable sensors for human daily activity recognition with stacked denoising autoencoders [J]. Sensors, 2020, 20(18):5114; Mekruksavanich [5] refers to Mekruksavanich S, Jitpattanakul A. Lstm networks using smartphone data for sensor-based human activity recognition in smart homes [J]. Sensors, 2021, 21(5):1636.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human activity recognition method based on federated learning, wherein multiple clients establish a connection and communicate with a server via Ethernet, characterized in that: The method comprises: Step 1: The server determines the training task and the corresponding activity data features, generates initial model parameters, and sends the initial model parameters to each client; Step 2: Each client performs local training based on the received initial model parameters or new global model parameters in combination with its own local activity data, generates local update parameters, and uploads the local update parameters to the server; Step 3: The server performs weighted average aggregation on the local update parameters received from each client to generate a new global model parameter, and sends the new global model parameter to each client; specifically, it includes: determining the value of the hyperparameter p, and using the hyperparameter p to adjust the weight of the local update parameter of each client; the server performs weighted average aggregation on the local update parameters received from each client according to formula (3) to generate a new global model parameter w t : in, K represents the number of clients, n k represents the sample size of the kth client, represents the weight of the local update parameter of the kth client, represents the local update parameter of the kth client, and t represents the number of iterations; Step 4: Repeat steps 2 to 3 until the training stop condition is reached to obtain the final global model parameters. The server sends the final global model parameters to each client.

2. The method for human activity recognition based on federated learning according to claim 1, characterized in that: Step 2 specifically includes: Each client performs a local single or multiple iterations on the initial model parameters or new global model parameters according to the set mini-batch value, number of iterations, optimization algorithm and learning rate, generates local update parameters, and then encrypts and transmits the local update parameters to the server.

3. The method for human activity recognition based on federated learning according to claim 2, characterized in that: The optimization algorithm adopts the gradient descent method and is calculated according to formula (1): Then according to Generate local update parameters according to formula (2) in, represents the gradient of the k-th client after the t-th iteration update, represents the gradient operator, w t represents the global model parameters after the tth iteration update, f k (*) represents the objective function on a given k-th client, η is the learning rate, represents the local update parameter of the kth client, and t represents the number of iterations.

4. The method for human activity recognition based on federated learning according to claim 1, characterized in that: During the training process, the function shown in formula (4) is used as the objective function: Where w represents the model parameter, f(w) represents the objective function, and F k (w) represents the average loss function of the kth client.

5. Human activity recognition system based on federated learning, characterized by: include: Server and multiple clients; Multiple clients are connected and communicate with the server via Ethernet; The server is used to determine the training task and the corresponding activity data features, generate initial model parameters, and send the initial model parameters to each client; and perform weighted average aggregation on the received local update parameters of each client to generate new global model parameters, and send the new global model parameters to each client; Specifically used to determine the value of the hyperparameter p, and use the hyperparameter p to adjust the weight of the local update parameters of each client; The server performs weighted average aggregation on the local update parameters received from each client according to formula (3) to generate a new global model parameter w t : in, K represents the number of clients, n k represents the sample size of the kth client, represents the weight of the local update parameter of the kth client, represents the local update parameter of the kth client, and t represents the number of iterations; Each of the clients is used to perform local training based on the received initial model parameters or new global model parameters in combination with its own local activity data, generate local update parameters, and upload the local update parameters to the server.

Images