A human motion recognition method based on individual information of personalized federated learning

Abstract

The application discloses a human motion recognition method based on individual information of personalized federated learning, comprising the following steps: 1, calculating the similarity between clients according to individual information; 2, deploying the model to the client by the server; 3, training the model according to the local data set by the client; 4, aggregating the personalized model for each client according to the similarity matrix by the server until the model converges, deploying the final model and performing real-time analysis. The application can guarantee the accuracy of classification under the condition that the data distribution of the client is heterogeneous, thereby reducing the communication cost of federated learning.

Description

Technical Field

[0001] This invention belongs to the field of sensor data processing and analysis, specifically a personalized federated learning method for human action recognition based on individual information. Background Technology

[0002] The development of artificial intelligence and mobile technologies has greatly promoted smart healthcare, and wearable sensor-based human motion recognition is a classic application of smart healthcare. Utilizing wearable sensors such as accelerometers, human motion recognition systems can collect vast amounts of real-time data and then employ advanced AI models such as deep learning to recognize a person's activities, such as standing and walking. Wearable sensor-based human motion recognition systems have improved people's quality of life. However, traditional sensor-based wearable human motion recognition is centralized, meaning that sensor data collected from multiple individuals is transmitted to the cloud or data center for aggregation, training a centralized AI model for the human motion recognition system. This approach has three limitations. First, it significantly increases security and privacy concerns, as unauthorized access or data leakage may occur during the transmission of data remotely. Furthermore, if hackers compromise the centralized data system storing sensor data, all data could be illegally transmitted to and accessed by unauthorized external parties. Second, the transmission of large amounts of wearable sensor data increases network traffic and may create potential bottlenecks, posing a significant obstacle to network communication. Third, sensor data is highly sensitive because it records detailed activities in people's daily lives and can infer many health conditions (e.g., the risk of Parkinson's disease). Therefore, the flow and intrusion of sensor data are strictly restricted by regulations.

[0003] Federated learning was proposed to address these challenges. Using federated learning, sensor data is collected and stored on individual devices, and decentralized AI models (called local models) are trained using each person's data. Compared to the centralized paradigm, federated learning offers the following advantages. First, because personal data is stored locally, the risk of data leakage during transmission is significantly reduced. Meanwhile, hackers must compromise all decentralized devices to access all data, making this task more difficult than attacking a centralized data system. Second, transmitting local model parameters requires fewer communication resources than sending the original data, as the parameter size is typically fixed and significantly smaller than the original data. Third, federated learning does not involve data circulation or aggregation, thus complying with legal regulations. Furthermore, because the global model summarizes useful information from all clients, federated learning-based models can often achieve accuracy comparable to AI models trained in traditional centralized settings. These advantages have inspired numerous innovative federated learning methods for sensor-based wearable human motion recognition. However, a major challenge these methods face is data heterogeneity, as different activity patterns among different individuals often result in different local data. The problem of heterogeneity hinders the generalization ability of federated learning across different individuals. To address this issue, existing methods can be categorized into three types. The first type adopts the idea of ​​FedAvg, performing a weighted average of all local models to obtain a global model for each client. While these methods alleviate the data heterogeneity problem to some extent, their effectiveness is limited because they do not explicitly consider heterogeneity in their design. The second type of research specifically addresses the heterogeneity problem, aiming to design a more general global model suitable for different individuals. However, a single model may struggle to adapt to all individuals, thus limiting its generalization ability. The third type overcomes this limitation by building a personalized global model for each individual. These methods typically rely on the similarity between individuals for personalization. However, measuring similarity by comparing the local models from each individual in each round introduces additional computational costs. Furthermore, the parameters of the local models will change during federated learning training, so similarity relationships calculated in one round may not hold in the next. Therefore, the stability and performance of federated learning may be hampered. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a personalized federated learning method for human action recognition based on individual information. This method aims to ensure the accuracy and stability of the human action recognition system while using fewer communication resources, even with heterogeneous client data. It also aims to provide users with more accurate human action recognition results using less private data, thereby protecting user privacy and reducing the risk of data privacy leaks.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The present invention provides a human action recognition method based on personalized federated learning using individual information, characterized by the following steps:

[0007] Step 1. Assume the k-th client has its own individual information d. k and using d k,i Representing individual information d k The information attribute of the i-th dimension;

[0008] Let the motion data of the local sensor preprocessed by the kth client be ξ. k , and ξ k =x k,1 ,y k,1 ),(x k,2 ,y k,2 ),…(x k,s ,y k,s ),…(x k,S ,y k,S ); where x k,s and y k,s Let represent the action sample of the s-th local sensor after preprocessing by the k-th client and its corresponding category, where k = 1, 2, ..., K, K is the total number of clients, and s = 1, 2, ..., S, S is the total number of action samples of the local sensor of the k-th client.

[0009] For individual information d k After standardization, the cosine similarity formula is used to calculate the similarity between the k-th client and the ... Individual information similarity between clients Thus, the similarity matrix S is obtained. This indicates that the two clients are completely similar. This indicates that the two clients are orthogonal;

[0010] Step 2. Define the global training epochs as t, and initialize t = 1; define the global maximum training epochs as Tmax;

[0011] During the t-th global training process, the server will deploy the personalized model trained in the t-th global training to K clients respectively;

[0012] Step 3. During the local training phase, each client updates and trains the personalized model of the t-th global training based on its own action data to obtain the updated personalized model under the t-th global training.

[0013] Step 4. The server performs personalized aggregation on the updated personalized models on K clients under the t-th global training according to the similarity matrix S, and obtains the K aggregated personalized models under the t-th global training. After these are used as the personalized models for the (t+1)-th global training, they are deployed to K clients respectively.

[0014] Step 5. After assigning t+1 to t, return to step 3 and execute sequentially until t>Tmax. This means that all K clients have completed local update training and aggregation, thus obtaining the personalized model of the Tmax+1th global training, which is used to perform human action recognition on the action data collected by local sensors in real time.

[0015] The human motion recognition method of the present invention is also characterized in that step 2 includes:

[0016] definition Let represent the initial parameters of the personalized model on the k-th client during the t-th global training. This represents the parameters of the aggregated personalized model on the k-th client during the t-th global training.

[0017] The server will Assign to The personalized model aggregated during the t-th global training is used as the personalized model on the k-th client and deployed on the k-th client.

[0018] Step 3 includes:

[0019] Step 3.1. The k-th client, based on the action data ξ k Using equation (1), the action dataset sampled on the k-th client during the t-th global training is calculated.

[0020]

[0021] In equation (1), Random represents the number of steps from ξ. k Randomly select r2×|ξ k |Quantity of action data, r2 represents the sampling ratio;

[0022] Step 3.2. Define the local training epoch as e and initialize e = 1; define the maximum local training epoch as E;

[0023] Step 3.3. Calculate the gradient of the k-th client during the e-th local training under the t-th global training using equation (2).

[0024]

[0025] In equation (2), SGD represents the stochastic gradient descent optimizer. Let represent the loss of the personalized model on the k-th client during the e-th local training under the t-th global training. This represents the parameters of the personalized model on the k-th client during the e-th local training under the t-th global training.

[0026] Step 3.4. Calculate the parameters of the personalized model on the k-th client during the (e+1)-th local training under the t-th global training using equation (4).

[0027]

[0028] In equation (3), This represents the learning rate of the k-th client during the e-th local training under the t-th global training;

[0029] Step 3.5. After assigning e+1 to e, return to step 3.3 and execute sequentially until e>E, thus obtaining the parameters of the personalized model on the k-th client during the E-th local training under the t-th global training. And it is uploaded to the server as the parameters of the personalized model updated in the t-th global training.

[0030] In step 4, the server uses equation (4) to calculate the parameters of the aggregated personalized model on the k-th client during the (t+1)-th global training. And used as the initial parameters for the personalized model in the (t+1)th global training;

[0031]

[0032] In equation (4), This indicates the t-th global training iteration and the E-th local training iteration. The parameters of the personalized model on each client.

[0033] In step 5, the server uses equation (5) to obtain the parameters of the personalized model of the k-th client during the (Tmax+1)-th global training.

[0034]

[0035] In equation (5), This indicates the Tmax-th global training iteration and the E-th local training iteration. The parameters of the personalized model on each client.

[0036] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the human motion recognition methods, and the processor is configured to execute the program stored in the memory.

[0037] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs any of the steps of the human action recognition method.

[0038] Compared to existing federated learning human action recognition schemes, the advantages of this invention are as follows:

[0039] 1. This invention utilizes individual information, such as age, gender, height, and weight, to calculate inter-individual similarity as a substitute for data similarity. When personalizing models for clients, weights are determined by similarity. In this way, local models from individuals with high similarity values ​​will be more helpful in personalizing the target individual than those less similar local models. Simultaneously, using individual information to calculate similarity enhances the model's generalization ability, reduces computational resources, improves training stability, and contributes to the development of personalized federated learning. By utilizing some personal information that users are willing to disclose to achieve human action recognition tasks based on federated learning, the risk of privacy leaks using private data such as medical records is avoided.

[0040] 2. This invention achieves superior performance across all metrics for human motion recognition using wearable sensors. Compared to conventional methods that employ federated learning for human motion recognition, this invention considers the influence of individual information on human motion patterns, such as age affecting standing duration and walking speed, thereby improving the accuracy of human motion recognition. This demonstrates that this invention outperforms other existing methods in this task. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0042] Figure 2 This is a diagram of the overall algorithm structure of the present invention. Detailed Implementation

[0043] The method of this invention, based on federated learning for human action recognition tasks, considers using the similarity of individual client information for personalized model aggregation, thereby reducing communication resources and enabling federated learning-based human action recognition tasks to achieve better classification results when facing more realistic data heterogeneity problems. To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods.

[0044] In this embodiment, as Figure 1 and Figure 2 As shown, a personalized federated learning method for human action recognition based on individual information is performed according to the following steps:

[0045] Step 1. Assume the k-th client has its own individual information d. k and using d k,i Representing individual information d k The information attribute of the i-th dimension;

[0046] Let ξ be the motion data of the local sensor preprocessed by the kth client. k , and ξ k =x k,1 ,y k,1 ),(x k,2 ,y k,2 ),…(x k,s ,y k,s ),…(x k,S ,y k,S ); where x k,s and y k,s Let represent the action sample of the s-th local sensor after preprocessing by the k-th client and its corresponding category, where k = 1, 2, ..., K, K is the total number of clients, and s = 1, 2, ..., S, S is the total number of action samples of the local sensor of the k-th client.

[0047] For individual information d k After standardization, the cosine similarity formula is used to calculate the similarity between the k-th client and the ... Individual information similarity between clients Thus, the similarity matrix S is obtained. This indicates that the two clients are completely similar. This indicates that the two clients are orthogonal;

[0048] Standardized Individual Information: Taking the RealWorld human motion recognition dataset as an example, the RealWorld dataset contains motion data and personal information from 15 individuals. Each subject acts as a client, and their individual information mainly includes gender, age, height, and attributes, denoted as d. k,g d k,a d k,w d k,h Since different attributes have different scales and ranges, they are standardized:

[0049]

[0050] Where, n k,iThis represents the standardized individual information attribute value.

[0051] Calculate the similarity between individuals using cosine similarity: individual k and individual k After the information vector is standardized by equation (1), individual k and individual k are calculated using equation (2). Similarity In practice, the `cosine_similarity` tool from the Python extension package (other methods can also be used in actual operation) is used to analyze individual k and individual k. Calculate similarity

[0052]

[0053] Where ||n|| is the L2 norm of vector n, Where 1 indicates complete similarity and 0 indicates orthogonality.

[0054] Step 2. Define the global training epochs as t, and initialize t = 1; define the global maximum training epochs as Tmax;

[0055] During the t-th global training process, the server will deploy the personalized model trained in the t-th global training to K clients respectively;

[0056] definition Let represent the initial parameters of the personalized model on the k-th client during the t-th global training. This represents the parameters of the aggregated personalized model on the k-th client during the t-th global training.

[0057] The server will Assign to The personalized model aggregated during the t-th global training is used as the personalized model on the k-th client and deployed on the k-th client. In specific implementation, the global model can be a deep learning model such as CNN or RNN for action recognition tasks.

[0058] Step 3. During the local training phase, each client updates and trains the personalized model of the t-th global training based on its own action data to obtain the updated personalized model under the t-th global training.

[0059] Step 3.1. The k-th client, based on the action data ξ k Equation (3) is used to calculate the action dataset sampled on the k-th client during the t-th global training.

[0060]

[0061] In equation (3), Random represents the number of steps from ξ. k Randomly select r2×|ξ k |Quantity of action data, r2 represents the sampling ratio;

[0062] Step 3.2. Define the local training epoch as e and initialize e = 1; define the maximum local training epoch as E;

[0063] Step 3.3. Calculate the gradient of the k-th client during the e-th local training under the t-th global training using equation (4).

[0064]

[0065] In equation (4), SGD represents the stochastic gradient descent optimizer. Let represent the loss of the personalized model on the k-th client during the e-th local training under the t-th global training. This represents the parameters of the personalized model on the k-th client during the e-th local training under the t-th global training. In practice, the optimizer from the Python extension package is used. The optimizer can be either SGD(·) or Adam(·). The loss function is the sparse_categorical_crossentropy loss function. The number of local iterations E can be set differently according to the learning performance of the model. In this example, if CNN is selected as the global model, E can be set to 2; if RNN is selected as the global model, E can be set to 3.

[0066] Step 3.4. Calculate the parameters of the personalized model on the k-th client during the (e+1)-th local training under the t-th global training using equation (4).

[0067]

[0068] In equation (5), This represents the learning rate of the k-th client during the e-th local training under the t-th global training;

[0069] Step 3.5. After assigning e+1 to e, return to step 3.3 and execute sequentially until e>E, thus obtaining the parameters of the personalized model on the k-th client during the E-th local training under the t-th global training. And it is uploaded to the server as the parameters of the personalized model updated in the t-th global training.

[0070] Step 4. The server performs personalized aggregation on the updated personalized models on K clients under the t-th global training according to the similarity matrix S, and obtains the K aggregated personalized models under the t-th global training. These personalized models are then used as the personalized models for the (t+1)-th global training and deployed to the K clients respectively.

[0071] The server uses equation (6) to calculate the parameters of the aggregated personalized model on the k-th client during the (t+1)-th global training. And used as the initial parameters for the personalized model in the (t+1)th global training;

[0072]

[0073] In equation (6), This indicates the t-th global training iteration and the E-th local training iteration. The parameters of the personalized model on each client.

[0074] Therefore, compared to individuals with lower similarity, the local model from highly similar individuals contributes more to the personalized model obtained by client k. This method utilizes information from other individuals while emphasizing information from highly similar individuals, thus making it an effective personalized federated learning method. Because the distribution of activity data across different clients is often unbalanced for human motion recognition, the model learned by the target client is more suitable for its own motion data. In practice, we set the target client's weight to 0.5, and the weights of other individual clients calculated based on similarity are summed to 0.5. While utilizing information from highly similar individuals for personalized aggregation, it is necessary to constrain the proportion of their parameter information, focusing on the model parameter information trained on its own motion data, and avoiding the local model being biased towards heterogeneous data from other clients.

[0075] Step 5. After assigning t+1 to t, return to step 3 and execute sequentially until t>Tmax. This indicates that all K clients have completed local update training and aggregation, thus obtaining the personalized model of the Tmax+1th global training iteration, which is used to perform human action recognition on the action data collected in real time by local sensors. In specific implementation, Tmax can be 100 rounds.

[0076]

[0077] In equation (7), This indicates the Tmax-th global training iteration and the E-th local training iteration. The parameters of the personalized model on each client.

[0078] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0079] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A human action recognition method based on personalized federated learning using individual information, characterized in that, The procedure is as follows: Step 1. Assume the k-th client has its own individual information d. k and using d k，i Representing individual information d k The information attribute of the i-th dimension; Let the motion data of the local sensor preprocessed by the kth client be ξ. k , and ξ k =(x k，1 y k，1 ), (x k，2 y k，2 ), ...(x k，s y k，s ), ...(x k，S y k，S ); where x k，s and y k，s Let represent the action sample of the s-th local sensor after preprocessing by the k-th client and its corresponding category, where k = 1, 2, ..., K, K is the total number of clients, and s = 1, 2, ..., S, S is the total number of action samples of the local sensor of the k-th client. For individual information d k After standardization, the cosine similarity formula is used to calculate the similarity between the k-th client and the ... Individual information similarity between clients Thus, the similarity matrix S is obtained. This indicates that the two clients are completely similar. This indicates that the two clients are orthogonal; Step 2. Define the global training epochs as t, and initialize t = 1; define the global maximum training epochs as Tmax; During the t-th global training process, the server will deploy the personalized model trained in the t-th global training to K clients respectively; Step 3. During the local training phase, each client updates and trains the personalized model of the t-th global training based on its own action data to obtain the updated personalized model under the t-th global training. Step 4. The server performs personalized aggregation on the updated personalized models on K clients under the t-th global training according to the similarity matrix S, and obtains the K aggregated personalized models under the t-th global training. After these are used as the personalized models for the (t+1)-th global training, they are deployed to K clients respectively. Step 5. After assigning t+1 to t, return to step 3 and execute sequentially until t > Tmax. This means that all K clients have completed local update training and aggregation, thus obtaining the personalized model of the Tmax+1th global training, which is used to perform human action recognition on the action data collected by local sensors in real time.

2. The human motion recognition method according to claim 1, characterized in that, Step 2 includes: definition Let represent the initial parameters of the personalized model on the k-th client during the t-th global training. This represents the parameters of the aggregated personalized model on the k-th client during the t-th global training. The server will Assign to The personalized model aggregated during the t-th global training is used as the personalized model on the k-th client and deployed on the k-th client.

3. The human motion recognition method according to claim 2, characterized in that, Step 3 includes: Step 3.

1. The k-th client, based on the action data ξ k Using equation (1), the action dataset sampled on the k-th client during the t-th global training is calculated. In equation (1), Random represents the number of steps from ξ. k Randomly select r2×|ξ k |Quantity of action data, r2 represents the sampling ratio; Step 3.

2. Define the local training epoch as e and initialize e = 1; define the maximum local training epoch as E; Step 3.

3. Calculate the gradient of the k-th client during the e-th local training under the t-th global training using equation (2). In equation (2), SGD represents the stochastic gradient descent optimizer. Let represent the loss of the personalized model on the k-th client during the e-th local training under the t-th global training. This represents the parameters of the personalized model on the k-th client during the e-th local training under the t-th global training. Step 3.

4. Calculate the parameters of the personalized model on the k-th client during the (e+1)-th local training under the t-th global training using equation (4). In equation (3), This represents the learning rate of the k-th client during the e-th local training under the t-th global training; Step 3.

5. After assigning e+1 to e, return to step 3.3 and execute sequentially until e > E, thus obtaining the parameters of the personalized model on the k-th client during the E-th local training under the t-th global training. And it is uploaded to the server as the parameters of the personalized model updated in the t-th global training.

4. The human motion recognition method according to claim 3, characterized in that, In step 4, the server uses equation (4) to calculate the parameters of the aggregated personalized model on the k-th client during the (t+1)-th global training. And used as the initial parameters for the personalized model in the (t+1)th global training; In equation (4), This indicates the t-th global training iteration and the E-th local training iteration. The parameters of the personalized model on each client.

5. The human motion recognition method according to claim 4, characterized in that, In step 5, the server uses equation (5) to obtain the parameters of the personalized model of the k-th client during the (Tmax+1)-th global training. In equation (5), This indicates the Tmax-th global training iteration and the E-th local training iteration. The parameters of the personalized model on each client.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the human motion recognition methods of claims 1-5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it executes the steps of any one of the human motion recognition methods described in claims 1-5.

Images