A method of human posture recognition using millimeter-wave radar based on federated learning

Abstract

A method for human motion recognition using millimeter-wave radar based on federated learning involves using millimeter-wave radar to acquire range-Doppler map data of human motion. This method then extracts shallow feature vectors and temporal correlation information from the radar data using a convolutional neural network and a long short-term memory network. Finally, a classifier network is used to generate detection results. Furthermore, a federated learning architecture is employed to aggregate model training parameters from multiple client devices, continuously updating the global model to improve accuracy. This approach, based on millimeter-wave radar human motion recognition, shifts data storage and model training to the local client, eliminating the need to directly collect user data. Instead, the central model is updated solely through the aggregation of model parameters. This results in a model with high accuracy and improved generalization capabilities while effectively protecting user data privacy.

Description

Technical Field

[0001] The present invention relates to the field of human posture recognition, and in particular to a millimeter-wave radar human posture recognition method based on federated learning. Background Art

[0002] In nursing homes or hospitals, motion recognition is often used to monitor the health status or fall risk of elderly patients. Wearable gesture detection methods often interfere with daily activities or cause users to forget to wear the device. While gesture detection methods based on optical cameras can intuitively capture human movements, they are affected by environmental factors such as indoor light brightness and viewing distance. In contrast, millimeter-wave radar extracts data from radar echo signals. It is a non-contact method that is unaffected by environmental factors such as light. Deep learning methods are used to construct machine learning models with multiple hidden layers. Using large amounts of training data, they learn useful features hidden in radar data, generating deep features for classification and recognition with high accuracy.

[0003] In smart homes, millimeter-wave radar can be used to monitor the health of family members. However, most deep learning-based millimeter-wave radar gesture recognition methods currently require a large number of human gesture examples to train the model and ensure accuracy. One possible approach is to collect the training data required for home health monitoring systems from a large number of clients or households. However, because user gesture data involves personal privacy, collecting and storing health monitoring data to support smart healthcare services based on data analysis poses significant privacy risks, making people reluctant to adopt such services. Federated learning allows users to generate and share trained models without the need for source data aggregation during machine learning. Therefore, as a new research hotspot in the field of cybersecurity, federated learning has attracted considerable attention and research.

[0004] The current solutions are as follows:

[0005] Publication number CN111368930A, titled "A Radar Human Posture Recognition Method and System Based on Multi-Class Spectral Fusion and Hierarchical Learning," describes the method as follows: It uses a stepped-frequency continuous-wave radar to collect current human posture data and processes this data using three time-frequency analysis methods to generate corresponding time-frequency spectrograms. Next, a neural network is trained to form a modified cascade neural network training model to achieve optimal recognition results. This method fuses multiple spectrograms through two-stage training, leveraging the respective strengths of the three time-frequency analysis methods to improve the accuracy of human motion posture recognition.

[0006] The proposed approach utilizes multiple time-frequency analysis methods, integrating different time-frequency spectrograms. This approach, combined through two-stage training, leverages the respective strengths of the three time-frequency analysis methods. However, it fails to consider user data privacy issues during application. Compared to traditional millimeter-wave radar human gesture recognition, this proposed approach utilizes a federated learning architecture to perform gesture recognition locally on the client. This aggregating a large number of client-side model parameters further improves model recognition accuracy, protects user data privacy, and enables large-scale distributed applications.

[0007] Publication number CN113499051A, named "A fatigue detection system based on millimeter-wave radar", the method includes a detection terminal and a remote server. The detection terminal obtains the vital status information of the staff and transmits it to the local processing terminal. The local processing terminal judges and processes the fatigue status according to the machine learning model, and mines the collected typical features and reports them to the remote server to update the machine learning model parameters; the system also includes providing customized detection services to customers.

[0008] The solution employed is to complete data processing and fatigue detection locally, then send typical feature data to a central server. This collected user data is then used to improve model accuracy. While model updates are possible, directly collecting user data poses privacy risks and carries high communication costs. Our approach, however, utilizes a federated learning architecture, where the central server and client only aggregate and distribute model parameters to complete model updates. This effectively protects user data privacy and offers improved communication efficiency. Summary of the Invention

[0009] The technical problems to be solved by the present invention are:

[0010] To address health monitoring challenges in communities and apartments, such as those using human gesture recognition, a method for millimeter-wave radar human gesture recognition based on federated learning was proposed to address issues such as sensor comfort, environmental impact, and privacy concerns associated with training models using large amounts of user data. Millimeter-wave radar uses intermediate-frequency signals to collect human gesture data, which is less susceptible to environmental influences. A neural network model can be used to extract effective features from this data for human gesture recognition. Through a federated learning architecture, only local model parameters need to be uploaded to a central server, effectively enabling human gesture recognition and addressing user data privacy concerns.

[0011] The present invention adopts the following technical solutions to solve the above technical problems:

[0012] The millimeter-wave radar human posture recognition method based on federated learning includes:

[0013] Step S1: The client collects the range Doppler spectrum of human motion as local data through the millimeter wave radar;

[0014] Step S2: The central server sends the global model to the client;

[0015] Step S3: After receiving the global model, the client uses a deep learning method of a neural network to perform human posture recognition on the local data in the global model and forms a local model;

[0016] Step S4: The client sends the local model parameters to the cloud central server, and the central server uses the FedAvg algorithm to weighted aggregate the received client model parameter gradients to complete the global model update;

[0017] Step S5: After multiple rounds of iterations of steps S1-S4, the central server determines whether the global model accuracy meets the requirements or the number of communications reaches the set value to decide whether to continue iterative optimization.

[0018] The method for collecting the range Doppler spectrum of human motion using millimeter wave radar in step S1 is specifically as follows:

[0019] Step 1: The millimeter wave radar transmits a frequency modulated continuous wave. The signal transmitted in one frequency modulation cycle is

[0020]

[0021] Where, is the amplitude of the transmitted signal, =79 GHz is the center frequency, B=4 GHz is the bandwidth, =40us is the signal frequency modulation period.

[0022] Step 2: After being reflected by the target and the environment, the millimeter wave radar receiving antenna obtains the echo signal

[0023]

[0024] Where, is the amplitude of the echo signal, For delay, represents the Doppler shift;

[0025] Step 3: Mix the transmitted signal and the echo signal and low-pass filter to obtain the intermediate frequency signal

[0026]

[0027] Where, Represents the frequency of the intermediate frequency signal at time t.

[0028] Step 4: Sample the intermediate frequency signal. At the mth sampling point of the nth repetition period, the discrete intermediate frequency signal is

[0029]

[0030] Where N is the number of repetition periods in one frame, and M is the number of sampling points in one FM period.

[0031] Step 5: Perform range fast Fourier transform (FFT) and Doppler FFT on the M intermediate frequency signal sampling point data of N signal frequency modulation cycles in one frame, and you can get a frame of range-Doppler distribution heat map of the target scene.

[0032] The method of performing human posture recognition on the local data in the global model through the deep learning method of the neural network in step S3 is specifically as follows:

[0033] Step 1: The human body posture Doppler heat map data obtained locally is segmented in chronological order. The heat map corresponding to each moment is input into the CNN network, and then the 8-channel feature vector sequence is obtained through the CNN network.

[0034] Step 2: In chronological order, the 1-channel feature vector corresponding to each moment is used as the input value of the LSTM network memory module to obtain the output sequence of the LSTM network;

[0035] Step 3: Decompose the output sequence into a multi-dimensional vector input to the fully connected layer, and use the ReLU function to obtain the final prediction result.

[0036] in, is the objective function of the client model

[0037]

[0038] Where, is the local dataset of the k-th client, Is a parameter The model is used for the dataset Examples in ( The resulting loss function.

[0039] By aggregating and calculating the objective function of the global model of the central server of federated learning, and minimizing the global model objective function

[0040]

[0041] The method in step S4 where the central server uses the FedAvg algorithm to weight and aggregate the received client model parameter gradients to complete the global model update is specifically as follows:

[0042] Step 1: The server randomly selects user devices, with a sampling ratio of 10% of the total number of users, and receives the model parameter gradients uploaded by local users;

[0043] Step 2: The server takes the weighted average of the sampled user model parameter gradients and adds them to the model parameters after the previous round of aggregation to update the global model parameters. The model weights of the local client are updated as follows:

[0044]

[0045] The model aggregation update of the communication center server in round t is as follows:

[0046]

[0047] Where, is the model parameter of the t-th round of communication, is the learning rate, K is the total number of clients, n is the total amount of data from all clients, is the data volume of the kth client, For each edge client k in the tth round of communication, calculate the current model parameters of its local data The average gradient under .

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] 1. The present invention uses millimeter-wave radar and a neural network model for human posture recognition. The non-contact method avoids the problems of users wearing wearable devices causing instability in daily life and optical cameras being easily affected by environmental factors such as light. The combination of neural networks (CNN) and LSTM enables the model to achieve a high recognition accuracy rate while ensuring low latency.

[0050] 2. By utilizing the federated learning architecture, we avoid collecting large amounts of user posture data directly on the server side for model training, effectively protecting user privacy and security. We also combine the local data training models from multiple clients, enabling the global model to achieve higher accuracy and better generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a system framework diagram of a millimeter-wave radar human posture recognition method based on federated learning;

[0052] Figure 2 This is a program flow chart of a method for millimeter-wave radar human posture recognition based on federated learning;

[0053] Figure 3 It is a method process in which the client uses neural networks to extract data features for human posture recognition. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0055] like Figure 1 As shown, the millimeter-wave radar human posture recognition method based on federated learning provided by the present invention has a system architecture mainly composed of a central server, a client server, user data and a millimeter-wave radar.

[0056] The central server aggregates model updates submitted by participating clients and averages these updates into its global model, completing the global model update and sending it to edge clients. The client server is deployed at the user end (e.g., indoors at home) and uses collected human posture data to learn the model. The learned model parameters are then sent to the central server. The millimeter-wave radar collects the user's human posture data as a Doppler map, which is stored locally on the user.

[0057] like Figure 2 As shown, the millimeter-wave radar human posture recognition method based on federated learning provided by the present invention mainly includes client data collection, downloading of global model, client local model training, uploading of local model and central server aggregation model.

[0058] Client data collection involves setting radar parameters and designing six human motions (falling, waving, standing up, standing still, walking, and rolling over) to simulate real-world scenarios. Data is collected from various objects, angles, directions, positions, and speeds. Millimeter-wave radars can use a 77-81 GHz frequency modulation bandwidth with a maximum of 4 GHz, with one transmitting antenna and one receiving antenna. The tester performs a range fast Fourier transform (FFT) and a Doppler FFT on the radar's intermediate frequency signal sampling point data to generate a single-frame range-Doppler distribution heat map of the target scene.

[0059] Downloading the global model includes the central server sending the initial global model to the client, or sending the model parameters after each global model update to the client.

[0060] Client-side local model training involves using the global model received from the PC and locally collected human posture data to train the local model. The model structure can utilize CNN, LSTM, and classifier networks to form a range Doppler heat map sequence detection network.

[0061] in, is the objective function of the client model

[0062]

[0063] Where, is the local dataset of the k-th client, Is a parameter The model is used for the dataset Examples in ( The resulting loss function.

[0064] By aggregating and calculating the objective function of the global model of the central server of federated learning, and minimizing the global model objective function

[0065]

[0066] Uploading the local model involves the client server uploading the locally trained model parameter update information to the central server instead of directly transmitting the data, thus avoiding the risk of data privacy leakage.

[0067] The central server aggregates the model and only uploads the gradient information of the model update, and a single global model w is trained on a large number of client devices.

[0068] In the objective function optimization algorithm of federated learning, a large-scale SGD (stochastic gradient descent) algorithm is usually used, that is, the loss function trained by the local client model is multiplied by a fixed learning rate. , calculate the new round of weight update. Therefore, the model weight update of the local client is as follows:

[0069]

[0070] The model aggregation update of the communication center server in round t is as follows:

[0071]

[0072] Where, is the model parameter of the t-th round of communication, is the learning rate, K is the total number of clients, n is the total amount of data from all clients, is the data volume of the kth client, For each edge client k in the tth round of communication, calculate the current model parameters of its local data The average gradient under .

[0073] In the FedAvg (Federated Average) algorithm, each edge client k calculates the current model parameters of its local data. The average gradient under The cloud server then aggregates all the gradients submitted by the client and updates the global model, achieving the same accuracy of training with fewer communication rounds.

[0074] like Figure 3As shown in the figure, the client uses a neural network to extract data features for human posture recognition. The process mainly includes:

[0075] Step 1: The human body posture Doppler heat map data obtained locally is segmented in chronological order. The heat map corresponding to each moment is input into the CNN network, and then the 8-channel feature vector sequence is obtained through the CNN network.

[0076] Step 2: In chronological order, the 1-channel feature vector corresponding to each moment is used as the input value of the LSTM network memory module to obtain the output sequence of the LSTM network;

[0077] Step 3: Decompose the output sequence into a multi-dimensional vector input to the fully connected layer, and use the ReLU function to obtain the final prediction result.

[0078] The raw data image is directly input into the CNN network for feature extraction, effectively extracting the abstract representation of the Doppler heat map through its nonlinear structure. The LSTM uses gating states to control the transmission state, retaining and extracting the temporal correlation between sequence data, thus representing the temporal association information of the sequence data. A classifier is a machine learning model used for classification tasks, whose goal is to predict the category of the input sample based on a set of features or attributes. The CNN has a two-layer structure: the first layer consists of a convolutional layer with a 4-channel 3×3 convolution kernel and a 3×3 maximum pooling layer, and the second layer consists of a convolutional layer with an 8-channel 3×3 convolution kernel and a 3×3 maximum pooling layer. The LSTM network consists of a single hidden layer with a hidden layer dimension of 208. The classifier network includes a split multidimensional vector network, a fully connected layer, and a linear rectifier (ReLU) function.

Claims

1. A method for human posture recognition using millimeter-wave radar based on federated learning, characterized by: Step S1: The client collects the range Doppler spectrum of human motion as local data through the millimeter wave radar; Step S2: The central server sends the global model to the client; Step S3: After receiving the global model, the client uses the CNN and LSTM networks to perform human posture recognition on the local data in the global model and forms a local model. Step S4: The client sends the local model parameters to the cloud central server. The central server uses the FedAvg algorithm to weighted aggregate the received client model parameter gradients to complete the global model update. The method in step S4 where the central server uses the FedAvg algorithm to weight and aggregate the received client model parameter gradients to complete the global model update is specifically as follows: Step 1: The server randomly selects user devices, with a sampling ratio of 10% of the total number of users, and receives the model parameter gradients uploaded by local users; Step 2: The server takes the weighted average of the sampled user model parameter gradients and adds them to the model parameters after the previous round of aggregation to update the global model parameters. The model weights of the local client are updated as follows: ； The model aggregation update of the communication center server in round t is as follows: ； Where, is the model parameter of the t-th round of communication, is the learning rate, K is the total number of clients, n is the total amount of data from all clients, is the data volume of the kth client, For each edge client k in the tth round of communication, calculate the current model parameters of its local data The average gradient under Step S5: After multiple rounds of iterations of steps S1-S4, the central server determines whether the global model accuracy meets the requirements or the number of communications reaches the set value to decide whether to continue iterative optimization.

2. The method for millimeter-wave radar human posture recognition based on federated learning according to claim 1, characterized in that: The method for collecting the range Doppler spectrum of human motion using millimeter wave radar in step S1 is specifically as follows: Step 1: The millimeter wave radar transmits a frequency modulated continuous wave. The signal transmitted in one frequency modulation cycle is ； Where, is the amplitude of the transmitted signal, =79 GHz is the center frequency, B=4 GHz is the bandwidth, =40us is the signal frequency modulation period; Step 2: After being reflected by the target and the environment, the millimeter wave radar receiving antenna obtains the echo signal ； Where, is the amplitude of the echo signal, For delay, represents the Doppler shift; Step 3: Mix the transmitted signal and the echo signal and use low-pass filtering to obtain the intermediate frequency signal ； Where, represents the frequency of the intermediate frequency signal at time t; Step 4: Sample the intermediate frequency signal. At the mth sampling point of the nth repetition period, the discrete intermediate frequency signal is ； Where N is the number of repetition periods in a frame, and M is the number of sampling points in a frequency modulation period; Step 5: For the M intermediate frequency signals containing N signal frequency modulation cycles in one frame, perform range fast Fourier transform (FFT) and Doppler FFT respectively to obtain a range-Doppler distribution heat map of the target scene.

3. The method for human posture recognition using millimeter-wave radar based on federated learning according to claim 1, wherein: The method of performing human posture recognition on the local data in the global model through the deep learning method of the neural network in step S3 is specifically as follows: Step 1: The human body posture Doppler heat map data obtained locally is segmented in chronological order. The heat map corresponding to each moment is input into the CNN network, and then the 8-channel feature vector sequence is obtained through the CNN network. Step 2: In chronological order, the 1-channel feature vector corresponding to each moment is used as the input value of the LSTM network memory module to obtain the output sequence of the LSTM network; Step 3: Decompose the output sequence into a multi-dimensional vector input to the fully connected layer, and use the ReLU function to obtain the final prediction result.

4. The method for human posture recognition using millimeter-wave radar based on federated learning according to claim 1, wherein: A method for human posture recognition using deep learning methods of neural networks in a global model. is the objective function of the client model ； Where, is the local dataset of the k-th client, Is a parameter The model is used for the dataset Examples in ( The resulting loss function; By aggregating and calculating the objective function of the global model of the central server of federated learning, and minimizing the global model objective function 。 5. The method for millimeter-wave radar human posture recognition based on federated learning according to claim 1, characterized in that: In the global model, a deep learning method of neural network is used to perform human posture recognition. The recognized actions include stillness, standing up, walking, waving, falling and turning over.

6. The method for millimeter-wave radar human posture recognition based on federated learning according to claim 4, characterized in that: The central server uses the FedAvg algorithm to weight and aggregate the received client model parameter gradients. The client continuously updates the model accuracy through the local SGD optimization algorithm. When the predetermined number of local training times is reached, the locally trained model parameters are uploaded to the server.

7. The method for millimeter-wave radar human posture recognition based on federated learning according to claim 5, characterized in that: The central server determines whether the global model accuracy meets the requirements. For classification tasks, the objective function is the cross entropy loss. Represents the parameters to be learned, i.e., weights and biases in a deep neural network.

Images