WiFi signal human body activity identification method based on time sequence alignment network

Abstract

The invention discloses a WiFi signal human body activity identification method based on a time sequence alignment network, and relates to the technical field of human body activity identification. Comprising the following steps: acquiring channel state information; dividing into a training set and a test set; constructing a time sequence alignment network model, introducing a deep large-kernel convolution enhanced backbone network to extract depth feature tensors, respectively inputting the depth feature tensors into a double-branch structure, performing local feature processing on an upper branch, converting the feature tensors into global feature vectors through global average pooling operation on a lower branch, and outputting the global feature vectors; performing joint optimization on the time sequence alignment network model by adopting a cross entropy loss function and a ternary loss function, and obtaining a trained time sequence alignment network model after a plurality of times of training; and inputting the test set into the trained time sequence alignment network model, and evaluating a human body activity identification result. According to the method, local feature dynamic matching is carried out on the depth features extracted by the backbone network, so that actions occurring in different time sequences are aligned, and the recognition accuracy is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human activity recognition, and more particularly to a WiFi signal human activity recognition method based on a time sequence alignment network. BACKGROUND

[0002] Human Activity Recognition (HAR) aims to obtain the signal changes caused by different motion states of the human body through sensor devices, and to process them to realize the classification and recognition of activities. HAR has been applied in many fields, such as fall detection, individual recognition, intelligent building, and healthcare system, etc.

[0003] HAR systems can be divided into three categories according to the sensors used: camera-based systems, wearable sensor-based systems, and wireless signal-based systems. Each system has its own advantages and disadvantages. For example, although the camera-based HAR system is convenient and efficient to deploy, it is easily limited by the camera's field of view, external environmental light, and various obstructions in actual use. Similarly, although the wearable sensor-based HAR system has high accuracy, it requires high-cost specialized equipment, and continuous use of wearable devices can cause discomfort to users, resulting in poor user experience in terms of user convenience. At the same time, the above methods inevitably collect sensitive information such as the user's appearance and physiology during the recognition process, which can lead to user privacy leakage. The wireless signal-based HAR system identifies different changes in the wireless signal propagation process caused by different human activities. Since it does not need to collect personal information and wear various devices, it has strong privacy and good user experience. Currently, the wireless signals available for HAR systems are radar, laser, and WiFi signals.

[0004] With the increasing demand for access to the Internet and the popularity of WiFi infrastructure, WiFi signal-based HAR has attracted the attention of researchers in the industry and academia. Compared with other wireless signal technologies, WiFi signals are concentrated in the high frequency band (above 2.4 GHz), so the influence of external light changes on the accuracy of HAR is avoided. At the same time, WiFi signals have the ability to penetrate, reducing the impact of obstructions.

[0005] Human activity recognition (HAR) systems based on WiFi signals mainly utilize two types of signal attributes: received signal strength (RSS) and channel state information (CSI). RSS estimates the distance between the transmitter and receiver based on the received signal power strength, which has shown significant results in indoor positioning and mobile target tracking. However, it struggles to capture the subtle signal fluctuations caused by human motion in complex scenarios. In contrast, CSI records the amplitude, phase, and frequency response of the communication link, providing stronger environmental robustness and measurement accuracy, making it widely used in gesture recognition and wireless sensing.

[0006] Existing systems achieve action recognition by detecting the changes in CSI caused by human behavior. However, the original CSI data is often disturbed by noise and redundancy. To address this issue, Shi Weiguang et al. proposed a dynamic feature enhancement algorithm that filters sub-carrier contributions and separates dynamic components, reducing redundant information and enhancing behavioral feature expression, resulting in a 30.5% improvement in recognition accuracy. Meng Qianxia et al. introduced the Mel-frequency cepstral coefficient (MFCC) into CSI time series analysis, inspired by audio processing methods, and combined it with a hybrid optimization SVM classifier to achieve a 91% recognition rate in complex scenarios. Additionally, Liu Miaomiao et al. used Butterworth filtering and principal component analysis (PCA) to denoise and reduce the dimensionality of CSI data, significantly improving the recognition accuracy of elderly home behavior to 96%. Early research focused on machine learning algorithms to improve the robustness of HAR; in recent years, with the breakthrough progress of neural networks in computer vision and natural language processing, WiFi-HAR research based on neural networks has gradually become mainstream. In terms of network architecture, Huang Bin et al. proposed a model based on Bi-LSTM and attention mechanism, which dynamically allocates feature weights to focus on key time segments, achieving an accuracy of 93% in recognizing six types of actions such as falling and walking. Wu Donghui et al. proposed a spatio-temporal network that integrates attention mechanisms, using a CNN-LSTM-Attention ternary architecture to recognize human behavior: first, CNN extracts spatial local features, then LSTM captures temporal dynamics, followed by attention mechanisms to enhance key features, and finally, a Softmax classifier outputs the recognition results. To break through the precision bottleneck of human motion behavior recognition under limited sensor conditions, Yao Minghui et al. proposed a lightweight method using a twin network, constructing a parameter-shared twin architecture to process pairs of acceleration signal samples, achieving a recognition accuracy of 98.89% on a self-built dataset, significantly better than the baseline model in small sample scenarios.

[0007] Human activity recognition (HAR) technology based on WiFi signals has received widespread attention due to its non-intrusive privacy protection, ease of deployment, and low cost. Existing methods treat input signals as a whole, failing to fully model the temporal dependence of human activity, resulting in limited recognition accuracy.

[0008] Therefore, a WiFi signal human activity recognition method based on a time sequence alignment network is proposed to solve the problems in the prior art, which is a problem that a person skilled in the art urgently needs to solve. SUMMARY

[0009] Therefore, the application provides a WiFi signal human activity recognition method based on a time sequence alignment network, which first extracts a deep feature tensor by introducing a deep large-core convolution enhanced backbone network, then inputs the deep feature tensor into a double-branch structure, the upper branch performs local feature processing, the lower branch converts the feature tensor into a global feature vector through a global average pooling operation, and finally the model is jointly optimized by using a cross-entropy loss function and a ternary loss function.

[0010] To achieve the above object, the application provides the following technical scheme. A WiFi signal human activity recognition method based on a time sequence alignment network, comprising the following steps: S1, collecting signal changes caused by different motion states of a human body through a sensor device to obtain channel state information; S2, preprocessing the obtained channel state information; S3, dividing the preprocessed channel state information into a training set and a test set; S4, constructing a time sequence alignment network model, extracting a deep feature tensor by introducing a deep large-core convolution enhanced backbone network, and inputting the deep feature tensor into a double-branch structure, the upper branch performing local feature processing, the lower branch converting the feature tensor into a global feature vector through a global average pooling operation, the time sequence alignment network model being jointly optimized by using a cross-entropy loss function and a ternary loss function, and obtaining a trained time sequence alignment network model after several times of training; S5, classification and recognition: inputting the test set into the trained time sequence alignment network model to evaluate the human activity recognition result.

[0011] Optionally, the obtained channel state information is subjected to Reshape normalization preprocessing in S2.

[0012] Optionally, the time sequence alignment network model constructed in S4 uses a ResNet-50 convolutional neural network as a backbone network, extracts a deep feature tensor by introducing a deep large-core convolution enhanced backbone network, and inputs the deep feature tensor into a double-branch structure, the upper branch performing local feature processing, and the lower branch converting the feature tensor into a global feature vector through a global average pooling operation. The feature map of the training set input into the backbone network is extracted, and the feature map dimension is assumed to be wherein, is the number of channels, For spatial dimension, the feature maps are fed into the upper and lower branches respectively; In the lower branch processing, the feature tensor is converted into a global feature vector by global average pooling operation, with dimension , let and denote the global feature tensors of signals A and B respectively, and the global feature is defined as:

[0013] The local feature processing is performed by the upper branch. Firstly, the horizontal average pooling is used to convert the feature map into a local feature tensor , and then the dimension is inverted to dimension; let and denote the local feature sets of signals A and B respectively, and before calculating the shortest path distance between A and B, the distance elements are normalized to the interval , and the distance between the first vertical block of the first CSI signal and the first vertical block of the second CSI signal is denoted by , the distance matrix is constructed based on the distance metric , and the element in the matrix is , and the local distance between the two signals is defined as the total distance of the shortest path from the starting point to the ending point in the matrix , and is defined as:

[0014] wherein denotes the cumulative distance of the shortest path from the row to the row in the distance matrix , and is the final shortest path between the two signals, i.e. the total local distance:

[0015] The total distance is calculated by the following formula:

[0016] wherein is the weight coefficient for balancing the global distance and the local distance, and is set to .

[0017] Optionally, the specific content of the joint optimization of the cross-entropy loss function and the triplet loss function in the time sequence alignment network model in S4 is as follows: Assuming that the model output prediction probability distribution is​​ , the real label is encoded , the cross-entropy loss is defined as:

[0018] The triplet loss function used restricts the distance between samples, thereby improving the discriminability of features, and given an anchor sample , a positive sample and a negative sample , defined as:

[0019] wherein, is an interval hyperparameter, ensuring the distance difference between positive and negative samples; The joint loss function is as follows:

[0020] wherein, is a weight coefficient, balancing the classification loss and the metric learning loss.

[0021] Via the technical solution described above, compared with the prior art, the present application provides a WiFi signal human activity recognition method based on a time sequence alignment network, which has the following beneficial effects: (1) The present application proposes a time sequence alignment network for extracting local time sequence features in WiFi signal channel state information, which sequentially sends the deep feature tensor extracted through the backbone network into the upper and lower branches, the upper branch performs local feature processing, and the lower branch converts the feature tensor into a global feature vector through a global average pooling operation, finally adjusts the parameters of the model through the cross-entropy loss function and the triplet loss function, and further improves the recognition accuracy; (2) Experiments show that the recognition accuracy of the time sequence alignment network designed by the present application on the UT-HAR and NTU-HAR data sets reaches 99.26% and 99.24% respectively, and the ablation experiment further verifies the effectiveness of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below, and obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0023] Figure 1 A flowchart of a WiFi signal human activity recognition method based on a time sequence alignment network provided by the present application is shown in the figure. Figure 2A timing alignment network model structure diagram provided by the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0025] Referring to Figure 1 As shown in the figure, the present application discloses a WiFi signal human activity recognition method based on a timing alignment network, comprising the following steps: S1, collecting signal changes caused by different motion states of the human body through a sensor device to obtain channel state information; S2, preprocessing the obtained channel state information; S3, dividing the preprocessed channel state information into a training set and a test set; S4, constructing a timing alignment network model, extracting a deep feature tensor by introducing a deep large kernel convolution enhanced backbone network, and inputting the deep feature tensor into a double-branch structure respectively, performing local feature processing in the upper branch, and converting the feature tensor into a global feature vector through a global average pooling operation in the lower branch, the timing alignment network model is jointly optimized by using a cross-entropy loss function and a ternary loss function, and after several times of training, a trained timing alignment network model is obtained; S5. Classification and identification: inputting the test set into the trained timing alignment network model to evaluate the human activity recognition result.

[0026] Further, the obtained channel state information is subjected to Reshape normalization preprocessing in S2.

[0027] Further, as Figure 2 shown in the figure, the timing alignment network model constructed in S4 takes ResNet-50 convolutional neural network as a backbone network, extracts a deep feature tensor by introducing a deep large kernel convolution enhanced backbone network, and inputs the deep feature tensor into a double-branch structure respectively, performs local feature processing in the upper branch, and converts the feature tensor into a global feature vector through a global average pooling operation in the lower branch. The feature map of the training set input into the backbone network is extracted, and the feature map dimension is assumed to be wherein, is the number of channels, is the spatial size, and the feature map is sent into the upper and lower branches respectively; In the lower branch processing, the feature tensor is converted into a global feature vector through a global average pooling operation, and the dimension is ,set up and Let A and B represent the global feature tensors, respectively. The global features are defined as follows:

[0028] Local feature processing is performed through the upper branch, firstly using horizontal average pooling to... Feature map conversion Local feature tensors, then inverted in dimension to Dimension; Let and Let A and B be the sets of local features respectively. Before calculating the shortest path distance between A and B, the distance elements are normalized to [value]. Interval, using Indicates the first CSI signal The first vertical block and the second CSI signal The distance between vertical blocks is used to construct a distance matrix based on the distance metric. , The element is The local distance between two signals is defined as a matrix. From the starting point To the finish line The total distance of the shortest path is defined as:

[0029] in, Distance matrix Zhong Cong marching to The cumulative distance of the shortest path, while The final shortest path between the two signals is the sum of the local distances:

[0030] The total distance is calculated using the following formula:

[0031] in, To balance the weighting coefficients of global distance and local distance, set .

[0032] Specifically, the input CSI signal is normalized to 256×128 dimensions and then fed into the backbone network to extract feature tensors, which are then downsampled by the deep convolutional residual module.

[0033] The depth-wise (DW) convolution in the depth convolution residual module is a convolution structure with high efficient calculation and strong representation ability. It first performs cross-channel convolution on the features of each channel, and then performs spatial point-by-point convolution on the obtained feature map. In order to further expand the receptive field of the model and reduce the calculation amount, a large kernel depth convolution is introduced in the residual learning framework, and a depth convolution residual module (DRM) and a down-sampling depth convolution residual module (DDRM) are constructed. As shown in Figure 2 , DRM and DDRM both adopt a cross-layer connection structure composed of two 1x1 convolution layers and a 7x7 depth convolution layer. The first 1x1 convolution is responsible for channel dimension lifting and cross-channel information fusion, followed by a large kernel depth convolution layer (kernel size 7x7, depth 3) that effectively captures large-range spatial context information, significantly enhancing the receptive field and context perception ability of the model; the second 1x1 convolution further introduces a nonlinear transformation to improve the feature expression ability. To adapt to the demand of multi-scale feature extraction, DDRM embeds a down-sampling layer in the cross-layer connection to realize efficient compression of the spatial dimension.

[0034] Further, the specific content of the joint optimization of the cross-entropy loss function and the triplet loss function in the time sequence alignment network model in S4 is as follows: Assuming that the model output prediction probability distribution is , and the true label is encoded , the cross-entropy loss is defined as:

[0035] The triplet loss function used constrains the distance between samples, thereby improving the discriminability of the features. Given an anchor sample , a positive sample and a negative sample , it is defined as:

[0036] where is an interval hyperparameter to ensure the distance difference between positive and negative samples; The joint loss function is

[0037] where is a weight coefficient to balance the classification loss and the metric learning loss.

[0038] In one specific embodiment, the following is included: acquiring signal changes caused by different motion states of the human body through a sensor device to obtain channel state information; preprocessing the obtained channel state information; dividing the preprocessed channel state information into a training set and a test set; constructing a time sequence alignment network model (AlignedNet), extracting a deep feature tensor by introducing a deep kernel convolution enhanced backbone network, and inputting the deep feature tensor into a double-branch structure, the upper branch performs local feature processing, and the lower branch converts the feature tensor into a global feature vector through a global average pooling operation, the time sequence alignment network model is jointly optimized by using a cross-entropy loss function and a ternary loss function, and after several times of training, a trained time sequence alignment network model is obtained; inputting the test set into the trained time sequence alignment network model to evaluate the human activity recognition result.

[0039] The data set of the verification experiment of the application is a HAR data set based on channel state information, which is UT-HAR and NTU-HAR respectively.

[0040] UT-HAR is constructed by the University of Toronto, and supports the recognition of seven types of human activities, including lying down (lie down), falling (fall), walking (walk), picking up (pick up), running (run), sitting down (sit down), and standing up (stand up). Data is collected in a fixed experimental scene using an Intel 5300 network interface controller (NIC) in cooperation with a CSI tool. The system configuration includes an access point (Access Point) equipped with a single transmit antenna and a computer equipped with three external receive antennas. Each pair of transmit-receive antennas can collect data at a sampling rate of 1000 Hz on 30 subcarriers, while recording the amplitude and phase information of the CSI data. Due to the relative stability of the amplitude value, the application only uses amplitude information for activity recognition. In order to suppress noise and redundancy, a sliding window is used for segmenting the data, and finally the dimension of the UT-HAR data sample is 3x30x250 (corresponding to the number of receive antennas, the number of subcarriers, and the number of time data packets respectively).

[0041] The NTU-HAR is proposed by Nanyang Technological University, and the data collection uses the Atheros CIS tool, which can capture 114 subcarriers for each pair of antennas under a 40 MHz bandwidth, with higher subcarrier resolution compared to the UT-HAR. The data set contains 1200 samples, covering six types of activities: boxing (box), circling (circle), cleaning (clean), falling (fall), running (run), and walking (walk), each sample consists of 500 data packets, with dimensions of 3x114x500 (corresponding to the number of receiving antennas, subcarriers, and time data packets, respectively).

[0042] The evaluation criterion used in the experiment is the accuracy (ACC), which is defined as follows:

[0043] All experiments were performed on a Lenovo P920 server with the following specifications: CPU Intel Xeon 3204, GPU Nvidia GeForce RTX 3090 Ti 24GB, hard disk 2TB HDD, and memory 28 GB. The operating system is Ubuntu 20.04, and the deep learning framework is Pytorch 1.13.0.

[0044] The input CSI data is uniformly adjusted to 256x128 standardized dimensions by the preprocessing module. For the UT-HAR data set, the preprocessing module consists of three layers of structure: 3x3 convolution layer, ReLU activation layer, 3x3 convolution layer with 3-pixel padding, and ReLU activation layer.

[0045] For the NTU-HAR data set, the preprocessing module contains four layers of structure: 3x3 convolution layer with 3-pixel padding, ReLU activation layer, 3x3 convolution layer, and ReLU activation layer. The model training uses the Adam optimizer to minimize the error, with a learning rate of 0.001 and beta parameters of 0.99 and 0.999. The interval hyperparameter of the triplet loss function is 0.2, and the weight coefficient is 0.5. Due to the difference in the total number of samples between the two data sets, the UT-HAR and NTU-HAR are trained for 200 and 30 cycles, respectively, to ensure model convergence, with a batch size of 16.

[0046] Experiments show that the time alignment network model of the present application improves the recognition accuracy on the UT-HAR and NTU-HAR data sets by 2.05 and 1.58 percentage points, respectively, and the effectiveness of the introduced local feature dynamic matching method is further verified through ablation experiments.

[0047] The various embodiments described in this specification are intended to be illustrative only and in no way limit the scope of the application. Changes and modifications can be made by those skilled in the art, which employ the principles of the application, without departing from the scope of the application. Accordingly, the application is not limited to the embodiments described herein, but instead has scope to encompass any choice whatsoever that is dependent on, or can be substituted in, the principal, new and inventive features that are described and claimed herein.

[0048] The above description of disclosed embodiments is intended to be illustrative only and not limiting of the application. Numerous modifications to these embodiments can be made by those skilled in the art without departing from the spirit or scope of the application. The scope of the application is not limited to the embodiments described herein, but rather extends to any that are dependent on, or can be substituted in, the principal, new and inventive features that are described and claimed herein.

Claims

1. A method for identifying human activity via WiFi signals based on time-aligned networks, characterized in that, Includes the following steps: S1. Collect signal changes caused by different human motion states through sensor devices to obtain channel state information; S2. Preprocess the acquired channel state information; S3. Divide the preprocessed channel state information into a training set and a test set; S4. Construct a temporal alignment network model. By introducing a backbone network enhanced by deep large kernel convolution, deep feature tensors are extracted and input into a two-branch structure. The upper branch performs local feature processing, and the lower branch transforms the feature tensors into global feature vectors through global average pooling. The temporal alignment network model is jointly optimized using cross-entropy loss function and ternary loss function. After several training iterations, a trained temporal alignment network model is obtained. S5. Classification and Recognition: Input the test set into the trained temporal alignment network model and evaluate the human activity recognition results.

2. The method for identifying human activity via WiFi signals based on time-aligned networks according to claim 1, characterized in that, In S2, the acquired channel state information is preprocessed using Reshape normalization.

3. The WiFi signal human activity recognition method based on a time-aligned network according to claim 1, characterized in that, The temporal alignment network model constructed in S4 uses a ResNet-50 convolutional neural network as its backbone. It extracts deep feature tensors by introducing a deep, large-kernel convolutional network and inputs them into a two-branch structure. The upper branch performs local feature processing, while the lower branch transforms the feature tensors into global feature vectors through global average pooling. The specific details are as follows: Extract the feature maps from the training set of the input backbone network, assuming the feature map dimension is... ,in, For the number of channels, For spatial dimensions, the feature maps are fed into the upper and lower branches respectively; In the next branch, the feature tensor is transformed into a global feature vector with dimension 1 through a global average pooling operation. ,set up and Let A and B represent the global feature tensors, respectively. The global features are defined as follows: Local feature processing is performed through the upper branch, firstly using horizontal average pooling to... Feature map conversion Local feature tensors, then inverted in dimension to Dimension; Let and Let A and B be the sets of local features respectively. Before calculating the shortest path distance between A and B, the distance elements are normalized to [value]. Interval, using Indicates the first CSI signal The first vertical block and the second CSI signal The distance between vertical blocks is used to construct a distance matrix based on the distance metric. , The element is The local distance between two signals is defined as a matrix. From the starting point To the finish line The total distance of the shortest path is defined as: in, Distance matrix Zhong Cong marching to The cumulative distance of the shortest path, while The final shortest path between the two signals is the sum of the local distances: The total distance is calculated using the following formula: in, To balance the weighting coefficients of global distance and local distance, set .

4. The WiFi signal human activity recognition method based on a time-aligned network according to claim 1, characterized in that, The specific details of the joint optimization of the time-aligned network model in S4 using the cross-entropy loss function and the ternary loss function are as follows: Assume the model outputs a predicted probability distribution as follows: The actual label is an encoding. Then the cross-entropy loss is defined as: The triplet loss function used constrains the distance between samples, thereby improving feature discriminativeness, given anchor samples. Positive samples and negative samples Defined as: in, This is a hyperparameter used to ensure the difference in distance between positive and negative samples; The joint loss function is expressed as follows: in, The weighting coefficients balance the classification loss and the metric learning loss.

Images