Multimodal collaborative behavior recognition method based on wi-fi signals and skeletal sequences

Abstract

This invention discloses a multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences. The method includes time-synchronized sensor data acquisition, employing conjugate multiplication and Butterworth bandpass filters to eliminate random phase errors, high and low frequency noise, and static components in the CSI (Continuous Sensor Indicator) signal. PCA is used to reduce the dimensionality of the denoised signal. Finally, short-time Fourier transform is used to perform time-frequency analysis on the first principal component to obtain a Doppler spectrum. A human pose estimation algorithm is used to obtain a skeleton sequence from video data acquired by a Kinect camera. Connections between key points of the human body are established based on the human physical structure and the relationship between consecutive time frames to construct a spatiotemporal skeleton map. Multimodal cooperative training is performed to establish an auxiliary relationship between the Wi-Fi signal and the skeleton sequence. Single-Wi-Fi modality testing is then conducted to complete the behavior recognition task. This invention solves the problem of traditional multimodal fusion strategies relying on data from all modalities during the testing phase.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition, and more particularly to a multimodal collaborative behavior recognition method based on Wi-Fi signals and skeleton sequences in the field of human behavior recognition. Background Technology

[0002] In recent years, with the rapid development and accelerated integration of technologies such as big data, the Internet of Things (IoT), and artificial intelligence (AI), the Artificial Intelligence of Things (AIoT) has been widely applied to various aspects to solve real-life problems. Human behavior recognition, as a key technology for realizing human-computer interaction, has become a research hotspot in the fields of computer vision and artificial intelligence. Humans are the main body of the real world, and human behavior, as the most basic element constituting life, contains rich information about daily activities. Therefore, accurate perception and understanding of human behavior is crucial. Human behavior recognition technology aims to perceive the external behavior of target individuals in real-world scenarios, analyze and identify the perception results, enabling computers to understand human behavior and assign labels. The general process of human behavior recognition technology is as follows: using sensors to detect and collect human activities, performing data preprocessing and normalization operations using specific methods, and extracting features through methods such as pattern recognition and machine learning to finally obtain classification results.

[0003] Action recognition technology is constantly innovating and developing for different application scenarios, greatly improving accuracy and efficiency. However, the inherent limitations of sensing devices make single-modal action recognition methods suffer from intractable drawbacks: wearable device-based methods are suitable for active monitoring scenarios, but require users to carry specific devices, creating an additional burden and limiting their application scope; RGB video or skeleton-based methods require cameras to acquire data, are easily affected by lighting conditions, and have issues such as infringing on user privacy; Wi-Fi signal-based methods, as a non-visual wireless sensing method, solve the privacy infringement problem, but because Wi-Fi signals are highly sensitive to environmental changes, the model's generalization ability is usually poor. Therefore, how to balance the accuracy and robustness of action recognition in real-world scenarios has become a significant challenge for human behavior recognition. Compared with single-modal data, multimodal data can provide multi-level and multi-perspective information, thus more comprehensively depicting human behavior, which has important theoretical value and practical significance for human behavior recognition research. Summary of the Invention

[0004] To address the aforementioned existing technologies, this invention provides a multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences. It constructs a multimodal cooperative training framework based on Wi-Fi signals and skeleton sequences, leveraging the complementary advantages between Wi-Fi signals and skeleton sequences. A deep learning network is used to capture complementary knowledge from multimodal data, enabling cooperation between the two modalities. This ultimately improves the testing performance of single-modal Wi-Fi networks and solves the problem of traditional multimodal fusion strategies relying on all modal data during the testing phase. The method constructs a behavior recognition network that combines accuracy, generalization ability, and privacy protection.

[0005] To address the aforementioned technical problems, this invention proposes a multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences, comprising the following steps:

[0006] Step 1: Deploy commercial Wi-Fi devices and Kinect cameras in a real-world scenario, simultaneously collecting CSI data packets and video data;

[0007] Step 2: Use device clock source synchronization and data timestamp synchronization methods to complete the time alignment of CSI data packets and video data in order to accurately obtain the changes in the user's motion state during the activity;

[0008] Step 3: Data preprocessing, including: For CSI collected by commercial Wi-Fi devices, firstly, the conjugate multiplication method and Butterworth bandpass filter are used to eliminate random phase errors, high and low frequency noise and static components in the CSI, and PCA is used to reduce the dimensionality of the denoised signal. Finally, short-time Fourier transform is used to perform time-frequency analysis on the first principal component to obtain the Doppler spectrum. For video data collected by Kinect cameras, human pose estimation algorithm is used to obtain skeleton sequences from the video data collected by Kinect cameras, and connections between human key points are established based on human physical structure and continuous time frame relationships to construct a spatiotemporal skeleton map.

[0009] Step 4: Multimodal collaborative training, including: In the model training phase, a multimodal collaborative approach is adopted, with Wi-Fi signals as the primary mode and skeleton sequences as the secondary mode. The Doppler spectrum and spatiotemporal skeleton obtained in Step 3 are used as inputs to the model. Feature extraction networks are designed for the primary mode and secondary mode respectively. A multi-scale feature fusion module is used to fuse the multi-scale features at different stages in the two feature extraction networks to obtain multi-level semantic features. Finally, complementary knowledge transfer is completed through spatiotemporal semantic alignment, enabling the model to learn the complementary knowledge between the primary mode and secondary mode, thereby establishing the auxiliary relationship between Wi-Fi signals and skeleton sequences.

[0010] Step 5: Single Wi-Fi Modality Testing, including: In the model testing phase, using only Wi-Fi modality data as input, the behavior recognition task is completed using a single-modality Wi-Fi network with embedded complementary knowledge.

[0011] Furthermore, the multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences described in this invention, wherein:

[0012] Step two includes the following steps:

[0013] 2-1) Unified clock source: UTC time is used as the unified time source to provide the reference time, and a high-precision clock is set on the commercial Wi-Fi device and Kinect camera. The clock time of the commercial Wi-Fi device and Kinect camera is calibrated according to the reference time using the IEEE 1588 clock synchronization protocol to ensure that the clock sources of the two devices are synchronized.

[0014] 2-2) Timestamp Synchronization: After the clock sources of the two devices are synchronized, the commercial Wi-Fi device and Kinect camera will record timestamp information for each data packet acquired according to the calibrated time, and use the timestamp alignment method to synchronize the data at each moment; for each video frame acquired, the CSI data packet and the video frame are aligned once using the interpolation method.

[0015] Step four includes the following specific steps:

[0016] 4-1) Design a feature extraction network for the main modality, using ResNet-34 as the feature extraction network for this main modality. The residual structure is represented as follows:

[0017] y=F(x)+x (4)

[0018] In equation (4), x represents the input of the residual block, y represents the output of the residual block, and F(x) represents the output of each layer of the neural network. The ResNet-34 network consists of 5 stages. Stage 0 uses a 7×7 convolution to perform image transformation, which is the initial feature extraction of the input image. Stages 1 to 4 are composed of 3, 4, 6, and 3 BasicBlocks, respectively.

[0019] 4-2) Design a feature extraction network for auxiliary modalities. Using the spatiotemporal skeleton graph obtained in step 3 as input, extract high-level semantic features from the skeleton sequence through an attention spatiotemporal graph convolutional network. The core modules of this network are the adjacency matrix attention module and the temporal attention module.

[0020] Adjacency Matrix Attention Module: Uses the spatiotemporal skeleton graph as input feature f in ∈R C×T×NWhere C represents the number of channels, T represents the total number of time frames, and N represents the total number of key points; through two embedding functions θ(*) and The input is mapped to different feature spaces and converted to R respectively. N×CT and R CT×N The feature matrices are obtained by multiplying them and then normalizing the matrix elements to arbitrary values ​​between 0 and 1 using a softmax operation, resulting in the correlation matrix B, where the elements b are... ij Represents vertex v i and vertex v j The correlation between them is shown below:

[0021]

[0022] In equation (5), the embedding function θ(*) and Implemented using two 1×1 convolutional layers, W θ and W φ The functions θ(*) and The parameters are: softmax is the activation function; the correlation matrix B is added to the original adjacency matrix A to obtain a new adaptive adjacency matrix, which is then compared with the input feature f. in Multiply and weight the results, assigning different weights to different key points, to obtain the output:

[0023] f out =Wf in (A+B) (6)

[0024] In equation (6), W is the parameter of the 1×1 convolutional layer;

[0025] Time Attention Module: Compression Operation F sq (*) The spatial dimension of the input feature x of size C×T×N is compressed to 1 by global average pooling, resulting in a 1×1×T feature vector U:

[0026]

[0027] Excitation operation F ex (*, W), using a fully connected layer to perform a non-linear transformation on the compressed feature map of size 1×1×T to obtain the weight vector S for each time frame:

[0028] S = F ex (U, W) = σ(W2δ(W1)U) (8)

[0029] In equation (8), σ(*) represents the Sigmoid activation function, δ(*) represents the ReLU function, W1 and W2 represent the weights of the two fully connected layers, respectively; finally, there is the weighting operation F. scale(*, *), multiply the contribution weight S of each time frame by the input feature to generate the weighted output feature:

[0030]

[0031] In equation (9), u t and s t Elements with characteristics U and S;

[0032] 4-3) Multi-scale feature fusion module; extracts shallow features F from CSI feature extraction network and skeleton feature extraction network at different network depths. s_csi F s_ske and deep features F d_csi F d_ske The fused features are represented as follows:

[0033]

[0034]

[0035] In equations (10) and (11), These represent the weight parameters of the four fully connected layers. These represent the weight parameters of two 1×1 convolutional layers, and concat indicates the connection operation.

[0036] 4-4) Complementary knowledge transfer is achieved through spatiotemporal semantic alignment, including:

[0037] 4-4-1) The following correlation matrix is ​​used to represent the correlation among all elements of a feature:

[0038]

[0039]

[0040] In equations (12) and (13), Let represent normalized matrices, and d represent the matrix dimension.

[0041] 4-4-2) Use spatiotemporal alignment loss to characterize the differences between different modalities, and minimize the spatiotemporal alignment loss during training to learn shared semantic knowledge:

[0042]

[0043] In equation (14), ρ is a regularization parameter used to prevent negative transfer; large differences between different modalities may negatively impact network performance; classification loss is used to measure the accuracy of the network's learned content, and the classification loss of two modalities is used. difference To measure differences in network performance; the regularization parameter ρ is defined based on the difference in classification loss:

[0044]

[0045] In equation (15), S(·) is the zero threshold function; when the difference ΔL is positive, knowledge transfer is performed; otherwise, knowledge transfer is not performed.

[0046] 4-4-3) The backpropagation process is constrained by a joint loss function to achieve spatiotemporal semantic alignment, thereby establishing an auxiliary relationship between the Wi-Fi signal and the skeleton sequence; the joint loss function is obtained by adding the classification loss and the spatiotemporal alignment loss.

[0047]

[0048] in, Let M represent the classification loss, M represent the total number of modes, and λ be a hyperparameter.

[0049] Step 5 includes the following steps: using the ResNet-34 already trained in Step 4 as the feature extraction network for the Wi-Fi modality during the testing phase; and using only CSI data packets collected by commercial Wi-Fi devices as test data, taking the spectrogram generated by CSI as input, and inputting it into the aforementioned trained ResNet-34, thereby completing the behavior recognition task.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] (1) The complementary advantages between Wi-Fi signals and skeleton sequences were discovered and utilized. Based on this, a multimodal collaborative training method based on Wi-Fi signals and skeleton sequences was proposed. This method solves the problem of improving the test performance of single-mode networks by utilizing multimodal complementary knowledge, and makes up for the lack of spatial information and environmental dependence in Wi-Fi signals. Without increasing the model complexity during testing, it improves the performance and generalization ability of single Wi-Fi mode networks.

[0052] (2) Based on the data format of the skeleton sequence and the topological structure of the human body, an adaptive and attention mechanism was adopted to design an attention spatiotemporal graph convolutional network (AST-GCN). The specific spatial relationships and temporal information between joints in consecutive frames in the skeleton sequence, the potential connection relationships between joints in various parts of the human body, and the long-distance dependencies between consecutive time frames were modeled, which fully explored the human motion information and spatial information contained in the skeleton sequence.

[0053] (3) The multi-scale feature fusion module extracts and fuses shallow and deep features at different depths in the network. Shallow features contain more local details of the original input, while deep features contain high-dimensional global abstract semantic information after nonlinear transformation. By fusing the extracted shallow and deep features, it is possible to extract high-level semantic information while including as much detailed information as possible from the original input data, thereby improving the network's ability to understand features. Attached Figure Description

[0054] Figure 1 This is the overall framework of the multimodal behavior recognition method based on Wi-Fi signals and skeleton sequences proposed in this invention;

[0055] Figure 2 This is a plan view of the experimental scenario in the multi-sensor data acquisition step of the present invention;

[0056] Figure 3 This is a flowchart of the method of the present invention;

[0057] Figure 4 This is a schematic diagram of the six actions collected in the multi-sensor data acquisition steps of the present invention;

[0058] Figure 5 This is a comparison chart of the CSI amplitude before and after processing in the data preprocessing step of this invention;

[0059] Figure 6 This is the Doppler spectrum diagram in the data preprocessing step of the present invention;

[0060] Figure 7 This is a modeling diagram of human key points and spatiotemporal skeleton diagram in the data preprocessing step of the present invention;

[0061] Figure 8 This is a network model diagram in the multimodal collaborative training step of the present invention;

[0062] Figure 9 This is a diagram of the Wi-Fi modality feature extraction network structure in the multimodal collaborative training step of this invention;

[0063] Figure 10 This is a diagram of the skeleton modality feature extraction network structure in the multimodal collaborative training step of this invention. Detailed Implementation

[0064] like Figure 1 and Figure 3 As shown, this invention proposes a multimodal cooperative behavior recognition method that integrates Wi-Fi signals and skeleton sequences, mainly including the following steps:

[0065] Step 1: Deploy commercial Wi-Fi devices and Kinect cameras in a real-world scenario, and simultaneously collect CSI data packets and video data.

[0066] Step Two: Time Synchronization. To ensure that the data from the commercial Wi-Fi device and the Kinect camera are consistent in time, device clock source synchronization and data timestamp synchronization methods are used to complete the time alignment of CSI data packets and video data, so as to accurately capture the changes in the user's motion state during the activity.

[0067] Step 3: Data preprocessing, including: For CSI data collected by commercial Wi-Fi devices, firstly, the conjugate multiplication method and Butterworth bandpass filter are used to eliminate random phase errors, high and low frequency noise, and static components in the CSI. Then, PCA is used to reduce the dimensionality of the denoised signal. Finally, short-time Fourier transform is used to perform time-frequency analysis on the first principal component to obtain the Doppler spectrum. For video data collected by Kinect cameras, a human pose estimation algorithm is used to obtain the skeleton sequence from the video data collected by the Kinect cameras. Connections between key points of the human body are established based on the human physical structure and the relationship between consecutive time frames to construct a spatiotemporal skeleton map. The data preprocessing steps are as follows:

[0068] 3-1) Noise reduction of the original CSI using the conjugate multiplication method:

[0069]

[0070] Equation (1) is divided into four sub-terms:

[0071] First item This represents the product of the static path components in the CSI received from the two links. By definition, when t is very small, this part can be considered a constant; the second term... This represents the product of the dynamic path components in the CSI received from the two links; it is usually very small and can be ignored. The third term... This represents the product of the dynamic path components of antenna 1 and antenna 2; the fourth term. This represents the product of the dynamic path component of antenna 2 and the static path component of antenna 1.

[0072] 3-2) A fifth-order Butterworth bandpass filter is used to remove high-frequency and low-frequency noise from the signal, with the lowest cutoff frequency set to 10Hz and the highest cutoff frequency set to 80Hz. Simultaneously, the values ​​are normalized to between -1 and 1 to eliminate dimensional effects.

[0073] 3-3) Principal component analysis (PCA) is used to eliminate data redundancy in CSI. The first principal component after dimensionality reduction is used for subsequent data analysis, while other components with low contribution are ignored.

[0074] 3-4) The Doppler frequency shift of the signal reflected from the human body can be expressed as:

[0075]

[0076] By performing time-frequency analysis on CSI using short-time Fourier transform (STFT), the waveform can be converted into a Doppler spectrum.

[0077] 3-5) Use the MediaPose human pose estimation algorithm under the MediaPipe project to extract human joints;

[0078] 3-6) Using human joints as nodes and human skeletons as edges, construct an undirected spatiotemporal graph G from the skeleton sequence with N joints over T time frames:

[0079] G = (V, E) (3)

[0080] In equation (3), the node set V = {v ti The sequence |t=1,...T,i=1,...N} includes all keypoints in the skeleton sequence, v ti Let E represent the i-th node in the t-th frame of graph G; edge set E = {E s E F} describes the connection relationships between the nodes in graph G, and is derived from E S E F It consists of two subsets, E S ={v ti v tj |(i,j)∈H} represents intra-frame edges, indicating the connection relationships between nodes within the same frame at the same time, where H represents the natural connection structure of the human body; E F ={v ti v (t+1)i} represents the inter-frame edge, indicating the connection relationship between the same node in adjacent frames, and describes the motion trajectory of the joint point i over time.

[0081] Step 4: Multimodal Collaborative Training, including: In the model training phase, a multimodal collaborative approach is adopted, using Wi-Fi signals as the primary mode and skeleton sequences as the auxiliary mode. The Doppler spectrum and spatiotemporal skeleton obtained in Step 3 are used as inputs to the model. Separate feature extraction networks are designed for the primary and auxiliary modes, respectively. A multi-scale feature fusion module is used to fuse multi-scale features at different stages from the two feature extraction networks to obtain multi-level semantic features. Finally, complementary knowledge transfer is achieved through spatiotemporal semantic alignment, enabling the model to learn the complementary knowledge between the primary and auxiliary modes, thereby establishing an auxiliary relationship between Wi-Fi signals and skeleton sequences, achieving the goal of multimodal collaboration. Specific steps include:

[0082] 4-1) Design a feature extraction network for the main modality. A 34-layer residual network ResNet-34 is used as the feature extraction network for this main modality (i.e., the Wi-Fi modality). The residual structure is represented as follows:

[0083] y=F(x)+x (4)

[0084] In equation (4), x represents the input of the residual block, y represents the output of the residual block, and F(x) represents the output of each layer of the neural network. The ResNet-34 network consists of 5 stages. Stage 0 uses a 7×7 convolution to perform image transformation, which can be regarded as the preliminary feature extraction of the input image. Stages 1 to 4 are composed of 3, 4, 6, and 3 BasicBlocks, respectively.

[0085] 4-2) Design a feature extraction network for auxiliary modalities. Using the spatiotemporal skeleton graph obtained in step 3 as input, extract high-level semantic features from the skeleton sequence through the attention spatiotemporal graph convolutional network (AST-GCN). The core modules of this network are the adjacency matrix attention (ASE) module and the temporal attention (TSE) module.

[0086] ASE module: uses the spatiotemporal skeleton graph as input feature f in ∈R C×T×N Where C represents the number of channels, T represents the total number of time frames, and N represents the total number of key points; through two embedding functions θ(*) and The input is mapped to different feature spaces and converted to R respectively. N×CT and R CT×N The feature matrices are obtained by multiplying them and then normalizing the matrix elements to arbitrary values ​​between 0 and 1 using a softmax operation, resulting in the correlation matrix B, where the elements b are... ij Represents vertex v i and vertex v j The correlation between them is shown below:

[0087]

[0088] In equation (5), the embedding function θ(*) and Implemented using two 1×1 convolutional layers, W θ and W φ The functions θ(*) and The parameters are: softmax is the activation function; to enhance the model's flexibility without sacrificing its performance, the correlation matrix B is added to the original adjacency matrix A to obtain a new adaptive adjacency matrix, which is then compared with the input feature f. in Multiply and weight the results, assigning different weights to different key points, to obtain the output:

[0089] f out =Wf in (A+B) (6)

[0090] In equation (6), W is the parameter of the 1×1 convolutional layer.

[0091] TSE module: Compression operation F sq (*) The spatial dimension of the input feature x of size C×T×N is compressed to 1 by global average pooling, resulting in a 1×1×T feature vector U:

[0092]

[0093] Excitation operation F ex (*, W), using a fully connected layer to perform a non-linear transformation on the compressed feature map of size 1×1×T to obtain the weight vector S for each time frame:

[0094] S = F ex (U, W) = σ(W2δ(W1)U) (8)

[0095] In equation (8), σ(*) represents the Sigmoid activation function, δ(*) represents the ReLU function, W1 and W2 represent the weights of the two fully connected layers, respectively; finally, there is the weighting operation F. scale (*, *), multiply the contribution weight S of each time frame by the input feature to generate the weighted output feature:

[0096]

[0097] In equation (9), u t and s t Elements of features U and S.

[0098] 4-3) Multi-scale feature fusion module; extracts shallow features F from CSI feature extraction network and skeleton feature extraction network at different network depths. s_csi F s_ske and deep features F d_csi F d_ske The shallow features contain more local details of the 3D velocity contour and spatiotemporal skeleton. The deep features contain higher-dimensional global abstract semantic information. They are mapped to the same feature dimension through four fully connected layers, and multi-scale feature fusion is performed using convolution operations. This allows the fused features to extract high-level semantic information while containing as much detail as possible from the original data. The fused features can be represented as follows:

[0099]

[0100]

[0101] In equations (10) and (11), These represent the weight parameters of the four fully connected layers. These represent the weight parameters of two 1×1 convolutional layers, and concat represents the connection operation; the multi-scale feature fusion module simultaneously aggregates low-level modal features and high-level complementary semantic features.

[0102] 4-4) Complementary knowledge transfer is achieved through spatiotemporal semantic alignment, including:

[0103] 4-4-1) The following correlation matrix is ​​used to represent the correlation among all elements of a feature:

[0104]

[0105]

[0106] In equations (12) and (13), Let represent normalized matrices, and d represent the matrix dimension.

[0107] 4-4-2) Use spatiotemporal alignment loss to characterize the differences between different modalities, and minimize the spatiotemporal alignment loss during training to learn shared semantic knowledge:

[0108]

[0109] In equation (14), ρ is a regularization parameter used to prevent negative transfer. Large differences between different modalities can negatively impact network performance. Classification loss can be used to measure the accuracy of the content learned by the network. A smaller classification loss indicates more accurate learning. In this invention, the cross-entropy function is used as the classification loss, and the classification loss of two modalities is used. difference This is used to measure the difference in network performance. A positive difference indicates that the current skeleton feature extraction network has learned more effective information, allowing for knowledge transfer. A negative difference indicates poor accuracy in learning the skeleton modalities, negatively impacting the CSI network performance. The regularization parameter ρ is defined based on the difference in classification loss.

[0110]

[0111] In equation (15), S(·) is the zero threshold function; knowledge transfer is performed when the difference ΔL is positive, otherwise, no knowledge transfer is performed. The correlation matrices of the skeleton modality and the CSI modality are semantically aligned to control the contribution weight of each modality's features to the shared knowledge; the cross-entropy loss function is used for classification loss, so the classification loss of the Wi-Fi modality feature extraction network ResNet-34 is:

[0112]

[0113] The classification loss of the skeleton modality feature extraction network AST-GCN is:

[0114]

[0115] In equations (16) and (17), Q csi Represents the Wi-Fi modality training set, Q ske This represents the skeleton modality training set, where M is the total number of classes, N is the total number of samples, and y i，c Indicates sample x i Real labels, p i，c This represents the predicted label of the model.

[0116] 4-4-3) The backpropagation process is constrained by a joint loss function to achieve spatiotemporal semantic alignment, thereby establishing an auxiliary relationship between the Wi-Fi signal and the skeleton sequence; the joint loss function is obtained by adding the classification loss and the spatiotemporal alignment loss.

[0117]

[0118] in, Let M represent the classification loss, M represent the total number of modes, and λ be a hyperparameter.

[0119] Step 5: Single Wi-Fi Modality Testing, including: In the model testing phase, using only Wi-Fi modality data as input, the behavior recognition task is completed using a single-modality Wi-Fi network with embedded complementary knowledge.

[0120] Research materials:

[0121] The complete process of using the multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences proposed in this invention to handle behavior recognition tasks in real-world scenarios can be found in [link to full documentation]. Figure 1 and Figure 3 Specifically, it includes the following steps:

[0122] I. In a real-world scenario, using commercial Wi-Fi devices and Kinect cameras to simultaneously collect CSI and video motion samples, the specific process is as follows:

[0123] 1-1) Two ThinkPad T-series laptops equipped with Intel 5300 wireless network cards and a Kinect camera were used to collect Wi-Fi signals and video data. Both laptops were running Ubuntu 16.04. One laptop had one external antenna as the transmitter, and the other had three external antennas as the receiver. Both laptops had the Linux 802.11n CSI Tool, a CSI data collection tool based on the IEEE 802.11n protocol, installed and set to run in Monitor mode on channel 161 in the 5GHz band. Each pair of transceiver antennas could collect 30 orthogonal frequency division multiplexing subcarriers. The system sampling frequency was 1000Hz. Ignoring packet loss, the three antennas at the receiver received one 90×1000 CSI signal per second. The Kinect camera parameters were set to 1920×1080 pixels and a frame rate of 30fps.

[0124] 1-2) See the experimental scenario. Figure 2 The environments are an empty room, an office, and a conference room. The gray area represents the area where the experimental equipment is deployed, with a floor plan dimension of 2.5m × 1.2m; the distance between the transmitting and receiving ends is 2.5m, and the height from the ground is approximately 80cm. The first environment is an empty room, with a floor plan dimension of 5.0m × 4.0m, containing no items other than the experimental equipment. The second environment is an office, with a floor plan dimension of 5.8m × 4.3m, containing three tables, two chairs, and a row of bookshelves in addition to the experimental equipment. The third environment is a conference room, with a floor plan dimension of 6.0m × 5.6m, containing a conference table and several chairs in addition to the experimental equipment. See the schematic diagram of the six common action samples collected in the experiment. Figure 4 These include pushing, sliding, clapping, drawing circles, walking, and squatting.

[0125] 1-3) Ten volunteers were invited, five males and five females, aged 18-32, with heights ranging from 150-185cm and weights from 42-85kg. Each volunteer repeated the above six actions in three different experimental environments, with each action lasting 5 seconds, for 50 repetitions, resulting in a total of 9000 action samples, which served as the experimental dataset.

[0126] 2. Synchronize the acquired raw CSI and video data. This mainly includes unifying the clock source and synchronizing timestamps.

[0127] 2-1) Unified Clock Source. To eliminate the time difference caused by clock drift between the internal clocks of each sensor, UTC time is used as a unified time source to provide a reference time. A high-precision clock is set on the commercial Wi-Fi device and the Kinect camera. The clock time of the commercial Wi-Fi device and the Kinect camera is calibrated according to the reference time using the IEEE 1588 clock synchronization protocol to ensure that the clock sources of the two devices are synchronized.

[0128] 2-2) Timestamp Synchronization. After the device clock source is synchronized, the commercial Wi-Fi device and Kinect camera record timestamp information for each data packet acquired according to the calibrated time, and use a timestamp alignment method to synchronize the data at each moment. The commercial Wi-Fi device has a sampling rate of 1000Hz, which means that the three pairs of transceiver antennas can acquire 30 orthogonal frequency division multiplexing subcarriers per second, totaling 90×1000 CSI data packets; the Kinect camera has a sampling frame rate of 30fps, which means that it acquires 30 video frames per second. Therefore, for each video frame acquired, interpolation is used to align the CSI and the video frame to ensure that the acquired CSI and video data are synchronized in time, thereby more accurately analyzing the changes in human motion state in the CSI and video.

[0129] III. Data Preprocessing and Modeling. This section mainly processes the raw CSI data collected by Wi-Fi devices and the video data captured by cameras, converting the CSI into Doppler spectrograms and modeling the skeleton sequences in the video into spatiotemporal skeleton diagrams. Details are as follows:

[0130] 3-1) Use equation (1) to perform conjugate multiplication on the original CSI to eliminate random phase error in the original signal;

[0131] 3-2) A fifth-order Butterworth bandpass filter is used to remove high-frequency and low-frequency noise from the signal. The lowest cutoff frequency is set to 10Hz, and the highest cutoff frequency is set to 80Hz. At the same time, the values ​​are normalized to between -1 and 1 to eliminate the influence of dimensions.

[0132] 3-3) Principal Component Analysis (PCA) is used to eliminate data redundancy in CSI. The first principal component after dimensionality reduction is used for subsequent data analysis, while other components with low contribution are ignored. See [link to PCA documentation] for the signal waveforms before and after the above processing. Figure 5 .

[0133] 3-4) Based on the principle of equation (2), the waveform is converted into a Doppler spectrum by performing time-frequency analysis on the CSI using short-time Fourier transform. See the Doppler spectrum diagrams for the six actions: pushing / pulling, stroking, clasping hands, drawing circles, walking, and squatting. Figure 6 .

[0134] 3-5) The MediaPose human pose estimation algorithm under the MediaPipe project was used to extract human key points. A total of 33 human key points were extracted per frame. The connection relationships between the key points are shown in [reference needed]. Figure 7 (a).

[0135] 3-6) According to equation (3), using human keypoints as nodes and human skeletons as edges, a spatiotemporal skeleton graph is constructed from the skeleton sequence with N keypoints within T time frames. The structure of the spatiotemporal skeleton graph is shown below. Figure 7 (b)

[0136] IV. Divide the dataset from Step 1 into two parts: 80% for the training set and 20% for the test set. Both sets contain samples from different environments, users, and actions. Train the model using data from two modalities: Wi-Fi signals and skeleton sequences. Refer to [link to model documentation] for details on the model used during training. Figure 8 The specific training process is as follows:

[0137] 4-1) A 34-layer residual network (ResNet-34) is used as the feature extraction network for Wi-Fi modes. See [link / reference]. Figure 9 The ResNet-34 network consists of 5 stages. Stage 0 uses a 7×7 convolution for image transformation, which can be considered as the initial feature extraction of the input image. Stages 1 through 4 consist of 3, 4, 6, and 3 BasicBlocks, respectively.

[0138] 4-2) The Attention-Spatiotemporal Graph Convolutional Network (AST-GCN) is used as the feature extraction network for the skeleton modality to extract spatial motion features and temporal features contained in the spatiotemporal skeleton graph. The network structure of AST-GCN is shown in [link to AST-GCN]. Figure 10The deep features are extracted by five attention graph convolutional blocks (AGCN Blocks). Each AGCN Block has the same structure, including an adjacency matrix attention module (ASE), a graph convolutional layer (GCN), and a temporal attention module (TSE). The core modules are the adjacency matrix attention module and the temporal attention module. According to equations (5) and (6), the adjacency matrix attention module parameterizes the topological structure of the graph and captures the potential connection relationships between key points. According to equations (7) to (9), the temporal attention module is used to capture the importance of different time frames and filter out the most important time frames in the process of motion state change. Through these two core modules, the mapping relationship between different key points and continuous time frames is learned, the features are filtered, useful features are enhanced, and useless features are suppressed, so that the network can simultaneously aggregate information in the temporal and spatial dimensions.

[0139] 4-3) Based on equations (10) and (11), extract shallow features F at different network depths from the CSI feature extraction network and the skeleton feature extraction network. s_csi F s_ske and deep features F d_csi F d_ske Multi-scale feature fusion is then performed. The fused high-level semantic features simultaneously aggregate low-level modality features and high-level complementary semantic features.

[0140] 4-4) Complementary knowledge transfer is achieved through spatiotemporal semantic alignment, including:

[0141] The correlation between the high-level semantic features of the two modalities is measured according to equations (12) and (13):

[0142] Based on equations (14) and (15), the spatiotemporal alignment loss is used to characterize the differences between different modalities. During training, the spatiotemporal alignment loss is minimized to learn shared semantic knowledge.

[0143] The network learning process is constrained by the joint loss function of equations (16) to (18), and backpropagation, gradient update, and knowledge transfer are performed during training. In equation (18), the total number of modes M = 2 and the hyperparameter λ = 0.1.

[0144] V. Testing was conducted using a single-mode Wi-Fi network. Details are as follows:

[0145] 5-1) The ResNet-34 trained in step four is used as the feature extraction network for the Wi-Fi modality during the testing phase:

[0146] 5-2) Using only CSI data packets collected by commercial Wi-Fi devices as test data, the spectrogram generated by CSI is used as input to the pre-trained ResNet-34, and the trained ResNet-34 outputs the classification result, thereby completing the behavior recognition task.

[0147] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of the present invention without departing from the spirit of the present invention, and these modifications are all within the protection scope of the present invention.

Claims

1. A method for recognizing multimodal cooperative behavior based on Wi-Fi signals and skeleton sequences, characterized in that, Includes the following steps: Step 1: Deploy commercial Wi-Fi devices and Kinect cameras in a real-world scenario, simultaneously collecting CSI data packets and video data; Step 2: Use device clock source synchronization and data timestamp synchronization methods to complete the time alignment of CSI data packets and video data in order to accurately obtain the changes in the user's motion state during the activity; Step 3: Data preprocessing, including: For CSI collected by commercial Wi-Fi devices, firstly, the conjugate multiplication method and Butterworth bandpass filter are used to eliminate random phase errors, high and low frequency noise and static components in the CSI, and PCA is used to reduce the dimensionality of the denoised signal. Finally, short-time Fourier transform is used to perform time-frequency analysis on the first principal component to obtain the Doppler spectrum. For video data collected by Kinect cameras, human pose estimation algorithm is used to obtain skeleton sequences from the video data collected by Kinect cameras, and connections between human key points are established based on human physical structure and continuous time frame relationships to construct a spatiotemporal skeleton map. Step 4: Multimodal collaborative training, including: In the model training phase, a multimodal collaborative approach is adopted, with Wi-Fi signals as the primary mode and skeleton sequences as the secondary mode. The Doppler spectrum and spatiotemporal skeleton obtained in Step 3 are used as inputs to the model. Feature extraction networks are designed for the primary mode and secondary mode respectively. A multi-scale feature fusion module is used to fuse the multi-scale features at different stages in the two feature extraction networks to obtain multi-level semantic features. Finally, complementary knowledge transfer is completed through spatiotemporal semantic alignment, enabling the model to learn the complementary knowledge between the primary mode and secondary mode, thereby establishing the auxiliary relationship between Wi-Fi signals and skeleton sequences. Step 5: Single Wi-Fi Modality Testing, including: In the model testing phase, using only Wi-Fi modality data as input, the behavior recognition task is completed using a single-modality Wi-Fi network with embedded complementary knowledge.

2. The multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences according to claim 1, wherein, Step two includes the following steps: 2-1) Unified clock source: UTC time is used as the unified time source to provide the reference time, and a high-precision clock is set on the commercial Wi-Fi device and Kinect camera. The clock time of the commercial Wi-Fi device and Kinect camera is calibrated according to the reference time using the IEEE 1588 clock synchronization protocol to ensure that the clock sources of the two devices are synchronized. 2-2) Timestamp Synchronization: After the clock sources of the two devices are synchronized, the commercial Wi-Fi device and Kinect camera will record timestamp information for each data packet acquired according to the calibrated time, and use the timestamp alignment method to synchronize the data at each moment; for each video frame acquired, the CSI data packet and the video frame are aligned once using the interpolation method.

3. The multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences according to claim 1, wherein, Step four includes the following specific steps: 4-1) Design a feature extraction network for the main modality, using ResNet-34 as the feature extraction network for this main modality. The residual structure is represented as follows: （4）； In equation (4), where, The input representing the residual block. This represents the output of the residual block. This represents the output of each layer of the neural network; the ResNet-34 network consists of 5 stages. Stage 0 uses a 7×7 convolution for image transformation, which is the initial feature extraction of the input image. Stages 1 through 4 consist of 3, 4, 6, and 3 BasicBlocks, respectively. 4-2) Design a feature extraction network for auxiliary modalities. Using the spatiotemporal skeleton graph obtained in step 3 as input, extract high-level semantic features from the skeleton sequence through an attention spatiotemporal graph convolutional network. The core modules of this network are the adjacency matrix attention module and the temporal attention module. Adjacency Matrix Attention Module: Uses the spatiotemporal skeleton graph as input feature ,in, Indicates the number of channels. Indicates the total number of time frames. Represents the total number of key points; through two embedding functions. and The input is mapped to different feature spaces and transformed into... and The feature matrices are then multiplied together and their elements are normalized to arbitrary values ​​between 0 and 1 using a softmax operation to obtain the correlation matrix. , of which elements Represents vertices and vertex The correlation between them is shown below: （5）； In equation (5), the embedding function and Use two respectively Convolutional layer implementation, and functions respectively and The parameters, The activation function is used to transform the correlation matrix. With the original adjacency matrix Add them together to obtain a new adaptive adjacency matrix, and then combine it with the input features. Multiply and weight the results, assigning different weights to different key points, to obtain the output: （6）； In equation (6), These are the parameters of a 1×1 convolutional layer; Time Attention Module: Compression Operation Global average pooling operation is used to pool the size of Input features Compressing the spatial dimension to 1 yields... eigenvectors : （7）； Incentive Operation Using a fully connected layer for the compressed size is The feature map is subjected to a nonlinear transformation to obtain the weight vector for each time frame. : （8）； In equation (8), This represents the Sigmoid activation function. Represents the ReLU function. and These represent the weights of the two fully connected layers respectively; the last part is the weighting operation. The weight vector of each time frame Multiplying the input features by the weighted output features generates the weighted output features: （9）； In equation (9), and Features , Element; 4-3) Multi-scale feature fusion module; extracts shallow features at different network depths from the CSI feature extraction network and the skeleton feature extraction network. , and deep features , The fused features are represented as follows: （10）； （11）； In equations (10) and (11), , , , These represent the weight parameters of the four fully connected layers. , These represent the weight parameters of two 1×1 convolutional layers. Indicates a connection operation; 4-4) Complementary knowledge transfer is achieved through spatiotemporal semantic alignment, including: 4-4-1) The following correlation matrix is ​​used to represent the correlation among all elements of a feature: （12）； （13）； In equations (12) and (13), , Let represent normalized matrices, and d represent the matrix dimension. 4-4-2) Use spatiotemporal alignment loss to characterize the differences between different modalities, and minimize the spatiotemporal alignment loss during training to learn shared semantic knowledge: （14）； In equation (14), This is a regularization parameter used to prevent negative transfer; large differences between different modalities can negatively impact network performance; classification loss measures the accuracy of the network's learned content, using the classification loss of both modalities. , difference To measure differences in network performance; Regularization parameters are defined based on the classification loss difference. : （15）； In equation (15), It is a zero threshold function; when the difference If the value is positive, knowledge transfer is performed; otherwise, knowledge transfer is not performed. 4-4-3) By using the joint loss function to constrain the backpropagation process, spatiotemporal semantic alignment is completed, thereby establishing an auxiliary relationship between the Wi-Fi signal and the skeleton sequence; The joint loss function is obtained by adding the classification loss and the spatiotemporal alignment loss. （18）； in, Represents classification loss. Indicates the total number of modes. It's a hyperparameter.

4. The multimodal cooperative behavior recognition method based on Wi-Fi signals and skeleton sequences according to claim 3, wherein, Step five includes the following specific steps: 5-1) The ResNet-34 trained in step four is used as the feature extraction network for the Wi-Fi modality during the testing phase; 5-2) Using only CSI data packets collected by commercial Wi-Fi devices as test data, the spectrogram generated by CSI is used as input to the pre-trained ResNet-34, and the trained ResNet-34 outputs the classification result to complete the behavior recognition task.

Images