Human body behavior recognition method, device and system based on WiFi and vision multi-mode fusion

Abstract

The invention relates to a human body behavior recognition method, device and system based on WiFi and vision multi-mode fusion. The human body behavior recognition method processes WiFi channel state information and video data through two independent network models. The method comprises the following steps: firstly, carrying out feature extraction and recognition on WiFi channel state information by utilizing a WiFi deep learning model; then, performing feature extraction and recognition on the video data by using a visual deep learning model; and finally, splicing the identification results of the two and carrying out decision fusion. According to the method, multi-modal data fusion in a feature layer is avoided, and the influence of modal information change on a prediction result is reduced, so that the robustness of the human body behavior recognition method under the conditions of data noise, modal failure and the like and the accuracy of human body behavior recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of human behavior recognition, and in particular to a method, device and system for human behavior recognition by integrating WiFi and visual multimodality. Background Art

[0002] With the rapid development of information technology, human behavior recognition has become a research field that has attracted much attention. Its background applications extend from indoor positioning to security monitoring of confidential areas, home medical monitoring, new human-computer interaction interfaces, etc., and have broad application prospects. Human behavior recognition refers to the recognition and identification of the action type and behavior pattern of the observed object, and is an important part of many mobile computing applications.

[0003] At present, the research on human behavior recognition is mainly based on vision and wireless signal methods. Human behavior recognition based on vision is to take photos or videos of the human body through a camera, and combine certain technologies to process and analyze video images to identify the corresponding human behavior. After years of research and development, human behavior recognition based on vision has made significant progress, especially in recent years, deep learning methods that can automatically extract features have been proposed, which greatly improves the accuracy of human behavior recognition. However, due to the limitations of visual information itself, the camera may not be able to capture effective appearance information under poor lighting conditions or occlusion.

[0004] With the rapid development of wireless network technology, WiFi has spread to every corner of people's lives. At present, WiFi can not only be used for indoor and outdoor wireless positioning, but has also been applied to the field of human behavior recognition. Based on its own characteristics, using WiFi signals to recognize human behavior can effectively solve many restrictive problems existing in traditional methods. Compared with the human behavior recognition method based on vision, using WiFi signals to recognize human behavior has a series of potential advantages such as non-line-of-sight, no need to carry equipment, low cost, easy deployment, no restrictions on lighting conditions, and protection of personal privacy, which can make up for the shortcomings of visual information. However, WiFi signals themselves are easily interfered by environmental noise and other human activities, making it difficult to achieve a high action recognition rate or good robustness. Therefore, human behavior recognition technology based on WiFi and vision fusion has attracted the attention of researchers.

[0005] However, at present, most of the research on human behavior recognition based on WiFi and vision fusion is feature fusion. Although the feature fusion method can make good use of the correlation between multiple features of different modalities, and the overall structure is relatively simple. However, feature fusion is very sensitive to changes in modal information and has poor error correction capabilities. If data noise or modal failure occurs, the recognition performance will be seriously affected. Summary of the invention

[0006] Based on this, the purpose of the present invention is to provide a human behavior recognition method based on WiFi and visual multimodal fusion, which improves the accuracy of human behavior recognition in the multimodal data human behavior recognition method when data noise, modal failure, etc. appear in the data source.

[0007] The present invention is implemented by the following technical solutions: In the first aspect, the present invention provides a method for human behavior recognition based on WiFi and visual multimodal fusion, which includes: WiFi feature extraction and recognition step: extracting and recognizing features of collected WiFi channel state information (CSI) to obtain WiFi recognition output results;

[0008] Video feature extraction and recognition step: extract and recognize features of the collected video data to obtain video recognition output results;

[0009] Decision fusion step: splicing the WiFi recognition output result and the video recognition output result, and performing decision fusion on the spliced ​​results to obtain a behavior recognition result.

[0010] The present invention processes WiFi CSI and video data through two independent network models. First, the WiFi CSI is feature extracted and recognized using the WiFi deep learning model; then, the video data is feature extracted and recognized using the visual deep learning model; finally, the recognition results of the two are spliced ​​and decision fusion is performed. This method avoids the fusion of multimodal data at the feature layer, reduces the impact of changes in modal information on the prediction results, and thus improves the robustness of the human behavior recognition method in the face of data noise, modal failure, and the like.

[0011] Furthermore, the WiFi feature extraction and identification step includes:

[0012] S10.A: Perform data preprocessing on the collected WiFi CSI to obtain preprocessed WiFi CSI;

[0013] S20.A: Extract features from the pre-processed WiFi CSI to obtain WiFi features.

[0014] S30.A: Identify the WiFi features and obtain a WiFi identification output result.

[0015] Further, step S10.A includes:

[0016] S10.A.1: Perform Hampel filtering on the collected WiFi CSI to obtain filtered WiFi CSI;

[0017] S10.A.2: Perform wavelet transform on the filtered WiFi CSI to obtain wavelet transformed WiFi CSI;

[0018] S10.A.3: Perform principal component analysis on the WiFi CSIs of the three antennas after wavelet transformation to obtain the preprocessed WiFi CSIs.

[0019] Further, step S20.A includes:

[0020] S20.A.1: Convolve the preprocessed WiFi CSI to obtain a first intermediate feature;

[0021] S20.A.2: Performing a first hole convolution process on the first intermediate features in parallel to obtain a plurality of first intermediate feature information, and concatenating the plurality of first intermediate feature information and performing convolution to obtain a second intermediate feature;

[0022] S20.A.3: Perform a parallel second dilated convolution process on the second intermediate features, and then perform batch normalization and activation function processing to obtain a plurality of second intermediate feature information, and after splicing the plurality of second intermediate feature information, perform convolution, batch normalization and activation function processing in sequence to obtain a third intermediate feature;

[0023] S20.A.4: Perform channel attention processing and spatial attention processing on the third intermediate feature in sequence to obtain the WiFi feature.

[0024] Furthermore, the second intermediate feature Y in step S20.A.2 is obtained by the following formula:

[0025] Y=Concat(C 1,1 (X),C 6,3 (X),C 12,3 (X),C 18,3 (X),P(X))

[0026] Among them, X is the first intermediate feature, C m,n (X) represents a dilated convolution with a dilation rate of m and a convolution kernel size of n×n on X, and P(X) represents the use of the global average pooling method to extract features from X and perform convolution and sampling operations.

[0027] Furthermore, the processing formula for the channel attention processing in step S20.A.4 is as follows:

[0028]

[0029] Among them, M crepresents a 1D channel attention map, σ is the Sigmoid activation function, W0 and W1 represent the weight values ​​of the two-layer multilayer perceptron, and They represent the features after average pooling and maximum pooling on the channel respectively, and r represents the reduction ratio of the intermediate channel.

[0030] Furthermore, the processing formula for spatial attention processing in step S20.A.4 is as follows:

[0031]

[0032] Among them, M s represents the 2D spatial attention map, σ is the Sigmoid activation function, and f 7×7 refers to the convolution kernel of size 7×7, and They represent the features after spatial average pooling and the features after maximum pooling, respectively.

[0033] On the other hand, the present invention also provides a human behavior recognition device based on WiFi and visual multimodal fusion, comprising: a WiFi deep learning model, a visual deep learning model and a decision fusion module;

[0034] WiFi deep learning model: used to extract and identify the features of the collected WiFi CSI and obtain WiFi identification output results;

[0035] Visual deep learning model: used to extract and identify features of collected video data and obtain video recognition output results;

[0036] A decision fusion module is used to splice the WiFi recognition output result and the video recognition output result, and perform decision fusion on the spliced ​​results to obtain a behavior recognition result.

[0037] Furthermore, the WiFi deep learning model includes:

[0038] WiFi CSI preprocessing module: used to preprocess the collected WiFi CSI data to obtain the preprocessed WiFi CSI;

[0039] WiFi feature extraction module: used to extract features from the pre-processed WiFi CSI to obtain WiFi features;

[0040] WiFi feature recognition module: used to recognize the WiFi features and obtain WiFi recognition output results.

[0041] On the other hand, the present invention also provides a human behavior recognition system based on WiFi and visual multimodal fusion, which includes:

[0042] Data collection equipment and any of the above human behavior recognition devices;

[0043] The data acquisition device is used to collect WiFi CSI and video data and transmit them to the human behavior recognition device.

[0044] In summary, the present invention processes WiFi CSI and video data through two independent network models. First, the WiFi CSI is feature extracted and identified using a WiFi deep learning model; then, the video data is feature extracted and identified using a visual deep learning model; finally, the recognition results of the two are spliced ​​and decision fusion is performed. This method avoids the fusion of multimodal data at the feature layer, reduces the impact of changes in modal information on the prediction results, and thus improves the robustness of the human behavior recognition method in the face of data noise, modal failure, and the like. After WiFi CSI is collected, the original CSI data is preprocessed in a variety of processing methods to filter out noise, reduce the dimension of the data, and extract the main features, while minimizing information loss and improving the generalization ability of the model. In the WiFi feature extraction stage, dilated convolution and global average pooling are used to capture rich contextual semantics, improve the performance and accuracy of the model, and improve the quality of the output features. And use the channel attention and spatial attention mechanisms to collaboratively learn the key local detail information in the image, assign higher weights to the key areas in the features, and lower weights to the rest, so as to increase the network's attention to the feature content, thereby improving the network's feature learning and expression capabilities, and ultimately making the features used for decision fusion more accurate, thereby improving the accuracy of human behavior recognition.

[0045] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The overall framework diagram of an exemplary human behavior recognition system based on WiFi and visual multimodal fusion provided by the present invention;

[0047] Figure 2 A structural block diagram of a human behavior recognition device based on WiFi and visual multimodal fusion provided by the present invention;

[0048] Figure 3 for Figure 2 A flow chart of a human behavior recognition method performed by a human behavior recognition device;

[0049] Figure 4An exemplary original CSI signal waveform diagram of a drinking action provided by an embodiment of the present invention;

[0050] Figure 5 for Figure 4 The signal waveform of the CSI signal after Hampel filtering;

[0051] Figure 6 for Figure 5 The signal waveform of the CSI signal after wavelet transformation;

[0052] Figure 7 for Figure 6 The signal waveform of the CSI signal after principal component analysis;

[0053] Figure 8 A framework diagram of an exemplary WiFi feature extraction and recognition module provided by the present invention;

[0054] Fig. 9 It is a framework diagram of the first hollow space pyramid pooling submodule of the present invention;

[0055] Fig.10 It is a framework diagram of the convolutional attention submodule of the present invention;

[0056] Fig.11 Schematic diagram of the channel attention module of the present invention;

[0057] Fig.12 Schematic diagram of the spatial attention module of the present invention;

[0058] Fig.13 It is a framework diagram of the decision fusion module of the present invention;

[0059] Fig.14 It is a schematic diagram of the classroom floor plan used in the verification experiment of the present invention. DETAILED DESCRIPTION

[0060] Multimodal fusion is divided into three levels: data layer fusion, feature layer fusion (feature fusion) and decision layer fusion (decision fusion). Feature fusion is used in the existing human behavior recognition research of WiFi and vision fusion. Feature fusion can effectively combine the features of multiple data sources to achieve the accuracy and effectiveness of feature extraction. Some commonly used methods include: concatenation, weighted average, principal component analysis, convolutional neural network, etc. However, as mentioned above, this type of prediction method is very sensitive to the change of modal information. When there is noise in the features used for feature fusion, its prediction effect is not good, which in turn affects the performance of human behavior recognition. After research, the present invention processes WiFi CSI and video data through two independent network models. First, the WiFi deep learning model is used to extract and identify the features of WiFi CSI; then, the visual deep learning model is used to extract and identify the features of video data; finally, the recognition results of the two are spliced ​​and decision fusion is performed. This method avoids the fusion of multimodal data at the feature layer, reduces the impact of changes in modal information on the prediction results, and thus improves the robustness of the human behavior recognition method in the face of data noise, modal failure, etc.

[0061] See also Figure 1 Based on the above analysis, the present invention proposes a human behavior recognition system based on WiFi and visual multimodal fusion, which includes a data acquisition device and a human behavior recognition device.

[0062] Among them, the data acquisition equipment includes a router, a receiving antenna and a camera; the router serves as the CSI data transmitter, and the CSI signal is received by the receiving antenna after identifying the object, while the camera synchronously collects the video signal.

[0063] See also Figure 2 The human behavior recognition device of the present invention includes a WiFi deep learning model, a visual deep learning model and a decision fusion module; wherein the WiFi deep learning model includes a WiFi CSI preprocessing module and a WiFi feature extraction and recognition module, and the visual deep learning model includes a video data preprocessing module and a video feature extraction and recognition module. The human behavior recognition device is used to perform a human behavior recognition method, which includes a WiFi feature extraction step, a video feature extraction step and a decision fusion step; the WiFi deep learning model performs the WiFi feature extraction step, the visual deep learning model performs the video feature extraction step, and the decision fusion module performs the decision fusion step. Please refer to Figure 3 The specific execution process of each module of the human behavior recognition system is as follows:

[0064] The WiFi CSI preprocessing module 10.A executes step S10.A: preprocessing the collected WiFi CSI data to obtain the preprocessed WiFi CSI. Specifically, step S10.A includes:

[0065] S10.A.1: Perform a Hampel filter on the collected WiFi CSI to obtain a filtered WiFi CSI.

[0066] Due to factors such as environmental noise, electromagnetic interference, or changes in the internal state of the device, the received WiFi CSI will have some outliers, which appear as spikes on the waveform. These outliers will cause great interference in extracting the behavioral features of the CSI signal. The Hampel filter can be used to detect and replace outliers. It can filter out all outliers outside the interval [μ-λ×σ,μ+λ×σ], where μ, λ, and σ represent the mean, standard deviation, and mean absolute deviation, respectively. Therefore, the Hampel filter is first used to preprocess the CSI data. Figure 4 This is the original CSI signal waveform of a drinking action. Figure 5 This is the signal waveform after Hampel filtering. It can be clearly observed that the mutation noise is well filtered out.

[0067] S10.A.2: Perform wavelet transform on the filtered WiFi CSI to obtain wavelet transformed WiFi CSI.

[0068] Although the Hampel filter is used to filter out the outliers in the CSI signal, it does not achieve the best denoising effect. Wavelet transform can show good properties in multi-resolution and time-frequency localization, so this method can be used to further eliminate the noise in the CSI signal. The calculation formula of wavelet transform is as follows:

[0069]

[0070] Among them, a is the scale of the wavelet function and τ is the translation.

[0071] The CSI signal waveform after wavelet transform is as follows Figure 6 As shown, it can be observed that the signal waveform after Hampel filtering is smoother and fully retains the characteristics of the original signal.

[0072] S10.A.3: Perform principal component analysis on the WiFi CSIs of the three antennas after wavelet transformation to obtain the preprocessed WiFi CSIs.

[0073] Principal component analysis (PCA) can transform a set of related variables into a set of linearly independent variables through linear transformation. Its main goal is to reduce the dimension of the data and extract the main features, while minimizing information loss and improving the generalization ability of the model. In addition, since the position of each antenna is slightly different, they will carry different characteristic information, so the inventors performed PCA on the three antennas separately, and finally added and averaged them to obtain most of the information. The results are as follows: Figure 7 As shown, the three dotted lines 1, 2, and 3 represent the PCA results of three different antennas, and the solid line in the middle represents the result of adding the PCA results of the three parts and then averaging them.

[0074] The specific steps are as follows: First, subtract each subcarrier of the CSI from the corresponding mean to achieve data centering. Second, calculate the covariance matrix to describe the relationship between different features. Then, perform eigendecomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors, which are sorted and expressed as {q1,q2,…,q k Finally, the first j eigenvectors are selected according to their contribution rate and multiplied with the CSI matrix to obtain the corresponding principal components.

[0075] The WiFi feature extraction module 20.A executes step S20.A: extracts features from the pre-processed WiFi CSI and outputs WiFi features.

[0076] Specifically, S20.A includes:

[0077] S20.A.1: Convolve the preprocessed WiFi CSI to obtain a first intermediate feature.

[0078] In the present invention, the structure of the WiFi feature extraction and recognition module is as follows: Figure 8 As shown, it includes 3 convolutional layers, the first atrous spatial pyramid pooling (ASPP) submodule, the second atrous spatial pyramid pooling submodule and a convolutional block attention submodule (CBAM).

[0079] S20.A.2: Perform a first hole convolution process on the first intermediate feature in parallel to obtain multiple first intermediate feature information, and convolve the multiple first intermediate feature information after splicing to obtain a second intermediate feature.

[0080] ASPP uses multiple parallel dilated convolution branches, each with a different dilated rate, to capture features of different scales. In addition, ASPP also introduces a global average pooling branch to capture global contextual information. Figure 8 In the present invention, the structure of the first ASPP submodule is Fig. 9The structure is consistent with that shown in the figure. ASPP includes a 1×1 convolution layer, three 3×3 convolution layers with different void rates, and a pooling layer. The feature information after pooling is restored to the expected value through 1×1 convolution and upsampling. Finally, all the results are concatenated to expand the number of channels, and then the number of channels is reduced to the expected value through 1×1 convolution for output. The calculation process is as follows:

[0081] Y=Concat(C 1,1 (X),C 6,3 (X),C 12,3 (X),C 18,3 (X),P(X)

[0082] Among them, X is the first intermediate feature, C m,n (X) represents a dilated convolution with a dilation rate of m and a convolution kernel size of n×n on X, and P(X) represents the use of the global average pooling method to extract features from X and perform convolution and sampling operations.

[0083] S20.A.3: Perform a second hole convolution process on the second intermediate feature in parallel, and then perform batch normalization and activation function processing to obtain multiple second intermediate feature information. After splicing the multiple second intermediate feature information, convolution, batch normalization and activation function processing are performed in sequence to obtain the third intermediate feature.

[0084] Compared with the first ASPP submodule, the second ASPP submodule adds a batch normalization layer and an activation function after each convolutional layer, which helps to improve the stability of the model, increase the expressiveness of the model, and facilitate the subsequent learning of the attention mechanism.

[0085] S20.A.4: Perform channel attention processing and spatial attention processing on the third intermediate feature in turn to obtain WiFi features.

[0086] See also Fig.10 ,Convolutional Block Attention Submodule (CBAM) is a convolutional neural network module based on the attention mechanism, which consists of a channel attention submodule and a spatial attention module. It synergistically learns the key local detail information in the image, assigns higher weights to the key areas in the neural network feature map, and assigns lower weights to the rest, which improves the neural network's attention to the image content, thereby improving the network's feature learning and expression capabilities.

[0087] Among them, the channel attention submodule adaptively adjusts the weights of different channel features, performs global maximum pooling and average pooling on the feature maps on different channels to obtain the maximum pooled channel attention vector and the average pooled channel attention vector; the vector size is C×1×1, and then these two vectors are input into the weight-sharing MLP with only one hidden layer, where the shape of the weight vector of the hidden layer is C / r×1×1, and two processed channel attention vectors are obtained. Finally, the two vectors are element-by-element summary operation and Sigmoid activation function processing, and the elements are multiplied with the original feature map to obtain a new feature map. The channel attention submodule is as follows Fig.11 As shown, the calculation formula is as follows:

[0088]

[0089] Among them, M c represents a 1D channel attention map, σ is the Sigmoid activation function, MLP represents a multi-layer perceptron, W0 and W1 represent the weight values ​​of two layers of multi-layer perceptrons, and They represent the features after average pooling and maximum pooling on the channel respectively, and r represents the reduction ratio of the intermediate channel.

[0090] The spatial attention module determines the importance of different regions by analyzing spatial structure and location information. It performs global maximum pooling and average pooling operations on the input feature map, compresses it at the channel level, and obtains two 2D feature maps. These are concatenated into a feature map with 2 channels according to the channel dimension, and then a single convolution layer is performed to make the processed features consistent with the input in spatial dimension. Fig.12 As shown, the calculation formula is as follows:

[0091]

[0092] Among them, M s represents the 2D spatial attention map, σ is the Sigmoid activation function, and f 7×7 refers to the convolution kernel of size 7×7, and They represent the features after spatial average pooling and the features after maximum pooling, respectively.

[0093] The WiFi feature recognition module 30.A executes step S30.A: recognizes the WiFi feature and obtains a WiFi recognition output result.

[0094] Behavior recognition is performed based on the extracted WiFi features to obtain WiFi recognition output results. While extracting and identifying WiFi features, this system also processes the video signal synchronously. The processing process is as follows:

[0095] The video data preprocessing module 10.B executes step S10.B: performing data preprocessing on the collected video data to obtain preprocessed video data.

[0096] In the present invention, a 5s video with 30 frames per second and a total of 150 frames of data is used as the original visual information, and 1 frame of image is extracted every 8 frames, for a total of 19 frames. In a dark environment, the video becomes 15 frames per second, for a total of 75 frames, and 1 frame of image is extracted every 4 frames, for a total of 19 frames. This can be regarded as extracting images at intervals of the same length in both environments, and unifying the length of the data, which helps to avoid information loss and simplify model design. The original size of this series of RGB frame images is 1280×720, which is cropped to 720×720 by cutting off the excess parts on both sides, and then adjusting the resolution to 180×180 as the final video data.

[0097] The video feature extraction module 20.B executes step S20.B: extracting features from the preprocessed video data to obtain video features.

[0098] The video feature recognition module 30.B executes step S30.B: completes recognition according to the extracted features and outputs the visual recognition output result.

[0099] In the present invention, the video data feature extraction module is mainly composed of 5 3D convolutional layers and 2 non-identical ASPP modules. Compared with the WiFi deep learning model, 2 convolutional layers are added, mainly because the continuous frame images are data with higher dimensional structure, containing rich spatial and temporal information, and the actions, characters and scenes therein have complex hierarchical structures and temporal relationships. More convolutional layers can provide deeper feature extraction and expression capabilities, better extract feature representations of different levels in the image, and gradually extract higher-level semantic features. The structure of the ASPP module is similar to that in the WiFi deep learning model, and it is extended to a 3D structure, and can perform feature extraction in time and space dimensions with a variety of hole rates, while capturing local details and spatiotemporal context information, which helps the model to better understand the temporal relationship and spatial structure in the continuous frame images, improve the classification performance of the model, and finally perform behavior recognition according to the extracted video features to obtain video recognition output results.

[0100] The decision fusion module 40 executes step S40: splicing the WiFi output result and the video output result, and performing decision fusion on the spliced ​​output results to obtain a behavior recognition result.

[0101] The decision fusion module concatenates the outputs of the WiFi deep learning model and the visual deep learning model and inputs them into the random forest method for decision fusion. Assuming that each batch has k synchronized WiFi and visual output results and j action categories, each model will output a k×j matrix; concatenate the outputs of the two models and use a matrix of size k×2j as the input of the random forest, such as Fig.13 As shown in the figure, the concatenation method can make comprehensive use of the output results of the two models, retain their entire contents, and provide richer feature information. On the other hand, by connecting the output results by position, the correlation between features is enhanced, which helps to better capture the relationship between related features in the input data.

[0102] Common decision fusion methods can be divided into three categories: mathematical calculation methods, machine learning methods, and neural network methods. Mathematical calculation methods specifically include addition, averaging, weighting, etc. Although these methods are easy to understand and simple to implement, they are too dependent on the results of the previous model and have weak robustness. After analyzing the spliced ​​data, it is found that the overall data is relatively simple. Neural network methods usually require sufficient samples and complexity to exert their powerful capabilities. It may be difficult to learn meaningful feature representations for simple data, and it is easy to cause network overfitting. Machine learning methods are usually more adaptable to simple data, and the random forest method is more robust to noise and missing data, and can handle a certain degree of incomplete or noisy data. Random forest improves overall performance by combining the results of multiple decision trees. Each decision tree is built based on randomly selected samples and feature subsets, so that they have certain differences. Then the results of the decision trees are integrated by majority voting, thereby increasing the diversity and robustness of the model.

[0103] In summary, the present invention processes WiFi CSI and video data through two independent network models. First, the WiFi CSI is feature extracted and identified using a WiFi deep learning model; then, the video data is feature extracted and identified using a visual deep learning model; finally, the recognition results of the two are spliced ​​and decision fusion is performed. This method avoids the fusion of multimodal data at the feature layer, reduces the impact of changes in modal information on the prediction results, and thus improves the robustness of the human behavior recognition method in the face of data noise, modal failure, and the like. After WiFi CSI is collected, the original CSI data is preprocessed in a variety of processing methods to filter out noise, reduce the dimension of the data, and extract the main features, while minimizing information loss and improving the generalization ability of the model. In the WiFi feature extraction stage, dilated convolution and global average pooling are used to capture rich contextual semantics, improve the performance and accuracy of the model, and improve the quality of the output features. And use the channel attention and spatial attention mechanisms to collaboratively learn the key local detail information in the image, assign higher weights to the key areas in the features, and lower weights to the rest, so as to increase the network's attention to the feature content, thereby improving the network's feature learning and expression capabilities, and ultimately making the features used for decision fusion more accurate, thereby improving the accuracy of human behavior recognition.

[0104] In order to verify the effectiveness and excellence of the present invention, a verification experiment is also provided. The inventors set up an experimental site in a common classroom. The tables and chairs are fixed on the ground. The router and the receiving antenna are placed on both sides of the aisle, and the camera is located diagonally in front. Fig.14 The specific experiment is described as follows: The CSI data collection transmitter uses a TP-Link 886N wireless router, and the receiver uses a ThinkPad X200 laptop with an Intel 5300 wireless network card and Ubuntu 11.04 system, equipped with three omnidirectional antennas connected to the network card. The data collection frequency is set to 100Hz. The video data uses an Aoni A25 high-definition camera and is collected synchronously with the CSI data.

[0105] The data collection subjects are two students of different heights and weights. The actions are drinking water, picking up, clapping, standing up, and walking. A part of the data is collected in a dark environment, and each action lasts about 5 seconds. In order to verify the effectiveness of the method in this paper, the inventor added black rectangular blocks to the video content to simulate the occlusion of various objects in the daily environment.

[0106] In order to verify the robustness of the fusion module under different lighting conditions, the inventors conducted experiments in daily environments and night environments. In daily environments, the WiFi deep learning model achieved an accuracy of 96.86%, and the visual deep learning model achieved an accuracy of 97.43%. After the decision fusion of the random forest method, it can inherit the advantages of WiFi and vision, and the performance is better than each single modality, with an accuracy of up to 99.71%.

[0107] In the dark environment, the inventor turned off all the lights in the classroom. The entire environment was only illuminated by a faint light from the window. The walls and blackboard were vaguely visible, and the rest of the environment was dark and difficult to distinguish. The accuracy of using a visual deep learning model to recognize actions in a dark environment is only 62%, which is difficult to meet the needs of any work. In contrast, WiFi information is insensitive to changes in lighting and is basically unaffected. The recognition accuracy of the WiFi deep learning model is 96.33%, which is basically consistent with the daily environment. After the two are fused by random forest, an accuracy of 99% can be achieved, which proves that the inventor's fusion module has strong robustness and can achieve high recognition accuracy under different lighting conditions. The comparison of the accuracy between each modality is shown in Table 1

[0108]

[0109] Table 1

[0110] In order to verify the performance of the random forest fusion method, the inventors selected four other common methods for comparative experiments. The results are shown in Table 2:

[0111]

[0112]

[0113] Table 2

[0114] The addition method is to simply add the outputs of the two deep learning models to get the final result. The effect is very good in daily environments, with an accuracy of 99.71%, which is the same as the random forest fusion method; but in the dark environment, due to the low accuracy of the visual model, it has some impact on the result of the addition, and the accuracy is only 96.67%. The deep neural network (DNN) method does not work well in daily environments, and the accuracy is even lower than that of a single visual model; but in the dark environment, it has a certain improvement compared to the two single models, reaching 98%. The Transformer method has a certain improvement in both daily and dark environments. The support vector machine (SVM) method works very well in daily environments, with an accuracy of 99.14%, which is close to the random forest fusion method.

[0115] The above four methods can reach or approach the accuracy of the random forest fusion method in daily environments, but are lower than the 99% accuracy of the random forest fusion method in night environments, which can prove that the fusion module of the present invention has better performance and is superior to the other four common methods.

[0116] In order to verify the robustness of the fusion model under various occlusion conditions, the inventors simulated the occlusion of various objects in daily environments by adding black rectangular blocks. For the lower occlusion, it can be seen from Table 3 that the addition and SVM methods have a significant decrease compared to the normal situation, the DNN, Transformer and random forest methods have only small fluctuations, and the random forest fusion method can still get the best result of 99.14%. For the middle occlusion, it can be seen from Table 3 that the SVM method has a significant decrease compared to the normal situation, the DNN, Transformer and random forest methods have only small fluctuations, and the addition method has a partial increase, and the random forest fusion method also got the best result of 98.86%.

[0117] Through the comparison of the above two types of occlusion, it can be proved that the inventor's fusion model has strong robustness and can achieve high recognition accuracy under various occlusion conditions.

[0118]

[0119] Table 3

[0120] In summary, from multiple aspects of comparison, the human behavior recognition method of the present invention is superior to the existing recognition methods, and therefore has high research potential and value.

[0121] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for human behavior recognition based on WiFi and visual multimodal fusion as described in any one of the above embodiments is implemented.

[0122] The present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0123] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A human behavior recognition method based on WiFi and visual multimodal fusion, characterized in that: include: WiFi feature extraction and identification step: extract and identify the collected WiFi channel state information (CSI) to obtain WiFi identification output results; Video feature extraction and recognition step: extract and recognize features of the collected video data to obtain video recognition output results; Decision fusion steps; The WiFi recognition output result and the video recognition output result are spliced, and the spliced ​​results are subjected to decision fusion to obtain a behavior recognition result.

2. The human behavior recognition method based on WiFi and visual multimodal fusion according to claim 1 is characterized in that: The WiFi feature extraction and identification step comprises: S10.A: Perform data preprocessing on the collected WiFi CSI to obtain preprocessed WiFi CSI; S20.A: Extract features from the pre-processed WiFi CSI to obtain WiFi features. S30.A: Identify the WiFi features and obtain WiFi identification output results.

3. According to claim 2, a method for human behavior recognition based on WiFi and visual multimodal fusion, characterized in that: Step S10.A includes: S10.A.1: Perform Hampel filtering on the collected WiFi CSI to obtain filtered WiFi CSI; S10.A.2: Perform wavelet transform on the filtered WiFi CSI to obtain wavelet transformed WiFi CSI; S10.A.3: Perform principal component analysis on the WiFi CSIs of the three antennas after wavelet transformation to obtain the preprocessed WiFi CSIs.

4. According to claim 3, a method for human behavior recognition based on WiFi and visual multimodal fusion is characterized in that: Step S20.A includes: S20.A.1: Convolve the preprocessed WiFi CSI to obtain a first intermediate feature; S20.A.2: Performing a first hole convolution process on the first intermediate features in parallel to obtain a plurality of first intermediate feature information, and concatenating the plurality of first intermediate feature information and performing convolution to obtain a second intermediate feature; S20.A.3: Perform a parallel second dilated convolution process on the second intermediate features, and then perform batch normalization and activation function processing to obtain a plurality of second intermediate feature information, and after splicing the plurality of second intermediate feature information, perform convolution, batch normalization and activation function processing in sequence to obtain a third intermediate feature; S20.A.4: Perform channel attention processing and spatial attention processing on the third intermediate feature in sequence to obtain the WiFi feature.

5. According to claim 4, a method for human behavior recognition based on WiFi and visual multimodal fusion is characterized in that: The second intermediate feature Y in step S20.A.2 is obtained by the following formula: Y=Concat(C 1,1 (X),C 6,3 (X),C 12,3 (X),C 18,3 (X),P(X)) Among them, X is the first intermediate feature, C m,n (X) represents a dilated convolution with a dilation rate of m and a convolution kernel size of n×n on X, and P(X) represents the use of the global average pooling method to extract features from X and perform convolution and sampling operations.

6. A method for human behavior recognition based on WiFi and visual multimodal fusion according to claim 5, characterized in that: The processing formula for channel attention processing in step S20.A.4 is as follows: Among them, M c represents a 1D channel attention map, σ is the Sigmoid activation function, W0 and W1 represent the weight values ​​of the two-layer multilayer perceptron, and They represent the features after average pooling and maximum pooling on the channel respectively, and r represents the reduction ratio of the intermediate channel.

7. A method for human behavior recognition based on WiFi and visual multimodal fusion according to claim 6, characterized in that: The processing formula for spatial attention processing in step S20.A.4 is as follows: Among them, M s represents the 2D spatial attention map, σ is the Sigmoid activation function, and f 7×7 refers to the convolution kernel of size 7×7, and They represent the features after spatial average pooling and the features after maximum pooling, respectively.

8. A human behavior recognition device based on WiFi and visual multimodal fusion, characterized in that: include: WiFi deep learning model, visual deep learning model and decision fusion module; WiFi deep learning model: used to extract and identify the features of the collected WiFi CSI and obtain WiFi identification output results; Visual deep learning model: used to extract and identify features of collected video data and obtain video recognition output results; Decision fusion module; It is used to splice the WiFi recognition output result and the video recognition output result, and perform decision fusion on the spliced ​​results to obtain the behavior recognition result.

9. The human behavior recognition device based on WiFi and visual multimodal fusion according to claim 8, characterized in that: The WiFi deep learning model includes: WiFi CSI preprocessing module: used to preprocess the collected WiFi CSI data to obtain the preprocessed WiFi CSI; WiFi feature extraction module: used to extract features from the pre-processed WiFi CSI to obtain WiFi features; WiFi feature recognition module: used to recognize the WiFi features and obtain WiFi recognition output results.

10. A human behavior recognition system based on WiFi and visual multimodal fusion, characterized in that: include: A data acquisition device and a human behavior recognition device as described in any one of claims 8 to 9; The data acquisition device is used to collect WiFi CSI and video data and transmit them to the human behavior recognition device.

Images