A new human behavior recognition method based on channel feature enhancement and deep learning

Abstract

A new method for human behavior recognition based on channel feature enhancement and deep learning specifically includes the following steps: building a human behavior detection system based on Wi-Fi signals; collecting CSI data and calculating subcarrier correlation coefficients; completing subcarrier dynamic selection according to the subcarrier spatial frequency constraints based on the calculated correlation coefficients; extracting the main components of the selected subcarriers and extracting the CSI behavior feature matrix; introducing a model structure combining Transformer and convolutional network for training, and testing after training. The present invention reduces training time and greatly improves the accuracy of human behavior recognition by reading CSI data, preprocessing CSI data based on dynamic subcarrier selection, and extracting the CSI behavior feature matrix.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile wireless communications, and designs a novel human behavior recognition method based on channel feature enhancement and deep learning. Background Art

[0002] With the surge in wireless networks and the continuous advancement of mobile devices, the Internet of Things technology has developed rapidly, opening the door for non-contact environmental sensing to achieve human behavior recognition, and has been widely used in logistics warehousing, smart space, smart home and other fields. As an important part of the Internet of Things technology, the human behavior recognition technology based on Wi-Fi does not require any equipment to be attached to the monitoring target, which reduces the active cooperation requirements of the monitoring process.

[0003] As an important part of contactless identification technology, Wi-Fi technology is widely used in various fields due to its advantages such as wide coverage, fast communication speed and relatively mature technology. At the same time, with the development of deep learning technology, deep learning has also been rapidly integrated into related fields with its powerful computing power. In the process of combining deep learning with other fields, the rapid development of Transformer in recent years has also further promoted the development of interdisciplinary subjects, especially the field of human behavior recognition. The traditional Wi-Fi human behavior recognition system uses directly collected CSI data, and then performs simple preprocessing on the original CSI to perform behavior recognition, and the classification method adopted is the traditional machine learning method. The above method has the problems of large data dimension, relatively rough extracted features, low classification accuracy, and inability to achieve fine-grained distinction of various actions. In view of the above problems, the present invention proposes a new human behavior recognition method based on channel feature enhancement and deep learning. Summary of the invention

[0004] The purpose of the present invention is to provide a new human behavior recognition method based on channel feature enhancement and deep learning, which can solve the problems of low accuracy, high implementation complexity and long time consumption in human behavior recognition in the prior art by enhancing the features of CSI data and combining the network model of Transformer and convolutional network. The technical solution adopted by the present invention is to collect CSI data using the Intel 5300 network card based on Wi-Fi, combine it with deep learning, and use the network model combining Transformer and CNN for training and classification, which specifically includes the following steps:

[0005] Step 1: Build a human behavior detection system based on Wi-Fi signals. The system consists of two mobile devices equipped with Intel 5300 network cards. The framework of the human behavior recognition method mainly includes four parts: CSI data collection, CSI subcarrier dynamic selection, extraction of CSI behavior feature matrix, and human behavior recognition;

[0006] Step 2: Based on the human behavior detection system built in step 1, CSI data collection is performed. The test subject is asked to perform running, jumping, walking slowly, falling, bowing and other actions in the system. The CSI data H generated by the actions are collected through the CSI Tool matched with the Intel 5300 network card. H is a set of n data packets, that is, H = [H1, H2, ..., H i , ..., H n ], where the i-th packet Indicates H i The CSI vector of the kth subcarrier in , where N represents the number of subcarriers, usually 30;

[0007] Step 3: Considering that the original CSI data contains a lot of redundant information, the subcarrier selection algorithm is used to reduce the redundant information in the CSI data and reduce the data dimension, thereby improving the robustness and operation speed of the system. According to the CSI data collected in step 2, the correlation coefficient r between the subcarriers is calculated. a，b ,in a i With b i represents the i-th element in sequences a and b, l represents the length of sequences a and b, and Represent the average values ​​of sequences a and b respectively. According to the above formula, the correlation coefficient between each subcarrier and other subcarriers is calculated to obtain a 30×30 subcarrier correlation coefficient matrix R f , weakly correlated subcarriers will contain more information, alleviating the problem of decreased accuracy caused by information loss, so it is necessary to determine the subcarrier combination with the weakest correlation in the sample;

[0008] Step 4: Obtain the subcarrier correlation coefficient matrix R in step 3 f , traverse R f After all elements are found, find the position of the smallest correlation coefficient in each column, store the row and column index of its position in the new matrix, and get the 30×2 minimum correlation coefficient matrix C f , filter all subcarriers and count C f The number of occurrences of subcarrier numbers from 1 to 30 in the above formula is calculated. According to the principle of "the minority obeys the majority", the subcarrier numbers with the largest number of occurrences are added to the weakly correlated subcarrier combination S in sequence until the number of subcarriers in S reaches the preset value a, which is usually 3. At this time, S only retains C f The number of subcarriers that appear at most is 3;

[0009] Step 5: The longer the interval between the two subcarrier indexes, the greater the difference between the subcarrier frequencies ω, and ultimately the more significant the difference between the phase changes of the two subcarriers caused by the same motion. In order to balance the influence of the subcarrier phase change difference and the subcarrier correlation, the frequency space constraint of the subcarrier is introduced. The subcarrier finally selected in S should be selected from three subcarrier intervals, that is, one is selected from each of the three intervals of subcarrier indexes [1, 10], [11, 20] and [21, 30]. If the subcarrier combination output by S in step 4 does not meet the condition, the second most frequently occurring subcarrier combination is selected. Similarly, if the second most frequently occurring subcarrier combination does not meet the condition, the third most frequently occurring subcarrier combination is selected, and so on. The above operations are performed on different CSI samples to ensure that S is the most representative weakly correlated subcarrier combination of the sample. According to the obtained S, only the three subcarriers selected in S are retained, and the data of the remaining subcarriers are deleted, thereby realizing dynamic subcarrier selection;

[0010] Step 6: In order to extract feature information that is more dependent on activity and environment, it is necessary to reduce data that is not related to activity and retain information related to activity. Therefore, the CSI data is divided into a static component H that is not related to human activity. s and the dynamic component H caused by human motion d , that is, H=H s +H d , H d It is related to human activities, but human behavior has limited impact on the environment. Directly using H for human behavior recognition will significantly reduce the performance. In order to capture various distortions, phase shifts, amplitude attenuation, etc. caused by human behavior in CSI data, the static component H is converted from H s Remove to significantly improve the quality of extracted feature components, and use the exponential weighted average method to perform recursive calculations to estimate H s ,Right now in H represents the mth data packet s The estimated value of , β is the forgetting factor, usually taken as 0.005, is equal to 0, the estimated Subtract, get H represents the mth data packet d The estimated value of , thereby obtaining the dynamic component in the CSI data;

[0011] Step 7: To further filter out The noise component and the component unrelated to the activity in the image are used to enhance the feature, based on the dynamic component obtained in step 6. First, find The covariance matrix C of Then, find the eigenvalue D and the corresponding eigenvector V of the covariance matrix C, arrange the eigenvector V into a matrix from top to bottom according to the corresponding eigenvalue size, take the first k rows to form the matrix P, and Subtract the mean value M from all elements in and multiply it by the matrix P to get Principal component matrix Finally and Multiply them together to get the CSI feature matrix C p ,Right now Different principal components are related and can provide additional information for human behavior recognition. p With its transposed matrix Multiply to get the CSI behavior feature matrix D C ,Right now

[0012] Step 8: Introduce the model structure combining Transformer and convolutional network in deep learning as the training network. First, connect a local attention module to each of the three convolutional networks with different convolution kernel sizes to extract local features of different dimensions. Then connect the three convolutional networks connected to the local attention modules in parallel. Then, concatenate the three feature vectors output after the convolutional networks are connected in parallel to avoid gradient explosion in subsequent training. The concatenated feature vectors are input into the four-layer series-connected Transformer. The global attention mechanism of Transformer can extract the global features of the image well. After the output results of the four-layer Transformer are pulled into a one-dimensional vector, they are output after passing through two fully connected layers to complete the forward propagation.

[0013] Step 9: The training sample is a two-dimensional heat map generated by the CSI behavior feature matrix extracted in step 7. Each human behavior is regarded as a type of training sample. The number of training samples in each type should be no less than 100, and the number of iterations should be set to no less than 200 times. At the same time, the ratio of training set, validation set, and test set is recommended to be 8:1:1. During the training process, the network will be trained and validated at the same time, and the classification accuracy will be updated synchronously;

[0014] Step 10: The dynamic selection of subcarriers greatly reduces the data dimension and shortens the training time to a great extent. After completing several generations of training, according to the requirements of step 9, collect a suitable number of test samples, input the test samples into the network and match them with the training results, and calculate the classification accuracy of the network after training according to the accuracy formula: α = N c / N all ×100%, where α represents the accuracy, N c Indicates the number of samples classified correctly, N allRepresents the number of test samples. The extracted CSI behavior feature matrix has good robustness and distinguishability, which makes the trained classification network have a high recognition accuracy. Under the premise of ensuring a low training time, a high human behavior recognition accuracy is achieved.

[0015] The present invention realizes a new human behavior recognition method based on channel feature enhancement and deep learning. The method extracts the behavior feature matrix from the collected CSI data after completing the dynamic subcarrier selection, and uses the behavior feature matrix after completing the channel feature enhancement as the training data of the network. The model structure combining Transformer and convolutional network is introduced as the network model, which improves the accuracy of human behavior recognition while ensuring the network training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the present invention;

[0017] Figure 2 It is a schematic diagram of the deployment of the human behavior detection system based on Wi-Fi of the present invention;

[0018] Figure 3 It is a flowchart of the subcarrier dynamic selection algorithm of the present invention;

[0019] Figure 4 is a network structure diagram of the present invention;

[0020] Figure 5 It is a training curve diagram of the present invention;

[0021] Figure 6 This is a curve chart comparing the test results of the present invention with other networks. DETAILED DESCRIPTION

[0022] The purpose of the present invention is to propose a new human behavior recognition method based on channel feature enhancement and deep learning. The method takes into account the characteristics that human movements will cause phase shift and distortion to CSI data. By extracting subcarrier dynamic selection and extracting the CSI behavior feature matrix, the influence caused by human behavior is captured, and the accurate extraction of human behavior features in CSI data is achieved, thereby realizing CSI channel feature enhancement. The method also combines the Transformer and convolutional networks in deep learning to complete training and classification.

[0023] The following is combined with Figure 1 , Attachment Figure 2 , Attachment Figure 3 , Attachment Figure 4 , Attachment Figure 5 , Attachment Figure 6 The embodiments of the present invention are described in further detail.

[0024] The design framework of the present invention is shown in the attached Figure 1 As shown, we first build a human behavior detection system based on Wi-Fi signals. The system consists of two mobile devices equipped with Intel 5300 network cards. The framework of the human behavior recognition method mainly includes four parts: CSI data acquisition, CSI subcarrier dynamic selection, extraction of CSI behavior feature matrix, and human behavior recognition.

[0025] According to the attached Figure 2 After the system is built, CSI data collection is performed. The tester is asked to perform running, jumping, walking slowly, falling, bowing and other actions in the system. The CSI data H generated by the actions are collected through the CSI Tool matched with the Intel 5300 network card. H is a set of n data packets, that is, H = [H1, H2, ..., H i , ..., H n ], where the i-th packet Indicates H i The CSI vector of the kth subcarrier in , where N represents the number of subcarriers, usually 30.

[0026] Then, considering that the original CSI data contains more redundant information, Figure 3 The subcarrier selection algorithm reduces the redundant information in the CSI data and reduces the data dimension, thereby improving the robustness and operation speed of the system. The collected CSI data is used to calculate the correlation coefficient r between the subcarriers. a，b ,in a i With b i represents the i-th element in sequences a and b, l represents the length of sequences a and b, and Represent the average values ​​of sequences a and b respectively. According to the above formula, the correlation coefficient between each subcarrier and other subcarriers is calculated to obtain a 30×30 subcarrier correlation coefficient matrix R f , weakly correlated subcarriers will contain more information, alleviating the problem of decreased accuracy caused by information loss. Therefore, it is necessary to determine the subcarrier combination with the weakest correlation in the sample. In the obtained subcarrier correlation coefficient matrix R f , traverse R f After all elements are found, find the position of the smallest correlation coefficient in each column, store the row and column index of its position in the new matrix, and get the 30×2 minimum correlation coefficient matrix C f , filter all subcarriers and count C f The number of occurrences of subcarrier numbers from 1 to 30 in the above formula is calculated. According to the principle of "the minority obeys the majority", the subcarrier numbers with the largest number of occurrences are added to the weakly correlated subcarrier combination S in sequence until the number of subcarriers in S reaches the preset value a, which is usually 3. At this time, S only retains C fThe maximum number of subcarriers that appear in the .

[0027] Since the longer the interval between two subcarrier indexes, the greater the difference between the subcarrier frequencies ω, and ultimately the more significant the difference between the phase changes of the two subcarriers caused by the same motion, in order to balance the influence of the subcarrier phase change difference and the subcarrier correlation, the frequency space constraint of the subcarrier is introduced. The subcarrier finally selected in S should be selected from three subcarrier intervals, that is, one is selected from each of the three intervals of subcarrier index [1, 10], [11, 20] and [21, 30]. If the subcarrier combination output by S does not meet the condition, the second most frequently occurring subcarrier combination is selected. Similarly, if the second most frequently occurring subcarrier combination does not meet the condition, the third most frequently occurring subcarrier combination is selected, and so on. The above operations are performed on different CSI samples to ensure that S is the most representative weakly correlated subcarrier combination of the sample. According to the obtained S, only the three subcarriers selected in S are retained, and the data of the remaining subcarriers are deleted, thereby realizing dynamic subcarrier selection.

[0028] In order to extract feature information that is more dependent on activity and environment, it is necessary to reduce data irrelevant to activity and retain information related to activity. Therefore, the CSI data is divided into a static component H that is irrelevant to human activity. s and the dynamic component H caused by human motion d , that is, H=H s +H d , H d It is related to human activities, but human behavior has limited impact on the environment. Directly using H for human behavior recognition will significantly reduce the performance. In order to capture various distortions, phase shifts, amplitude attenuation, etc. caused by human behavior in CSI data, the static component H is converted from H s Remove to significantly improve the quality of extracted feature components, and use the exponential weighted average method to perform recursive calculations to estimate H s ,Right now in H represents the mth data packet s The estimated value of , β is the forgetting factor, usually taken as 0.005, is equal to 0, the estimated Subtract, get H represents the mth data packet d The estimated value of , thus obtaining the dynamic component in the CSI data, in order to further filter out The noise component and the component unrelated to the activity in the channel are enhanced. First, we find The covariance matrix C of Then, find the eigenvalue D and the corresponding eigenvector V of the covariance matrix C, arrange the eigenvector V into a matrix from top to bottom according to the corresponding eigenvalue size, take the first k rows to form the matrix P, and Subtract the mean value M from all elements in and multiply it by the matrix P to get Principal component matrix Finally and Multiply them together to get the CSI feature matrix C p ,Right now Different principal components are related and can provide additional information for human behavior recognition. p With its transposed matrix Multiply to get the CSI behavior feature matrix D C ,Right now

[0029] According to the attached Figure 4 As shown in the figure, the model structure combining Transformer and convolutional network in deep learning is introduced as the training network. First, a local attention module is connected to each of the three convolutional networks with different convolutional kernel sizes to extract local features of different dimensions. Then, the three convolutional networks connected to the local attention module are connected in parallel. Then, the three feature vectors output after the convolutional networks are connected in parallel are spliced ​​to avoid gradient explosion in subsequent training. The spliced ​​feature vector is input into the four-layer series Transformer. The global attention mechanism of Transformer can extract the global features of the image well. After the output result of the four-layer Transformer is pulled into a one-dimensional vector, it is output after passing through two fully connected layers to complete the forward propagation. Figure 4 is the training curve during network training.

[0030] The training samples are two-dimensional heat maps generated by the extracted CSI behavior feature matrix. Each type of human behavior is regarded as a type of training sample. The number of training samples in each type should be no less than 100, and the number of iterations should be set to no less than 200 times. At the same time, the ratio of training set, validation set, and test set is recommended to be 8:1:1. During the training process, the network will be trained and verified at the same time, and the classification accuracy will be updated synchronously.

[0031] The dynamic selection of subcarriers greatly reduces the data dimension and shortens the training time to a great extent. After completing several generations of training, collect a suitable number of test samples, input the test samples into the network and match them with the training results, and calculate the classification accuracy of the network after training according to the accuracy formula: α = N c / N all ×100%, where α represents the accuracy, N c Indicates the number of samples classified correctly, N allRepresents the number of test samples. The extracted CSI behavior feature matrix has good robustness and distinguishability, which makes the trained classification network have a high recognition accuracy. Under the premise of ensuring a low training time, a high human behavior recognition accuracy is achieved.

[0032] The following is the specific implementation: Figure 2 As shown in the figure, the scene size of data collection is 4.5m×4m, and the human behavior collection system consists of two mobile devices equipped with Intel 5300 network cards. During the sample collection process, a total of 5 actions were collected, namely: running, jumping, walking slowly, sitting down, and bowing, and 150 samples were collected for each action as training data. After the data is collected, the subcarriers are screened according to the subcarrier dynamic selection algorithm, and then the main components are extracted from the selected subcarriers to obtain 5 kinds of behavior feature matrices, and a total of 5×150 samples are used as training data. The collected training samples are input into the training network to start training, and the number of iterations is set to 300 times. The training time is about 70 minutes.

[0033] Use the built system to collect test data, and collect 20 samples for each action. After collecting the test data, use the trained network to test it, as shown in the attached figure. Figure 6 As shown, the results show that the classification accuracy is above 98%, and the original network classification accuracy has converged when it is maintained at 95%. The results show that the present invention has a significant improvement over previous methods.

Claims

1. A new human behavior recognition method based on channel feature enhancement and deep learning, characterized by , the specific steps are as follows: Step 1: Build a human behavior detection system based on Wi-Fi signals. The system consists of two mobile devices equipped with Intel 5300 network cards. The framework of the human behavior recognition method includes four parts: CSI data collection, dynamic selection of CSI subcarriers, extraction of CSI behavior feature matrix to achieve channel feature enhancement, and combination with Transformer and convolutional network in deep learning to achieve human behavior recognition; Step 2: Based on the human behavior detection system built in step 1, CSI data collection is performed. The test subject is asked to run, jump, walk slowly, fall, and bow in the system. The CSI data H generated by the actions are collected through the CSI Tool matched with the Intel 5300 network card. H is a set of n data packets, that is, H = [H1, H2, ..., H i , ..., H n ], where the i-th packet Indicates H i The CSI vector of the kth subcarrier in , where N represents the number of subcarriers and is 30; Step 3: Considering that the original CSI data contains a lot of redundant information, the subcarrier selection algorithm is used to reduce the redundant information in the CSI data and reduce the data dimension, thereby improving the robustness and operation speed of the system. According to the CSI data collected in step 2, the correlation coefficient r between the subcarriers is calculated. a,b ,in a i With b i represents the i-th element in sequences a and b, l represents the length of sequences a and b, and Represent the average values ​​of sequences a and b respectively. According to the above formula, the correlation coefficient between each subcarrier and other subcarriers is calculated to obtain a 30×30 subcarrier correlation coefficient matrix R f , weakly correlated subcarriers will contain more information, alleviating the problem of decreased accuracy caused by information loss, so it is necessary to determine the subcarrier combination with the weakest correlation in the sample; Step 4: The subcarrier correlation coefficient matrix R obtained in step 3 f , traverse R f After all elements are found, find the position of the smallest correlation coefficient in each column, store the row and column index of its position in the new matrix, and get the 30×2 minimum correlation coefficient matrix C f , filter all subcarriers and count C f According to the principle of "the minority obeys the majority", the subcarrier numbers with the largest number of occurrences are added to the weakly correlated subcarrier combination S in sequence until the number of subcarriers in S reaches the preset value a, where a is 3. At this time, S only retains C f The number of subcarriers that appear at most is 3; Step 5: The longer the interval between the two subcarrier indexes, the greater the difference between the subcarrier frequencies ω, and ultimately the more significant the difference between the phase changes of the two subcarriers caused by the same motion. In order to balance the influence of the subcarrier phase change difference and the subcarrier correlation, the frequency space constraint of the subcarrier is introduced. The subcarrier finally selected in S should be selected from three subcarrier intervals, that is, one from each of the three intervals of subcarrier indexes [1, 10], [11, 20] and [21, 30]. If the subcarrier combination output by S in step 4 does not meet this condition, the second most frequently occurring subcarrier combination is selected. Similarly, if the second most frequently occurring subcarrier combination does not meet this condition, the third most frequently occurring subcarrier combination is selected, and so on. The above operations are performed on different CSI samples to ensure that S is the most representative weakly correlated subcarrier combination of the sample. According to the obtained S, only the three subcarriers selected in S are retained, and the data of the remaining subcarriers are deleted, thereby realizing dynamic subcarrier selection; Step 6: In order to extract feature information that is more dependent on activity and environment, it is necessary to reduce data that is not related to activity and retain information related to activity. Therefore, the CSI data is divided into a static component H that is not related to human activity. s and the dynamic component H caused by human motion d , that is, H=H s +H d , H d It is related to human activities, but human behavior has limited impact on the environment. Directly using H for human behavior recognition will significantly reduce the performance. In order to capture the various distortions, phase shifts, and amplitude attenuation caused by human behavior in CSI data, the static component H is extracted from H. s Remove to significantly improve the quality of extracted feature components, and use the exponential weighted average method to perform recursive calculations to estimate H s ,Right now in H represents the mth data packet s The estimated value of , β is the forgetting factor, which is 0.005, is equal to 0, the estimated Subtract, get H represents the mth data packet d The estimated value of , thereby obtaining the dynamic component in the CSI data; Step 7: To further filter out The noise component and the component unrelated to the activity in the channel are enhanced, based on the dynamic component obtained in step 6. First, find The covariance matrix C of Then, find the eigenvalue D and the corresponding eigenvector V of the covariance matrix C, arrange the eigenvector V into a matrix from top to bottom according to the corresponding eigenvalue size, take the first k rows to form the matrix P, and Subtract the mean value M from all elements in and multiply it by the matrix P to get Principal component matrix Finally and Multiply them together to get the CSI feature matrix C p ,Right now Different principal components are related and can provide additional information for human behavior recognition. p With its transposed matrix Multiply to get the CSI behavior feature matrix D C ,Right now Step 8: Introduce the model structure combining Transformer and convolutional network in deep learning as the training network. First, connect a local attention module to each of the three convolutional networks with different convolution kernel sizes to extract local features of different dimensions. Then connect the three convolutional networks connected to the local attention modules in parallel. Then, concatenate the three feature vectors output after the convolutional networks are connected in parallel to avoid gradient explosion in subsequent training. The concatenated feature vectors are input into the four-layer series-connected Transformer. The global attention mechanism of Transformer can extract the global features of the image well. After the output results of the four-layer Transformer are pulled into a one-dimensional vector, they are output after passing through two fully connected layers to complete the forward propagation. Step 9: The training sample is a two-dimensional heat map generated by the CSI behavior feature matrix extracted in step 7. Each human behavior is regarded as a type of training sample. The number of training samples in each type should be no less than 100, and the number of iterations should be set to no less than 200 times. At the same time, the ratio of training set, validation set, and test set is 8:1:

1. During the training process, the network will be trained and validated at the same time, and the classification accuracy will be updated synchronously; Step 10: The dynamic selection of subcarriers greatly reduces the data dimension and shortens the training time to a great extent. After completing several generations of training, according to the requirements of step 9, collect a suitable number of test samples, input the test samples into the network and match them with the training results, and calculate the classification accuracy of the network after training according to the accuracy formula: α = N c / N all ×100%, where α represents the accuracy, N c Indicates the number of samples classified correctly, N all It represents the number of test samples. The extracted CSI behavior feature matrix has good robustness and distinguishability, which makes the classification network recognition accuracy after training higher. Under the premise of ensuring a lower training time, a higher human behavior recognition accuracy can be achieved.

2. The method according to claim 1, characterized in that Step 8 The reason for choosing the combination of Transformer and convolutional network CNN for training the network model structure is that: different from the traditional convolutional network CNN and recurrent neural network LSTM, Transformer has advantages that are difficult to achieve with its special global attention mechanism. A single CNN can only extract local features of the image, while Transformer can obtain the global features of the image very well. Transformer can see wider than CNN and has a shorter training time than LSTM. Combining the two can better combine the advantages of the two networks and achieve advantage compensation. Therefore, using Transformer and CNN to train the extracted human behavior features is conducive to obtaining the optimal behavior recognition accuracy and shorter training time, reducing time and labor costs.

Images