A cross-domain wireless action recognition method based on sample generation and modality fusion

Abstract

The present invention provides a cross-domain wireless action recognition method based on sample generation and modal fusion. In order to realize high-precision cross-domain action recognition with a small amount of data, the same action category data of the source domain and the target domain are linearly transformed to obtain new data containing the environmental information of the source domain and the target domain, and then the environmental information of the source domain is removed and the environmental information of the target domain is enhanced by using a generative encoder and a generative decoder. Through continuous training, a large amount of virtual amplitude data and virtual phase data of the target domain are generated. In order to further improve the accuracy of cross-domain action recognition, the virtual amplitude and virtual phase are feature fused. The action features are extracted from the amplitude and phase by a fusion encoder, and the action features of different modalities are made consistent by similarity loss, and then the action feature extraction capability is enhanced by a fusion decoder. Finally, the action features of different modalities are channel spliced, and the spliced ​​action features are classified by an action classifier to obtain the action recognition result.

Description

Technical Field

[0001] The invention relates to a motion recognition technology, and in particular to a sample-based motion recognition technology based on WiFi. Background Art

[0002] Motion recognition is of great significance to protecting people's health and safety, and can be widely used in daily applications such as fall detection, smart home, public safety, etc. With the advent of an aging society, the phenomenon of empty-nest elderly people has become common, and it has become increasingly important to protect the health and safety of elderly people living alone at home.

[0003] The channel state information CSI (Channel State Information) of WiFi devices is used to describe the attenuation factor of the signal on each transmission path. It can reflect information such as signal scattering, environmental attenuation, and distance attenuation. When people perform different actions in a space covered by WiFi, the propagation of WiFi signals will be affected differently, causing different changes in the amplitude and phase of the collected CSI. Based on this principle, we can perform motion recognition by analyzing CSI data. Motion recognition based on WiFi can realize fall detection and abnormal behavior monitoring, and timely alarm when an elderly person has an accident so that timely assistance can be obtained. Motion recognition based on WiFi can also be applied to smart homes. By recognizing user actions, intelligent control of the home can be achieved to enhance the intelligent experience; by detecting intrusion behavior, intelligent security can be achieved to ensure home safety. Motion recognition based on WiFi can also be applied to public safety. For example, for campus violence that occurs in non-monitored areas, motion recognition can be used to detect and stop violent behavior in a timely manner to ensure the safety of students on campus. There are countless similar applications. Existing motion recognition technologies are generally based on video surveillance or wearable devices. Video surveillance can achieve high-precision motion recognition, but it needs to work under light conditions and there is a risk of privacy leakage. Wearable devices do not pose a risk of privacy leakage and do not require lighting conditions, but they require users to wear the device all the time, which brings inconvenience to users.

[0004] Most of the existing WiFi-based motion recognition solutions can only solve the motion recognition in a single scenario. The basic idea is to collect a large amount of labeled motion data under the supervised learning framework, and use this data to train the classifier to achieve the classification and recognition of the motion.

[0005] The applicant proposed two action recognition methods 201911148775.2 and 202010950803.9 in November 2019 and September 2020, respectively, both of which are aimed at reducing the workload of labeling. Part of the collected CSI samples are labeled as source domain data, and the other part is not labeled as target domain data, and then the target domain data is pseudo-labeled. Finally, the source domain data and the pseudo-labeled target domain data are used together as training data. However, these two methods can only reduce the workload of labeling. It is still necessary to collect a large amount of action sample data in various scenarios and mark the corresponding action labels or pseudo-labels. The classifier is trained with these large amounts of labeled or pseudo-labeled sample data to classify and identify different actions. The disadvantage of this type of method is that the action recognition effect will drop sharply when the domain changes. The domain changes here include user changes, user position changes, or scene changes. In order to adapt to domain changes and maintain recognition accuracy, it is necessary to collect a large amount of action data and retrain the model. However, it is too difficult to collect action data in various domains, so it is not suitable for practical applications. Summary of the invention

[0006] In order to solve the problem of reduced recognition accuracy caused by reliance on a large amount of sampled data and environmental changes in existing WiFi-based action recognition methods, the present invention provides a method for cross-domain action recognition with a small sample size.

[0007] The technical solution adopted by the present invention to solve the above technical problems is a cross-domain wireless action recognition method based on sample generation and modality fusion, comprising the steps of:

[0008] 1) Sensing environment deployment steps: deploy a pair of WiFi transmitters and WiFi receivers in the sensing area, and the WiFi signal needs to cover the entire sensing area;

[0009] 2) CSI data collection steps: divide the position in the sensing area, set the action type, and there are c actions in total; select designated users and designated positions, and each designated user repeats c actions at each designated position, and each action is performed N times; the remaining undesignated users collect K times for each action at each undesignated position, K<<N; during the action execution, collect the channel state information CSI data corresponding to the action and mark it;

[0010] 3) Data division step: The CSI data sampled by the specified user at the specified location is used as the source domain data set The CSI data sampled from unspecified users at unspecified locations is used as the target domain support set. in, Indicates that the action is classified as c i The source domain CSI data, Indicates that the action is classified as c i The category label of Indicates that the action is classified as c i Target domain data;

[0011] 4) Virtual sample generation step: based on the source domain dataset X S With the target domain support set X T Generate virtual samples of the target domain:

[0012] 4-1) Select the same category label Source domain CSI data With the target domain CSI data Generate approximate virtual data in the target domain through linear transformation Will iterate over the category labels The source domain data and target domain data structure, for action classification c i N×K target domain approximate virtual data will be generated; the weight α of the source domain CSI data is a value close to 0 obtained by encoder training, and the weight β of the target domain CSI data is a value close to 1 obtained by encoder training, with a value range of approximately α∈[-0.1,0.15], β∈[0.85,1.1];

[0013] 4-2) Use the generative encoder to approximate virtual data in the target domain Extract action features to obtain action features Then use the generated decoder to parse the action features Reconstruct and obtain reconstructed virtual data

[0014] Among them, when generating encoders and decoders, the reconstruction loss is set so that the reconstructed virtual data Approximate to target domain data for the goal;

[0015] 4-3) Reconstruct data separately The amplitude and phase of the virtual amplitude x a With virtual phase x p ;

[0016] 5) Modal fusion step: Use the fusion encoder to respectively derive the virtual amplitude x a and virtual phase x p Perform feature extraction to obtain the virtual amplitude x a Action characteristics and virtual phase x p Action characteristics Then and Perform channel fusion to obtain fused modal action features

[0017] The training process of the fusion encoder is as follows: the fusion decoder connected to the fusion encoder is used to train and Reconstruct and obtain the reconstructed virtual amplitude x a ′ and the reconstructed virtual phase x′ p , set the reconstruction loss so that the reconstructed virtual amplitude x a ′ and the reconstructed virtual phase x′ p Approximate virtual amplitude x a With virtual phase x p as the goal; and set the similarity loss, the similarity loss is and The goal is to maximize similarity;

[0018] 6) Input the fused action feature f as training set data into the action classifier to train the action classifier;

[0019] 7) The amplitude and phase of the CSI data to be classified are input into the trained fusion encoder. The fusion encoder extracts the motion features of the amplitude and the phase and then performs channel fusion. Finally, the fused modal motion features are input into the trained motion classifier to complete motion recognition.

[0020] The present invention extracts amplitude and phase information from CSI data to identify actions based on the impact of different actions performed by the experimenter in the perception area on the WiFi signal. In order to achieve high-precision cross-domain action recognition with a small amount of data, the same action category data of the source domain and the target domain are linearly transformed to obtain new data containing the environmental information of the source domain and the target domain, and then the environmental information of the source domain is removed and the environmental information of the target domain is enhanced by using a generative encoder and a generative decoder. Through continuous training, a large amount of virtual amplitude data and virtual phase data of the target domain are generated. In order to further improve the accuracy of cross-domain action recognition, feature fusion is performed on the virtual amplitude and virtual phase. The action features are extracted from the amplitude and phase by a fusion encoder, and the action features of different modes are made consistent by similarity loss, and then the fusion decoder is used to enhance the action feature extraction capability. Finally, the action features of different modes are channel spliced, and the spliced ​​action features are classified by an action classifier to obtain the action recognition result.

[0021] The beneficial effect of the present invention is that, through the sample generation step, the present invention only needs a few target domain (transformed environment) real samples to generate a large number of target domain virtual samples, thereby solving the problem of requiring a large amount of data in the existing method and realizing small sample cross-domain action recognition. In order to further improve the accuracy of cross-domain action recognition, the present invention simultaneously extracts amplitude and phase information from CSI, and performs feature mode fusion on amplitude and phase, thereby solving the problem of low accuracy of the existing method in the case of small samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is an environment deployment diagram of an embodiment;

[0023] Figure 2 This is the action classification training flow chart of the present invention. DETAILED DESCRIPTION

[0024] The embodiment is carried out in the area covered by WiFi. A pair of WiFi transmitters and WiFi receivers are deployed in the sensing area, both of which are equipped with Intel WiFi Link 5300 wireless network cards. Each network card is equipped with 3 antennas, forming a total of 9 signal links. The CSI tool toolkit is used to obtain CSI information from the Intel WiFi Link 5300 network card. Each antenna pair can obtain 30 groups of subcarrier information, and each data packet has a total of 270 groups of subcarrier information.

[0025] The specific implementation steps are as follows:

[0026] Step 1: Deploy a pair of WiFi transmitters and WiFi receivers in the sensing area. The WiFi signal is required to cover the entire sensing area. The environment is deployed as follows: Figure 1 As shown, a pair of WiFi transmitters and WiFi receivers are deployed in the laboratory and the hall respectively.

[0027] Step 2: Collect CSI motion data in the sensing area. Select several locations, and each user performs several common actions at each location, including squatting, standing up, walking, jumping, falling, climbing up, etc. Each type of action is repeated several times. The sampling frequency is set to 100Hz.

[0028] Step 3: Select some locations or users as the source domain (environment before the change) to form the source domain dataset Each user repeats each action N times at each location. The remaining locations or users are used as target domains for cross-domain learning. Each user only needs to collect K times for each action at each location, K<<N, which differs by an order of magnitude or more, to form the target domain support set in, Indicates that the action is classified as c i The source domain CSI data, Indicates that the action is classified as c i The category label of Indicates that the action is classified as c i The target domain data.

[0029] The CSI subcarrier can be described as a complex form Extract the amplitude and phase from CSI, including:

[0030] Amplitude extraction is to Hi Take the absolute value to form the amplitude data set X a ={H i |};

[0031] There is an offset problem in the phase, which needs to be calibrated. The original measured phase can be expressed as:

[0032]

[0033] Among them, ∠H i is the true phase, m i is the sequence number of the ith subcarrier, N is the fast Fourier size, Δt is the time deviation, β is the phase deviation, and Z is the negligible measurement noise. Perform deconvolution to obtain a continuously changing phase and introduce coefficients and Represents the slope and mean of the measured phase, using the original measured phase minus (km i +b), the calibration phase can be obtained, that is, Forming the phase dataset X p = {∠H i};

[0034] The extracted amplitude and calibration phase are sampled at intervals so that the sample size of each action is 200×270. The sampled data can be expressed as:

[0035] x=[x1,x2,...,x 200 ]

[0036] Among them, 200 is the number of packets per action, x i =[h1,h2,...,h 270 ] is a data packet containing 270 subcarriers.

[0037] Step 4: Based on the idea of ​​sample generation, a generative encoder is used to generate virtual data in the target domain, thereby expanding the target domain training set, such as Figure 2 As shown:

[0038] Step 4-1: Select source domain data Same action category data as the target domain Both have the same category label Make linear changes to two data to form new data

[0039]

[0040] Among them, α and β are the weight values ​​of the two data, and the new data Contains action information, source domain environment information and target domain environment information. During the training process, when α→0, β→1, Approaching the target domain data Contains action information and target domain environment information, which can be used as target domain training set data;

[0041] Step 4-2: Generate encoder pair using Perform feature extraction to obtain action features

[0042]

[0043] The encoder is a convolutional neural network with 4 convolutional layers, with an input dimension of 200×270×1 and an output dimension of 22×31×4.

[0044] Step 4-3: Generate decoder pair using Reconstruct and generate virtual data

[0045]

[0046] The decoder has the opposite structure to the encoder and is also a 4-layer convolutional neural network with an input dimension of 22×31×4 and an output dimension of 200×270×1.

[0047] The generative encoder and the generative decoder use mean square error (MSE) as the loss function during training, and the virtual data Target domain data As MSE input, the loss function can be expressed as

[0048]

[0049] L MSE1 Train the linear weight values ​​α, β and the encoder and decoder weights so that the reconstructed data Approaching target domain data

[0050] Step 4-4: reconstruct the source domain data and the target domain data of the same action category in sequence, and generate N×K virtual data in the target domain for each type of action;

[0051] Perform the above steps 4-1 to 4-4 for the amplitude and phase of the CSI data respectively to generate a virtual amplitude x in the target domain. a With virtual phase x p ;

[0052] Step 5: Generate the virtual amplitude x in the target domain a and virtual phase x p Feature fusion is performed to improve the accuracy of cross-domain action recognition. The process of modal fusion is as follows: Figure 2As shown:

[0053] Step 5-1: Use the fusion encoder to respectively calculate the virtual amplitude x a and virtual phase x p Extract features and obtain amplitude action features Phase action characteristics The fusion encoder consists of 4 convolution blocks, each of which includes 1 convolution layer and 1 pooling layer. The fusion encoder input dimension is 200×270×1, and the output dimension is 11×15×4;

[0054] The cosine distance to amplitude action features is used to train the fusion encoder Phase action characteristics The similarity measure is expressed as

[0055]

[0056] where f i a With f i p They are and The elements after conversion to a one-dimensional vector.

[0057] Step 5-3: Amplitude action characteristics and phase action characteristics Use a fusion decoder consisting of 4 deconvolution layers to reconstruct the output as x a ′ and x′ p The input dimension of the fusion decoder is 11×15×4, and the output dimension is 200×270×1. To ensure that the fusion encoder in step 5-2 extracts action features rather than simple consistency encoding, MSE is also used as the reconstruction loss L when training the fusion encoder. MSE2 , which can be expressed as

[0058] L MSE2 =MSE(x a ,x a ′)+MSE(x p ,x′ p )

[0059] Step 5-4: Virtual Amplitude Characteristics With virtual phase characteristics The dimensions are 11×15×4, and Channel concatenation is performed to form an enhanced feature f' with a dimension of 11×15×8, expressed as

[0060]

[0061] Step 6: Convert the enhanced feature f' into a one-dimensional feature vector and use a 4-layer fully connected neural network for classification. The final classification result vector is Y = (y1, y2, ..., y c ), the action category y′ can be expressed as;

[0062]

[0063] After the training of the fully connected neural network is completed, for the CSI data to be classified, the amplitude and phase are directly input into the trained fusion encoder. The fusion encoder extracts the motion features of the amplitude and the motion features of the phase, and then performs channel fusion on the motion features of the virtual amplitude and the motion features of the virtual phase. The motion features of the fused modality are input into the trained motion classifier to complete motion recognition.

[0064] In the verification experiment, the user position change and environment change were regarded as domain changes, and 2 labeled data were collected for each type of action in the target domain. For cross-location action recognition in a single room, the target domain action recognition accuracy in the laboratory and the hall reached more than 85%. For cross-room action recognition between the laboratory and the hall, the target domain action recognition accuracy reached more than 80%.

Claims

1. A cross-domain wireless action recognition method based on sample generation and modality fusion, characterized in that: The following steps are involved: 1) Sensing environment deployment steps: deploy a pair of WiFi transmitters and WiFi receivers in the sensing area, and the WiFi signal needs to cover the entire sensing area; 2) CSI data collection steps: divide the position in the sensing area, set the action type, and there are c actions in total; select designated users and designated positions, and each designated user repeats c actions at each designated position, and each action is performed N times; the remaining undesignated users collect K times for each action at each undesignated position, K<<N, << indicates a less than relationship of one order of magnitude or more; during the action execution, collect the channel state information CSI data corresponding to the action and mark it; 3) Data division step: The CSI data sampled by the specified user at the specified location is used as the source domain data set The CSI data sampled from unspecified users at unspecified locations is used as the target domain support set. in, Indicates that the action is classified as c i The source domain CSI data, Indicates that the action is classified as c i The category label of Indicates that the action is classified as c i Target domain CSI data; 4) Virtual sample generation step: based on the source domain dataset X S With the target domain support set X T Generate virtual samples of the target domain: 4-1) Select the same category label Source domain CSI data With the target domain CSI data Obtain approximate virtual data in the target domain through linear transformation Will iterate over the category labels The source domain data and target domain data are composed of N×K target domain approximate virtual data. For each action classification, the weight α of the source domain CSI data is a value close to 0 obtained by training the generative encoder, and the weight β of the target domain CSI data is a value close to 1 obtained by training the generative encoder. 4-2) Use the generative encoder to approximate virtual data in the target domain Extract action features to obtain action features Then use the trained generative decoder to parse the action features Reconstruct and obtain reconstructed virtual data Among them, when generating encoders and decoders, the reconstruction loss is set so that the reconstructed virtual data Approximate to target domain data for the goal; 4-3) Reconstruct data separately The amplitude and phase of the virtual amplitude x a With virtual phase x p ; 5) Modal fusion step: Use the fusion encoder to respectively derive the virtual amplitude x a and virtual phase x p Perform feature extraction to obtain the virtual amplitude x a Action features to be fused and virtual phase x p Action features to be fused Then and Perform channel fusion to obtain fused modal action features The training process of the fusion encoder is as follows: the fusion decoder connected to the fusion encoder is used to train and Reconstruct and obtain the reconstructed virtual amplitude x′ a and the reconstructed virtual phase x′ p , the action feature extraction of the modality to be fused is controlled by setting the reconstruction loss and similarity metric; The reconstruction loss is such that the reconstructed virtual amplitude x′ a and the reconstructed virtual phase x′ p Approximate virtual amplitude x a With virtual phase x p The goal is to measure the similarity and The goal is to maximize similarity; 6) Input the fused modal action feature f as training set data into the action classifier to train the action classifier; 7) The amplitude and phase of the CSI data to be classified are input into the trained fusion encoder. The fusion encoder extracts the motion features of the amplitude and the phase and then performs channel fusion. Finally, the fused modal motion features are input into the trained motion classifier to complete motion recognition.

2. The method according to claim 1, characterized in that: The weight α of the source domain CSI data and the weight β of the target domain CSI data are trained by the generative encoder.

3. The method according to claim 1, characterized in that: The generative encoder and fusion encoder use convolutional neural networks.

Images