A real-time WiFi signal gesture recognition method allowing user authentication

Abstract

The present application relates to a kind of real-time WiFi signal gesture recognition methods allowing user authentication, comprising: step 1: acquisition CSI data in different environments, pre-process CSI data, obtain gesture execution CSI data set;Step 2: according to gesture execution CSI data set, corresponding Doppler spectrum diagram is extracted;Step 3: according to Doppler spectrum diagram, corresponding arm movement acceleration model is constructed;Step 4: construct the dual-task deep neural network for user identification and gesture recognition;Step 5: arm movement acceleration model is input into dual-task deep neural network as training sample and it is trained;Step 6: user identification and gesture recognition are realized using the collaborative dual-task deep neural network trained. The method of the present application can realize the simultaneous identification of gesture and the user of execution gesture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of gesture recognition, and particularly relates to a real-time WiFi signal gesture recognition method allowing user authentication. BACKGROUND

[0002] Gesture generally refers to an action of limbs to express thoughts, emotions or attitudes. In daily life, gesture is also a very important non-verbal communication form and occupies a very important position. Even in the background of human-computer interaction, we still rely on gestures to convey messages, commands and even feelings to the computers around us.

[0003] Human gesture recognition is the core of smart home, security monitoring, intelligent education and virtual reality applications. At present, there are many gesture recognition schemes, including camera-based gesture recognition, wearable sensor-based gesture recognition, Doppler radar-based gesture recognition and WiFi signal-based gesture recognition.

[0004] Camera-based gesture recognition has satisfactory recognition accuracy, but the requirements of favorable lighting conditions and potential privacy issues hinder their popular implementation. Wearable sensor-based gesture recognition is not widely popularized because wearable devices are expensive and inconvenient to carry. Doppler radar-based gesture recognition mainly detects motion by using the Doppler effect and then restores the corresponding gesture through machine learning. However, since radar devices are currently the most expensive in terms of deployment and use, it is difficult to apply them on a large scale in daily life. WiFi signal-based gesture recognition uses the CSI (Channel State Information) of WiFi to realize gesture recognition, and the cost and complexity of deploying a WiFi-based gesture recognition system can be almost negligible. It only needs two commercial WiFi devices to realize gesture recognition and can be more widely applied in daily life.

[0005] Current WiFi signal-based gesture recognition has the problem of losing real-time due to too complex algorithms in pursuit of higher recognition accuracy, making it difficult to apply in actual systems. Moreover, most algorithms rarely consider user recognition, which is also a problem to be solved in actual applications. SUMMARY

[0006] In order to solve the above problems existing in the prior art, the present application provides a real-time WiFi signal gesture recognition method allowing user authentication. The technical problem to be solved by the present application is realized by the following technical scheme:

[0007] The application provides a real-time WiFi signal gesture recognition method allowing user authentication, comprising:

[0008] Step 1: Collecting CSI data in different environments, pre-processing the CSI data, and obtaining a gesture execution CSI data set;

[0009] Step 2: Extracting a corresponding Doppler spectrum graph according to the gesture execution CSI data set;

[0010] Step 3: Constructing a corresponding arm movement acceleration model according to the Doppler spectrum graph;

[0011] Step 4: Constructing a dual-task deep neural network for user recognition and gesture recognition;

[0012] Step 5: Inputting the arm movement acceleration model as a training sample into the dual-task deep neural network for training;

[0013] Step 6: Realizing user recognition and gesture recognition by using the trained collaborative dual-task deep neural network.

[0014] In an embodiment of the application, the gesture execution CSI data set comprises CSI data corresponding to a plurality of users respectively performing a plurality of gestures in different environments.

[0015] In an embodiment of the application, the step 1 comprises:

[0016] Step 1.1: Collecting CSI data in different environments, pre-processing the CSI data, and obtaining denoised CSI data;

[0017] Step 1.2: Extracting the denoised CSI data according to a preset segmentation threshold, and obtaining a gesture execution CSI data set.

[0018] In an embodiment of the application, the step 1.1 comprises:

[0019] Step 1.1.1: Removing high-frequency noise interference signals in the CSI data by using fast Fourier transform and a low-pass filter;

[0020] Step 1.1.2: Removing static components, low-frequency interference and burst noise interference signals in the CSI data by using a Butterworth band-pass filter;

[0021] Step 1.1.3: Eliminating phase offsets in the CSI data by conjugate multiplication of CSI data of two antennas, and obtaining denoised CSI data.

[0022] In an embodiment of the application, the step 1.2 comprises:

[0023] Step 1.2.1: performing time-frequency analysis on the denoised CSI data to obtain a corresponding denoised Doppler spectrum diagram;

[0024] Step 1.2.2: calculating the variance of the amplitude in the frequency domain of the denoised Doppler spectrum diagram, and taking the denoised CSI data corresponding to the denoised Doppler spectrum diagram with a variance less than a preset segmentation threshold as a gesture execution CSI data set.

[0025] In an embodiment of the present application, the step 2 comprises:

[0026] Step 2.1: using principal component analysis to reduce the dimension and compress each gesture execution CSI data in the gesture execution CSI data set, and extracting a plurality of principal component components corresponding to each gesture execution CSI data;

[0027] Step 2.2: performing time-frequency analysis on each principal component component to obtain a corresponding Doppler spectrum diagram.

[0028] In an embodiment of the present application, the step 3 comprises:

[0029] Step 3.1: using a seam carving algorithm to obtain a dominant power carving path and a power boundary carving path corresponding to the Doppler spectrum diagram;

[0030] Step 3.2: constructing a corresponding arm movement acceleration model according to the dominant power carving path and the power boundary carving path to fill the gap between the body part acceleration sequence and the Doppler spectrum diagram power distribution;

[0031] wherein the arm movement acceleration model is:

[0032] ;

[0033] ;

[0034] wherein, represents a model using a Gaussian distribution for ; represents a relationship model between the power in the Doppler spectrum diagram and the body part superposition, represents an actual Doppler shift frequency extracted from the CSI signal, represents the number of frequency sets with short-time Fourier transform, i represents a frequency set, represents a weight of ; t represents time.

[0035] In one embodiment of the present application, the power in the Doppler spectrum is The relationship model between the body part superposition is:

[0036] ;

[0037] wherein c represents a proportional factor caused by propagation loss, K represents the number of body parts defining the gesture, represents the single reflection area S of the k th body part at time t represents the Doppler shift frequency of the th body part at time k represents the Doppler shift frequency of the t th body part at time

[0038] In one embodiment of the present application, the dual-task deep neural network comprises a feature extraction module, a temporal modeling module, a splicing module and an identification module, wherein,

[0039] The feature extraction module comprises a first feature extraction unit and a second feature extraction unit, the first feature extraction unit is used to extract the gesture spatial feature sequence in the input arm movement acceleration model, and the second feature extraction unit is used to extract the user spatial feature sequence in the input arm movement acceleration model.

[0040] The temporal modeling module comprises a first temporal modeling unit and a second temporal modeling unit connected with the corresponding first feature extraction unit and second feature extraction unit respectively, the first temporal modeling unit is used to perform temporal analysis on the input gesture spatial feature sequence to obtain the corresponding gesture temporal feature sequence, and the second temporal modeling unit is used to perform temporal analysis on the input user spatial feature sequence to obtain the corresponding user temporal feature sequence.

[0041] The splicing module is connected with the first temporal modeling unit and the second temporal modeling unit respectively, and is used to splice the input gesture temporal feature sequence and user temporal feature sequence to obtain a spliced feature sequence.

[0042] The identification module comprises a user identification unit and a gesture identification unit, and is used to identify the input spliced feature sequence to obtain a gesture identification result and a user identification result.

[0043] Compared with the prior art, the present application has the following advantages:

[0044] The application allows a real-time WiFi signal gesture recognition method for user authentication, finds original features that can reflect motion transformation by denoising and transforming CSI signals, and then establishes an arm motion acceleration model that can meet real-time and cross-domain capabilities according to the original features, uses a dual-task deep neural network for user recognition and gesture recognition to realize simultaneous recognition of gestures and users performing gestures.

[0045] The above description is only a summary of the technical solutions of the application, in order to enable the technical means of the application to be implemented according to the content of the specification, and in order to enable the above and other purposes, characteristics and advantages of the application to be more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Fig. 1 is a schematic diagram of a real-time WiFi signal gesture recognition method for user authentication provided by an embodiment of the application;

[0047] Figure 2 Fig. 2 is a flowchart of a real-time WiFi signal gesture recognition method for user authentication provided by an embodiment of the application;

[0048] Figure 3 Fig. 3 is a schematic diagram of a carving path of a Doppler spectrum;

[0049] Figure 4 Fig. 4 is a schematic diagram of a dual-task deep neural network provided by an embodiment of the application;

[0050] Figure 5 Fig. 5 is a schematic diagram of an environment for collecting provided by an embodiment of the application. DETAILED DESCRIPTION

[0051] In order to further illustrate the technical means and effects taken by the application to achieve the predetermined purpose, the following describes in detail a real-time WiFi signal gesture recognition method for user authentication according to the application in combination with the accompanying drawings and specific embodiments.

[0052] The foregoing and other technical contents, characteristics and effects of the application can be clearly presented in the following detailed description of the specific embodiments in combination with the accompanying drawings. Through the description of the specific embodiments, the technical means and effects taken by the application to achieve the predetermined purpose can be more deeply and specifically understood, however, the accompanying drawings are provided for reference and explanation only, and are not used to limit the technical solutions of the application.

[0053] Embodiment One

[0054] First, the basic principle of CSI data can be used for Wi-Fi sensing is introduced. Channel state information (CSI) data describes how a wireless signal propagates from a transmitter to a receiver at a specific carrier frequency, the amplitude and phase of CSI are affected by multipath effects, including amplitude attenuation and phase shift. The CSI in each data packet represents the channel frequency response (CFR) expressed as:

[0055] (1) ;

[0056] where, is the amplitude attenuation factor, is the propagation delay, is the carrier.

[0057] Since the amplitude and phase of CSI are affected by the location of the transmitter and receiver and the movement of surrounding objects and people. That is, CSI can capture the wireless characteristics of the nearby environment. With the help of mathematical modeling or machine learning algorithms, these characteristics can be used to implement gesture recognition.

[0058] Wireless channels that use MIMO (Multiple-Input Multiple-Output) are divided into multiple subcarriers by OFDM (Orthogonal frequency-division multiplexing) technology. To measure CSI, WIFI transmitters send long training symbols (LTFs) as the beginning of each data packet, which contain predefined symbols for each subcarrier. When receiving LTFs, WIFI receivers estimate the CSI matrix using the received signal and the original LTFs. For each subcarrier, the modeling method for the WiFi channel is where, is the received signal, is the transmitted signal, is the CSI matrix, is the noise vector. The receiver estimates the CSI matrix using the predefined signal and the received signal after removing the cyclic prefix, demapping, and OFDM demodulation, etc. The estimated CSI is a three-dimensional matrix of complex values.

[0059] Please refer to Figure 1 and Figure 2 , Figure 1 is a schematic diagram of a real-time WiFi signal gesture recognition method allowing user authentication provided by an embodiment of the present application; Figure 2is a flow chart of a real-time WiFi signal gesture recognition method allowing user authentication provided by an embodiment of the present application. As shown in the figure, the real-time WiFi signal gesture recognition method allowing user authentication of the present embodiment comprises:

[0060] Step 1: Collect CSI data in different environments, and pre-process the CSI data to obtain a gesture execution CSI data set;

[0061] Since in a real WiFi system, the measured CSI will be affected by the multipath channel, the transmission and processing, and the hardware / software error, the measured CSI from the baseband to the baseband is:

[0062] (2);

[0063] Among them, the first term is the multipath channel, the second term is the cyclic shift diversity, the third term is the sampling time offset, the fourth term is the sampling frequency offset, and the last term is the beamforming. The in the first term refers to the path length of the th path sent from the th transmitting antenna to the th receiving antenna, is the carrier frequency, is the time delay of the th transmitting antenna cyclic shift diversity (CSD), is the sampling time offset (STO), is the sampling frequency offset (SFO), and are the amplitude attenuation and phase offset of the beamforming matrix, respectively. The application of WIFI sensing needs to extract the multipath channel containing the change information of the surrounding environment, so the corresponding signal processing technology is needed to eliminate the influence of CSD, STO, SFO and beamforming.

[0064] The key to restoring the motion signal is that the changes caused by body motion are related in different CSI streams, and the reason for this correlation is that the CSI streams of different subcarriers are linear combinations of the same set of time-varying signals. Assuming that a person moves a small distance between time 0 and time t, this will cause a change in the path length of the signal In this case, there are where is the initial length of the path. When the initial phase offset is , the subcarrier in equation (2) seen by the receiver at time is The phase of the CSI at the kth subcarrier is as follows:

[0065] (3);

[0066] The time series of the CSI matrix illustrates the variation of the MIMO channel in different domains, such as time, frequency, and space. For a MIMO-OFDM channel with M transmit antennas, N receive antennas, and T subcarriers, the CSI matrix is a three-dimensional matrix , which represents the amplitude attenuation and phase shift of the multipath channel, where is a complex number, and each H represents the amplitude and phase of a subcarrier. The modulus and the argument of the complex number correspond to the amplitude and phase information, respectively. The CSI provides more information than the Received Signal Strength Indicator (RSSI). The three-dimensional CSI matrix is similar to a digital image with N x M spatial resolution and K color channels, and thus, the four-dimensional CSI tensor provides additional information in the time domain. In view of the actual situation, the collected CSI data needs to be preprocessed before subsequent recognition operations.

[0067] In this embodiment, step 1 includes:

[0068] Step 1.1: Collecting CSI data in different environments, and performing denoising processing on the CSI data to obtain denoised CSI data.

[0069] In this embodiment, the denoising processing on the CSI data is performed to remove irrelevant signal interference while ensuring that the gesture motion information is retained.

[0070] Since typical human activities only cause CSI changes with a frequency lower than 300 Hz. The original CSI contains significant static classification, low-frequency interference, and burst noise, which can make the Doppler shift containing motion information ambiguous. In a real commercial WiFi system, the original CSI measurement value contains phase shifts due to hardware and software errors.

[0071] In this embodiment, in order to remove the interference of the above signals, the following operations are specifically adopted:

[0072] Step 1.1.1: Removing high-frequency noise interference signals in the CSI data by using fast Fourier transform and a low-pass filter.

[0073] Step 1.1.2: Removing static components, low-frequency interference, and burst noise interference signals in the CSI data by using a Butterworth band-pass filter.

[0074] Step 1.1.3: Eliminate the phase shift in the CSI data by conjugate multiplication of the CSI data of two antennas to obtain the denoised CSI data.

[0075] It should be noted that since different antennas on the same Wi-Fi card share the same RF oscillator, the time-varying random phase shift of different antennas is the same, and therefore the conjugate multiplication method of calculating the CSI of two antennas on the same Wi-Fi card is adopted to eliminate the time-varying phase shift, as shown in formula (4), so that the out-of-band noise, quasi-static shift and random shift can be filtered out, and only the prominent multipath component of the non-zero DFS is retained.

[0076] (4);

[0077] wherein, is the output after conjugate multiplication, is the CSI of the first antenna, is the conjugate of the CSI of the second antenna, and are the moving path sets of the first antenna and the second antenna, respectively.

[0078] In the above equation, the product of the static path components of the two antennas is marked as (1), which can be considered as a constant in a short time, and it does not contain the Doppler velocity information we are interested in. However, the power of the static component can be very high because it contains a strong direct path signal. In order to avoid the interference of the static component on the Doppler velocity estimation, the static component is removed by subtracting the mean value from the conjugate multiplication.

[0079] In the above equation, the product of the moving path components is marked as (2), which is a very small value and can be ignored. (3) and (4) are two products of the static path component of one antenna and the dynamic path component of the other antenna, respectively, and these two items contain Doppler velocity information. Since two adjacent antennas have similar multipath, the Doppler velocity information in these two items has similar values and opposite directions. To obtain the Doppler velocity from the product of the dynamic path component of the first antenna and the static path component of the second antenna, that is, (3), therefore, the power of the static path component on the first antenna is reduced by subtracting the value of α, and the power of the static path component on the second antenna is increased by adding the value of β. Through the above power adjustment step, the item containing the correct Doppler velocity information has higher power in the multiplication output and can be identified in the spectrum diagram.

[0080] Step 1.2: Extract the denoised CSI data according to the preset segmentation threshold to obtain a gesture execution CSI data set.

[0081] In real-time, it is not necessary to process the original CSI measurement value at each moment, which will not only waste resources, but also produce unnecessary identification errors. Only when the current environment has human gesture activity, processing is needed.

[0082] When the subject is stationary, no Doppler effect is observed and the spectrogram only contains noise, so the energy distribution of the spectrogram is over the entire frequency band. In contrast, when the subject moves, the spectrogram is controlled by the Doppler effect and the energy of the spectrogram is concentrated on the frequency of interest. Therefore, the variance of the energy distribution in the frequency domain can be calculated and smoothed for motion detection.

[0083] It is worth noting that only the filter in the CSI denoising process will affect the variance, because the filter determines the degree to which out-of-band components are filtered out. Therefore, for a fixed filter, it is feasible to use a preset threshold, and when the variance is lower than the threshold, the player's movement is detected.

[0084] Specifically, step 1.2 includes:

[0085] Step 1.2.1: Time-frequency analysis of the denoised CSI data to obtain the corresponding denoised Doppler spectrogram;

[0086] Step 1.2.2: Calculate the variance of the amplitude in the frequency domain of the denoised Doppler spectrogram, and use the denoised CSI data corresponding to the denoised Doppler spectrogram with a variance less than a preset segmentation threshold as the gesture execution CSI data set.

[0087] In this embodiment, the gesture execution CSI data set includes CSI data corresponding to multiple gestures performed by multiple users in different environments.

[0088] Step 2: Extract the corresponding Doppler spectrogram from the gesture execution CSI data set;

[0089] Since the original CSI measurement value contains irrelevant or redundant signals, it is necessary to compress the signals in the gesture execution CSI data set, and then capture the motion information existing in different principal component components through time-frequency analysis method.

[0090] Specifically, step 2 includes:

[0091] Step 2.1: Use principal component analysis to reduce the dimension and compress each gesture execution CSI data in the gesture execution CSI data set, and extract a plurality of principal component components corresponding to each gesture execution CSI data;

[0092] Since the peaks and troughs have similar shapes across all CSI streams across different antenna pairs and different subcarriers, and the phases of the CSI streams vary smoothly across different subcarriers in the same antenna pair, secondly, there is no single CSI stream that can accurately reflect the motion information, which means that a method needs to be found to combine different CSI streams to obtain the best value that can reflect the motion information. Simply using a weighted average on the CSI streams cannot provide good results, because the phases of different CSI streams are different, and if they are added together, they can cancel each other out, that is, the peak value of one CSI stream can be the trough value of another CSI stream. Therefore, a better method needs to be found to combine the CSI streams so that the best value that can reflect the motion information can be generated.

[0093] In the present embodiment, the method of PCA (Principal Component Analysis) is used to find the correlation between the CSI streams. Through PCA, the correlation between the CSI streams according to time changes can be tracked, and the principal components of the CSI streams can be extracted by combining them in the best way. PCA is applied to the CSI streams mainly through the following four steps:

[0094] 1. Preprocessing: First, subtract the corresponding constant offset from each CSI stream to remove the static path component. It calculates the constant offset of each CSI stream by long-term averaging, such as calculating the average CSI amplitude of 4 seconds. Then, the CSI stream is cut into blocks containing samples obtained within one second interval, and the blocks of different CSI streams are arranged in columns to form a matrix In the present embodiment, the interval size is selected to be 1 second, so that the distance of the object moving is short, and at the same time the number of samples is large enough to ensure accurate correlation estimation.

[0095] 2. Correlation estimation: the correlation matrix is calculated as , the dimension of the correlation matrix is , where N is the number of CSI streams. For example, in a Tx-Rx link containing 30 subcarriers, N = 30 for one Tx-Rx link.

[0096] 3. Eigen decomposition: the correlation matrix is subjected to eigen decomposition to calculate the eigenvectors.

[0097] 4. Motion signal reconstruction: the principal components are constructed using the equation , where and are the eigenvectors and principal components, respectively.

[0098] In the present embodiment, the first principal component is discarded , the remaining five principal components are reserved for feature extraction. There is noise in all CSI streams caused by the change of internal state, which is captured together with the human motion signal due to the high correlation . However, all the information about the human motion signal captured in is also captured in other principal components, because according to formula (2), the phase of a subcarrier is a linear combination of two orthogonal components: and . Since the PCA components are uncorrelated, the first principal component contains only one of these orthogonal components, while the other component is reserved in the remaining principal components. Therefore, the first principal component can be safely discarded without losing any information.

[0099] It should be noted that the number of principal components used for feature extraction is selected empirically, and in this embodiment, the number of principal components is selected to be 5 to achieve a good trade-off between classification performance and computational complexity.

[0100] Step 2.2: Time-frequency analysis is performed on each principal component to obtain the corresponding Doppler spectrogram.

[0101] Commonly used time-frequency analysis methods include Fast Fourier Transform (FFT), Short Time Fourier Transform (STFT), Discrete Hilbert Transform (DHT), and Discrete Wavelet Transform (DWT).

[0102] In this embodiment, in order to analyze the signal and obtain a variable that can reflect the motion change information by associating the time domain with the frequency domain, a short-time Fourier transform is used to perform time-frequency analysis on the principal components obtained by dimensionality reduction, and the corresponding Doppler power spectrum is extracted to obtain the corresponding Doppler spectrogram.

[0103] Specifically, a Gaussian window with a length less than 0.15s is applied in the STFT, because the amplitude and Doppler shift are almost constant in a short time, and the Doppler shift can be obtained by time-frequency analysis in a short window. Further zero padding is applied to generate a more fine-grained spectrogram, and finally, the non-overlapping spectrograms of all CSI segments are spliced together to generate the entire spectrogram.

[0104] It should be noted that due to the uncertainty principle, there is a lower limit to the range of actions that can be detected. Specifically, assuming the time length of the STFT data window is T, the frequency resolution of the spectrogram is , in order to correctly identify the frequency-shifted signal, the amplitude of the frequency shift must fall in the non-DC bin, corresponding to the minimum frequency. For a signal segment with constant frequency shift, the frequency shift should satisfy where V is the rate of change of the reflected signal path, and is the wavelength of the signal. Then, the sensitivity of the range R is:

[0105] (5).

[0106] Step 3: An arm movement acceleration model is constructed according to the Doppler spectrum diagram;

[0107] In order to realize real-time cross-domain gesture recognition and user identification, the CSI signal features need to be modeled, and a model that can reflect user information and has real-time cross-domain gesture information is constructed through signal combination of multiple WiFi receivers.

[0108] In this embodiment, an arm movement acceleration model is constructed by applying a seam carving algorithm to the Doppler frequency shift.

[0109] Specifically, step 3 includes:

[0110] Step 3.1: Use the seam carving algorithm to obtain the dominant power carving path and the power boundary carving path corresponding to the Doppler spectrum diagram;

[0111] Specifically, by observing the Doppler Frequency Shift (DFS) spectrum diagram, it is noted that both power and time characteristics can indicate individualized arm movements in terms of speed and rhythm. Specifically, the DFS spectrum diagram can separate the movements of different arm parts when they move at different speeds, because the reflection area changes with the change of a specific speed component over time. Through the power of the DFS spectrum diagram, two carving paths can be derived, one is the dominant power, which reflects the maximum dominant power in the DFS spectrum diagram, and the other is the power boundary, which describes the dominant power area and the speed boundary, as shown in the carving path diagram of the Doppler spectrum diagram Figure 3 .

[0112] In addition, arm gestures can usually be divided into some atomic movements in time sequence (e.g., drawing a straight line, a circular arc), for example, drawing a rectangle contains four lines in four different directions, and the switching between two adjacent atomic movements is called movement change, indicating movement pause / movement restart. A carving path called arm movement change pattern can be extracted to represent the time of movement change, i.e., the movement with time rhythm in the gesture drawing process.

[0113] Due to the obtained Doppler shift is caused by the movement of the arm, as shown in equation (6), the Doppler shift is related to the speed of different body parts, where is the set of dynamic paths . Therefore, the DFS spectrum with dimension is derived, where R and P are the number of transceiver links and PCA components, respectively, and F and T represent the sampling points in the frequency and time domains, respectively.

[0114] (6).

[0115] As mentioned above, users exhibit unique personalized styles when performing the same gesture, which in some cases will be rhythmically increased, decreased, or even paused. Therefore, the signals reflected by different body parts produce consistent movement change patterns and form corresponding DFS spectrum sequences. That is, the rhythmically increased, decreased, or even paused will usually cause significant moving speed fluctuations detected by the DFS spectrum, which occurs at a specific time of the peak value of the speed change in the time domain. However, the intensive calculation of the speed derivative operation sacrifices real-time performance, and in order to balance the calculation cost while retaining personalized features, two types of movement curves are derived from the DFS spectrum, namely the dominant power carving path and the power boundary carving path.

[0116] Step 3.2: According to the dominant power carving path and the power boundary carving path, construct the corresponding arm movement acceleration model to fill the gap between the body part acceleration sequence and the Doppler spectrum power distribution;

[0117] In this embodiment, the arm movement change pattern related to acceleration is extracted as a biometric feature of the main body parts (such as the wrist, elbow, and arm). However, since DFS only displays the power values of specific speed components over time, due to the superposition of speed components at the receiver, it cannot provide accurate fine-grained acceleration corresponding to the body parts; secondly, the arm movement change pattern requires derivative calculation of high-dimensional data, which is computationally intensive and cannot run in real time; in addition, the DFS spectrum contains too much irrelevant interference, resulting in unnecessary calculation and storage.

[0118] Therefore, by constructing a model to fill the gap between the body part acceleration sequence and the DFS spectrum power distribution, the model is an arm movement acceleration model.

[0119] Specifically, the arm movement acceleration model is:

[0120] (7);

[0121] ;

[0122] wherein, represents using a Gaussian distribution for the model after represents the power in the Doppler spectrum between the relationship model of the body part superposition, represents the actual Doppler shift frequency extracted from the CSI signal, i represents the number of frequency sets with short-time Fourier transform, represents the frequency set, represents the weight of t represents time.

[0123] Further, the derivation process of the above arm motion acceleration model is specifically described.

[0124] Since the power distribution of the spectrum changes with the change of the reflection area S at a specific Doppler shift of the instance t, a proportional factor c caused by the propagation loss can be used as to define the power in the DFS spectrum . Assuming that there are K body parts to define gestures, the relationship between the power in the Doppler spectrum and the body part superposition can be modeled as:

[0125] (8);

[0126] wherein, represents the single reflection area S of the K-th body part at time k represents the Doppler shift frequency of the K-th body part at time t k t

[0127] Due to the resolution of the Wi-Fi signal and the superposition estimation of the body part K, the accurate cannot be obtained, but an experimental approximation with acceptable calculation error and attenuation factor can be obtained. In order to be closer to the true situation and facilitate derivation, a Gaussian distribution is used for the superposition of body part motion and modeled as:

[0128] (9)

[0129] Since represents the moving speed , and , for a fixed ​​​The corresponding acceleration of each body part It can be represented as Assuming a connection between the continuous DFS spectrum and... Compared to the cumulative effect, it can be ignored. If the variance is given, the rate of change of power can be derived as equation (10). Due to the limitations of the rigid body and acceleration extensibility of the human body, the derivative of the power power can be appropriately extracted as shown in equation (11) to simplify the DFS power change and acceleration. The relationship between them:

[0130]

[0131] Therefore, it can be observed that the rate of change of power K in all body parts increases with... As the number of users increases, personalized acceleration sequences that change over time can be detected as a biometric feature of the user by calculating the derivative of the DFS power spectrum graph when the user performs a gesture.

[0132] Since derivative calculations involve a large amount of redundant data, this paper draws inspiration from the seam carving problem in computer graphics, which is used for content-aware image resizing. In this embodiment, edge detection is used to filter redundant interference, and a difference scheme of convolution operators is used to optimize derivative calculations. Then, based on the seam carving algorithm, an effective method is designed to generate multiple main carving paths as mentioned above, serving as the arm motion variation model on each power spectrum spectrum in the PCA component. It is assumed that estimates of K main carving paths are considered, each path showing the most significant arm motion variation pattern over time. a frequency set and Used in a data packet to indicate The weights. Therefore, the optimal arm movement variation pattern (AMAM) along the frequency axis, as a function of the timestamp index, can be defined as shown in Equation (7).

[0133] It is worth noting that, in order to improve computational efficiency, one can use the DFS spectrum diagram. The Sobel operator is applied to the time axis to obtain the time gradient matrix for each power spectrum.

[0134] Specifically, the algorithm for extracting the arm motion acceleration model is as follows:

[0135]

[0136] The algorithm described above details the extraction process of the arm motion acceleration model. Note the number of segments. Specifying appropriate values ​​is very important because if Too large, segmentation can be instantaneous, and cannot guarantee to extract robust features with small sliding windows. Correspondingly, for smaller , the unique arm movement change pattern of a single user is too average to be distinguished with larger sliding windows. Moreover, the adaptive algorithm is too large in computation to be executed in real time.

[0137] In the embodiment, the setting is a constant value 60, so that the total sample length per segment duration is limited to 35ms-70ms, which is within the proper segmentation criterion range of DFS spectrogram. In addition, the experimental study of arm movement change pattern also shows that the most obvious difference between adjacent engraved paths is within the range of less than 70ms. Then, the resolution is set to 0.16m / s within the speed range of [-1.6, 1.6] to achieve 20 velocity bins. After generating the arm movement acceleration model (AMAM), it is input into the subsequent constructed dual-task deep neural network.

[0138] Step 4: Constructing a dual-task deep neural network for user identification and gesture identification;

[0139] In the embodiment, the dual-task deep neural network includes a feature extraction module, a temporal modeling module, a splicing module, and an identification module.

[0140] Among them, the feature extraction module includes a first feature extraction unit and a second feature extraction unit, the first feature extraction unit is used to extract the gesture spatial feature sequence in the input arm movement acceleration model, and the second feature extraction unit is used to extract the user spatial feature sequence in the input arm movement acceleration model.

[0141] The temporal modeling module includes a first temporal modeling unit and a second temporal modeling unit connected with the corresponding first feature extraction unit and the second feature extraction unit respectively, the first temporal modeling unit is used to perform temporal analysis on the input gesture spatial feature sequence to obtain the corresponding gesture temporal feature sequence, and the second temporal modeling unit is used to perform temporal analysis on the input user spatial feature sequence to obtain the corresponding user temporal feature sequence.

[0142] The splicing module is connected with the first temporal modeling unit and the second temporal modeling unit respectively, and is used to splice the input gesture temporal feature sequence and user temporal feature sequence to obtain a spliced feature sequence; the identification module includes a user identification unit and a gesture identification unit, and is used to identify the input spliced feature sequence to obtain a gesture identification result and a user identification result.

[0143] In the present embodiment, deep learning is used to extract domain-independent features for gesture recognition. A dual-task deep neural network is constructed to achieve gesture recognition and user recognition simultaneously and meet real-time requirements. The first task of the dual-task deep neural network is to extract domain-independent features to recognize different gestures, addressing the cross-domain requirement. The second task addresses the user recognition requirement.

[0144] Specifically, the dual-task deep neural network receives an arm motion acceleration model (AMAM) and reshapes the model into a digital image similar to a digital image with a spatial resolution of and color channels of , where represents the number of velocity sets, represents the number of receivers, represents the number of PCA principal components. The basic principle of the modification is that the signals on the receivers convey angle of arrival (AoA) information, while the velocity sets contain the motion of body parts, and the PCA components are set as color channels with different signal scales.

[0145] Referring to the schematic diagram of the dual-task deep neural network shown in Figure 4 . First, the feature extraction module extracts spatial features from a single arm motion change model, and then analyzes the time dependence of the entire feature sequence. In the present embodiment, a gated recurrent unit (GRU) network based on a convolutional neural network (CNN) is used, and the input tensor is sent to the CNN, which successively contains 16 3x3 filters and two 64-unit dense layers. ReLU functions and flatten layers are used for nonlinear feature mapping and dimension reshaping. Finally, the CNN outputs are used to represent sampling components.

[0146] Next, the spatial feature sequence is sent to the subsequent GRU (temporal modeling module) for temporal analysis. In the present embodiment, a single-layer GRU with 128 units is used to analyze the temporal relationship. To avoid overfitting, a dropout layer is further added, and then a Softmax layer with cross-entropy loss is added for dual-task prediction. The sequential model is a linear stack of CNN network layers.

[0147] Finally, the gesture temporal feature sequence and the user temporal feature sequence are spliced through a splicing module, and then the spliced feature sequence is input into the subsequent gesture recognition unit and user recognition unit, respectively, for gesture and user recognition.

[0148] Step 5: input the arm movement acceleration model as a training sample into the dual-task deep neural network to train it;

[0149] Specifically, the network training method is similar to the existing network training method, which will not be repeated here.

[0150] Step 6: use the trained collaborative dual-task deep neural network to realize user recognition and gesture recognition.

[0151] It should be noted that when using the trained collaborative dual-task deep neural network to realize user recognition and gesture recognition, the collected CSI data needs to be denoised, signal extracted, dimensionality reduced and compressed, time-frequency analyzed to obtain the corresponding Doppler spectrum, and the Doppler spectrum is used to construct an arm movement acceleration model, and then the arm movement acceleration model is input into the trained collaborative dual-task deep neural network for user recognition and gesture recognition. Specifically, the processing method and modeling process are referred to the processing and modeling process of the training sample described above, which will not be repeated here.

[0152] The real-time WiFi signal gesture recognition method allowing user authentication of the embodiment finds the original features that can reflect the motion transformation by denoising and transforming the CSI signal, and then establishes an arm movement acceleration model that can meet the real-time and cross-domain requirements according to the original features. The dual-task deep neural network for user recognition and gesture recognition is used to realize the simultaneous recognition of gestures and users performing gestures.

[0153] Embodiment Two

[0154] The embodiment illustrates the real-time WiFi signal gesture recognition method allowing user authentication described in Embodiment One through actual application.

[0155] The above method is connected to the Yo-Yo smart education platform of the smart education innovation laboratory of Xi'an University of Electronic Science and Technology, and the entire system will be further described below.

[0156] Dataset:

[0157] Four kinds of gestures commonly used in smart education are selected as our data collection gestures, which are 1. raising hands, 2. putting hands down, 3. waving hands, and 4. clapping hands.

[0158] As Figure 5The collection environment is shown, using a notebook as the transmitter, three notebooks as the three receivers, the transmitter activates one antenna, each receiver activates three antennas, the transmitter and receiver are selected from Intel 5300 wireless network card, and 802.11n CSI Tool is used as the collection tool, the monitor mode is selected, the packet sending interval is set to 1000Hz, the number of packets is 30000, the length of the packet is 100, the packet sending time is 30s, and the data is collected under 2.4GHz and 5GHz frequency bands respectively, and the data format is dat data.

[0159] There are 6 subjects in total, and data collection is carried out in two environments: 1. Rest room, 2. Meeting room, the subjects need to face three different directions and stand in three different positions to collect data, 3 min for each collection, and the subjects are told by a self-made audio file when to perform gestures.

[0160] The time allocation for 3 min of collection is as follows: after the subject hears the audio prompt, the corresponding gesture is performed, a new sample is generated every 3 seconds, that is, a gesture is collected every 3s (set back and forth gestures such as raising and lowering hands as a cycle), then collect for 30s, that is, 10 samples; rest for 15s, a total of 4 times in 3min, a total of 40 samples.

[0161] A total of 864 times of collection are required, a total of 103680 samples, 51840 samples for each signal, and each sample contains 3 seconds of data. Each gesture has 4320*2 (two signals) samples.

[0162] System environment:

[0163] Transmitter: 1 Think-pad T500; receiver: 3 Think-pad T500; network card: Intel 5300 NIC; processor CPU: 12th Gen Intel(R) Core(TM) i7-12700; processor GPU: NVIDIA 3090 24GB; linux kernel version: 3.2-4.2; receiving tool: linux-802.11n-CSI tool; transmission protocol: 802.11n; frequency band: 2.4G; bandwidth: 20MHZ; channel: 13; network card mode: monitor mode; collection frequency: 1000Hz; system platform: Yo-~Yo wisdom education platform. Matlab, pyhon and C language are used.

[0164] Experimental results:

[0165] The embodiment is realized by relying on a Yo-Yo intelligent education platform, gesture recognition accuracy and user recognition accuracy are calculated according to the method, real-time performance is evaluated, finally gesture recognition accuracy is 93.6%, under the premise of correct gesture recognition, user recognition accuracy is 92.3%; local response time is 400ms (the local response time represents the whole algorithm part delay of the application), system response time is 600ms (the system response time represents the whole Yo-Yo system gesture recognition delay)

[0166] It should be noted that, in this document, the terms such as first and second are used merely to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations exist in any actual relationship or order. Moreover, the terms "include", "contain" or any other variant are intended to cover non-exclusive inclusion, so that the article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the article or device including the element.

[0167] The above is a further detailed description of the application in combination with specific preferred embodiments, and the specific implementation of the application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, a number of simple deductions or substitutions can be made without departing from the concept of the application, and all should be considered within the protection scope of the application.

Claims

1. A real-time WiFi signal gesture recognition method allowing user authentication, characterized in that, The method comprises the following steps: Step 1: collecting CSI data in different environments, and pre-processing the CSI data to obtain gesture execution CSI data sets; Step 2: extracting corresponding Doppler spectrum graphs according to the gesture execution CSI data sets; Step 3: constructing a corresponding arm movement acceleration model according to the Doppler spectrum graphs; the step 3 comprises: Step 3.1: obtaining a dominant power carving path and a power boundary carving path corresponding to the Doppler spectrum graph by using a joint carving algorithm; Step 3.2: constructing a corresponding arm movement acceleration model according to the dominant power carving path and the power boundary carving path to fill the gap between the body part acceleration sequence and the Doppler spectrum graph power distribution; Wherein, the arm movement acceleration model is: s.t. |t i -t i-1 |<2; i = 1,..., F D ; wherein, represents a model using a Gaussian distribution for P ds (f D ,t) after the model, P ds (f D ,t) represents a power P ds in a Doppler spectrum, f D represents an actual Doppler shift frequency extracted from the CSI signal, F D represents the number of frequency sets with short-time Fourier transform, i represents a frequency set, represents a weight of , t represents time; The power P in the Doppler spectrum ds The relationship model between the superposition of the body part is: where c denotes a proportionality factor due to propagation loss, K denotes the number of body parts defining the gesture, Ref(k, t) denotes a single reflection area S of the kth body part at time t, f dfs (k, t) denotes the Doppler shift frequency of the kth body part at time t; Step 4: constructing a dual-task deep neural network for user identification and gesture identification; Step 5: inputting the arm movement acceleration model as a training sample into the dual-task deep neural network to train the dual-task deep neural network; Step 6: using the trained collaborative dual-task deep neural network to realize user identification and gesture identification. 2.The real-time WiFi signal gesture recognition method of allowing user authentication according to claim 1, wherein, The gesture execution CSI data set comprises CSI data corresponding to a plurality of gestures performed by a plurality of users in different environments. 3.The real-time WiFi signal gesture recognition method of allowing user authentication according to claim 2, wherein, The step 1 comprises: Step 1.1: collecting CSI data in different environments, and pre-processing the CSI data to obtain denoised CSI data; Step 1.2: extracting the denoised CSI data according to a preset segmentation threshold to obtain gesture execution CSI data sets. 4.The real-time WiFi signal gesture recognition method of allowing user authentication according to claim 3, wherein, The step 1.1 comprises: Step 1.1.1: removing high-frequency noise interference signals in the CSI data by using fast Fourier transform and a low-pass filter; Step 1.1.2: removing static components, low-frequency interference and burst noise interference signals in the CSI data by using a Butterworth band-pass filter; Step 1.1.3: eliminating phase offset in the CSI data by conjugate multiplication of the CSI data of two antennas to obtain denoised CSI data. 5.The real-time WiFi signal gesture recognition method of allowing user authentication according to claim 3, wherein, The step 1.2 comprises: Step 1.2.1: performing time-frequency analysis on the denoised CSI data to obtain corresponding denoised Doppler spectrum graphs; Step 1.2.2: calculating the variance of the amplitude in the frequency domain of the denoised Doppler spectrum graph, and taking the denoised CSI data corresponding to the denoised Doppler spectrum graph with a variance less than a preset segmentation threshold as the gesture execution CSI data set. 6.The real-time WiFi signal gesture recognition method of allowing user authentication according to claim 1, wherein, The step 2 comprises: Step 2.1: using principal component analysis to reduce the dimension and compress each gesture execution CSI data in the gesture execution CSI data set, and extracting a plurality of principal component components corresponding to each gesture execution CSI data; Step 2.2: performing time-frequency analysis on each principal component component to obtain a corresponding Doppler spectrum graph. 7.The real-time WiFi signal gesture recognition method of allowing user authentication according to claim 1, wherein, The dual-task deep neural network comprises a feature extraction module, a time modeling module, a splicing module and an identification module, wherein, The feature extraction module comprises a first feature extraction unit and a second feature extraction unit, the first feature extraction unit is configured to extract gesture space feature sequences in the input arm motion acceleration model, and the second feature extraction unit is configured to extract user space feature sequences in the input arm motion acceleration model; The temporal modeling module comprises a first temporal modeling unit and a second temporal modeling unit, the first temporal modeling unit is connected with the first feature extraction unit, and the second temporal modeling unit is connected with the second feature extraction unit, the first temporal modeling unit is configured to perform temporal analysis on the input gesture space feature sequences to obtain corresponding gesture temporal feature sequences, and the second temporal modeling unit is configured to perform temporal analysis on the input user space feature sequences to obtain corresponding user temporal feature sequences; The splicing module is connected with the first temporal modeling unit and the second temporal modeling unit respectively, and is configured to splice the input gesture temporal feature sequences and the user temporal feature sequences to obtain spliced feature sequences; The recognition module comprises a user recognition unit and a gesture recognition unit, and is configured to recognize the input spliced feature sequences to obtain gesture recognition results and user recognition results.

Images