Gesture recognition method based on signal spectrum enhancement and related equipment

Abstract

The invention provides a signal spectrum enhancement-based gesture recognition method and related equipment, and the method comprises the steps: collecting the information of human body gesture actions affecting a WiFi channel state based on wireless sensing, obtaining real Doppler spectrum data, inputting the real Doppler spectrum data into a spectrum cleaning network for processing, generating a polluted spectrum data sample, and training the spectrum cleaning network, obtaining a trained spectrum cleaning network; inputting real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data; inputting the enhanced spectrum data into a deep complex network, and training the deep complex network to obtain a gesture recognition network; and the complex channel feature data related to the target human body gesture are sequentially input into the trained frequency spectrum cleaning network and the gesture recognition network for enhancement and recognition, a gesture recognition result is obtained, and the recognition precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and Internet of Things, and in particular to a gesture recognition method based on signal spectrum enhancement and related equipment. Background Art

[0002] With the widespread deployment of WiFi infrastructure, WiFi sensing technology is gradually attracting widespread attention from industry and academia. Compared with traditional human activity sensing systems based on dedicated sensors, wearable devices or cameras, the advantage of WiFi sensing is that it can be seamlessly integrated with existing communication equipment, does not require additional hardware deployment, and has non-contact and privacy-friendly characteristics. Thanks to the rich complex channel feature data in WiFi sensing, a variety of IoT sensing applications based on WiFi are emerging, including vital sign monitoring and indoor positioning. Although these applications have shown broad prospects, WiFi sensing technology still faces a major challenge in in-depth research: the limited bandwidth of current WiFi cannot provide sufficient spatial resolution, which in turn limits the performance of WiFi sensing systems in the above applications.

[0003] Specifically, traditional WiFi sensing systems usually use short-time Fourier transform (STFT) to generate spectrum as a sensing primitive. In this process, signal processing usually requires window function operations (such as rectangular window, Hanning window, etc.). However, the limited length of the window function will cause the waveform in the frequency domain to expand, which will cause the signal energy to "disperse" from the actual frequency component to the nearby frequency component, generate other spurious frequency components, and cause spectrum pollution. Traditional methods use appropriate window functions to reduce the impact of spectrum pollution. The window function reduces the edge effect by applying weights to the time domain signal, thereby alleviating the pollution problem. Although these window functions can reduce pollution to a certain extent, they cannot completely eliminate it, especially in high-noise or complex scenarios such as WiFi sensing, the effect is limited. At the same time, due to the limited bandwidth of WiFi, the signal energy is limited to a smaller frequency band, which makes the pollution problem more prominent, thereby blurring important frequency features and affecting the overall accuracy of the system.

[0004] In addition, traditional WiFi-aware classification network designs usually only focus on the spectrum amplitude and ignore the phase information in the spectrum, which leads to feature loss and degrades the classification performance of the network. Summary of the invention

[0005] The present invention provides a gesture recognition method based on signal spectrum enhancement and related equipment, the purpose of which is to reduce spectrum pollution and improve recognition accuracy.

[0006] In order to achieve the above object, the present invention provides a gesture recognition method based on signal spectrum enhancement, comprising:

[0007] Step 1: Based on wireless sensing, collect information about the impact of human gestures on the WiFi channel state and obtain real Doppler spectrum data;

[0008] Step 2: Input the real Doppler spectrum data into the constructed spectrum cleaning network for processing to generate contaminated spectrum data samples, and use the contaminated spectrum data samples to train the spectrum cleaning network to obtain a trained spectrum cleaning network;

[0009] Step 3, input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data;

[0010] Step 4, inputting the enhanced spectrum data into the constructed deep complex network, training the deep complex network, and obtaining a gesture recognition network;

[0011] Step 5, inputting the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and gesture recognition network in sequence for enhanced recognition to obtain the gesture recognition result;

[0012] The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process, the extraction module is used to extract features from the data, and the transformation module is used to transform the features output from the previous level.

[0013] The deep complex network includes a feature extraction module, a feature compression module, and a classification module which are connected in sequence. The input end of the feature extraction module is connected to the output end of the transformation module. The feature compression module is used to calculate the probability of various gestures, and the classification module outputs the gesture recognition result.

[0014] More specifically, step 1 includes:

[0015] In wireless sensing, a receiving device obtains original complex channel feature data by receiving a wireless signal sent by a transmitting device. The original complex channel feature data is used to characterize human gestures, including static components and dynamic components.

[0016] The original complex channel characteristic data is subjected to data dimension reduction and denoising through principal component analysis to obtain complex channel characteristic data;

[0017] The complex channel characteristic data is divided into windows by short-time Fourier transform to obtain a plurality of complex channel characteristic data blocks, wherein one window corresponds to one complex channel characteristic data block;

[0018] Fourier transform is performed on the complex channel characteristic data blocks in each window to obtain real Doppler spectrum data.

[0019] Specifically, the original complex channel characteristic data y is:

[0020]

[0021] Among them, h s (f) represents the static component, h d (f,t) represents the dynamic component, h s (f)+h d (f, t) = h(f, t), h(f, t) represents the gain matrix, a(f, t) represents the amplitude, represents the Doppler frequency shift corresponding to the dynamic path d, D d represents the dynamic path d, x represents the transmitted signal, and noise represents the noise.

[0022] More specifically, step 2 includes:

[0023] Input the real Doppler spectrum data into the constructed spectrum cleaning network;

[0024] In the spectrum cleaning network, a uniformly distributed carrier signal is generated within the human activity frequency range according to the sampling rate of the real Doppler spectrum data, and the frequency, amplitude and phase of the carrier signal are determined by statistical analysis of the real Doppler spectrum data;

[0025] Generate a window function according to the window type and window size selected in the short-time Fourier transform process, and perform convolution and padding processing on the window function and the carrier signal to obtain a windowed signal matching the sampling rate;

[0026] Perform fast Fourier transform on the windowed signal to obtain the convolution matrix of the window function in the frequency domain, and truncate the convolution matrix to obtain the ideal spectrum data sample;

[0027] After convolving the ideal spectrum data sample with the convolution matrix, additive Gaussian white noise is added to generate polluted spectrum data samples;

[0028] The spectrum cleaning network is trained using the polluted spectrum data samples to obtain a trained spectrum cleaning network.

[0029] Further, the extraction module includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a third two-dimensional convolutional layer connected in sequence;

[0030] The transformation module includes a first complex fully connected layer, a second complex fully connected layer, a third complex fully connected layer, and a fourth complex fully connected layer connected in sequence;

[0031] The input end of the first two-dimensional convolutional layer is the input end of the spectrum cleaning network, which is used to receive the real Doppler spectrum data;

[0032] The output end of the third two-dimensional convolutional layer is connected to the input end of the first complex fully connected layer;

[0033] The output end of the fourth complex fully connected layer is the output end of the spectrum cleaning network, which is used to output the optimal model parameters in the training stage and to output the enhanced spectrum data in the application stage.

[0034] Specifically, the feature extraction module includes:

[0035] A first complex convolution layer and a second complex convolution layer for performing complex convolution operations on input data to extract features, a first maximum pooling layer and a second maximum pooling layer for downsampling and compressing the extracted features;

[0036] The input end of the first complex convolution layer is the input end of the deep complex network and is used to be connected to the output end of the fourth complex fully connected layer. The output end of the first complex convolution layer is connected to the input end of the first maximum pooling layer, the output end of the second maximum pooling layer is connected to the input end of the second complex convolution layer, the output end of the second complex convolution layer is connected to the input end of the second maximum pooling layer, and the output end of the second maximum pooling layer is connected to the input end of the feature compression module for extracting gesture-related features.

[0037] Specifically, the feature compression module includes the following connected in sequence:

[0038] A fifth complex fully connected layer and a sixth complex fully connected layer for integrating the extracted gesture-related features, a first real fully connected layer and a second real fully connected layer for mapping the integrated features to the real domain, and a Softmax activation function layer for generating a probability distribution;

[0039] The input end of the fifth complex fully connected layer is connected to the output end of the second maximum pooling layer, and the output end of the Softmax activation function layer is the output end of the deep complex network.

[0040] The present invention also provides a gesture recognition device based on signal spectrum enhancement, comprising:

[0041] An acquisition module is used to collect information about the impact of human gestures on the WiFi channel state based on wireless sensing, and obtain real Doppler spectrum data;

[0042] A processing module is used to input the real Doppler spectrum data into the constructed spectrum cleaning network for processing, generate contaminated spectrum data samples, and train the spectrum cleaning network using the contaminated spectrum data samples to obtain a trained spectrum cleaning network;

[0043] An enhancement module is used to input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data;

[0044] A training module, used for inputting the enhanced spectrum data into the constructed deep complex network, training the deep complex network, and obtaining a gesture recognition network;

[0045] A recognition module is used to sequentially input the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network for enhanced recognition to obtain a gesture recognition result;

[0046] The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process, the extraction module is used to extract features from the data, and the transformation module is used to transform the features output from the previous level.

[0047] The deep complex network includes a feature extraction module, a feature compression module, and a classification module which are connected in sequence. The input end of the feature extraction module is connected to the output end of the transformation module. The feature compression module is used to calculate the probability of various gestures, and the classification module outputs the gesture recognition result.

[0048] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a gesture recognition method based on signal spectrum enhancement when executing the computer program.

[0049] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a gesture recognition method based on signal spectrum enhancement is implemented.

[0050] The above scheme of the present invention has the following beneficial effects:

[0051] The present invention collects information about the impact of human gestures on the WiFi channel state based on wireless perception, and obtains real Doppler spectrum data; inputs the real Doppler spectrum data into a constructed spectrum cleaning network for processing, generates contaminated spectrum data samples, and uses the contaminated spectrum data samples to train the spectrum cleaning network to obtain a trained spectrum cleaning network; inputs the real Doppler spectrum data into the trained spectrum cleaning network for enhancement, and obtains enhanced spectrum data; inputs the enhanced spectrum data into a constructed deep complex network, trains the deep complex network, and obtains a gesture recognition network; inputs complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhanced recognition, and obtains a gesture recognition result; compared with the prior art, the present invention restores the contaminated spectrum data to high-resolution spectrum data through the spectrum cleaning network, minimizes spectrum contamination, thereby improving the accuracy of wireless perception, and then recognizes the high-resolution spectrum data through the deep complex network, retains the structural characteristics of the data, and thereby improves the recognition accuracy.

[0052] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0054] Figure 2 It is a schematic diagram of the structure of a spectrum cleaning network and a deep complex network in an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of the structure of a gesture recognition device in an embodiment of the present invention;

[0056] Figure 4 Schematic diagram of the structure of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In the description of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0059] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0061] In view of the existing problems, the present invention provides a gesture recognition method based on signal spectrum enhancement and related equipment.

[0062] like Figure 1 As shown, an embodiment of the present invention provides a gesture recognition method based on signal spectrum enhancement, comprising:

[0063] Step 1: Based on wireless sensing, collect information about the impact of human gestures on the WiFi channel state and obtain real Doppler spectrum data;

[0064] Step 2: Input the real Doppler spectrum data into the constructed spectrum cleaning network for processing to generate contaminated spectrum data samples, and use the contaminated spectrum data samples to train the spectrum cleaning network to obtain a trained spectrum cleaning network;

[0065] Step 3, input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data;

[0066] Step 4, inputting the enhanced spectrum data into the constructed deep complex network, training the deep complex network, and obtaining a gesture recognition network;

[0067] Step 5, inputting the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and gesture recognition network in sequence for enhanced recognition to obtain the gesture recognition result;

[0068] The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process, the extraction module is used to extract features from the data, and the transformation module is used to transform the features output from the previous level.

[0069] The deep complex network includes a feature extraction module, a feature compression module, and a classification module which are connected in sequence. The input end of the feature extraction module is connected to the output end of the transformation module. The feature compression module is used to calculate the probability of various gestures, and the classification module outputs the gesture recognition result.

[0070] Specifically, step 1 includes:

[0071] In wireless sensing, a receiving device obtains original complex channel feature data by receiving a wireless signal sent by a transmitting device. The original complex channel feature data is used to characterize human gestures, including static components and dynamic components.

[0072] Performing data dimension reduction and denoising on the original complex channel characteristic data through principal component analysis to obtain complex channel characteristic data;

[0073] The complex channel characteristic data is divided into windows by short-time Fourier transform to obtain a plurality of complex channel characteristic data blocks, wherein one window corresponds to one complex channel characteristic data block;

[0074] Fourier transform is performed on the complex channel characteristic data blocks in each window to obtain real Doppler spectrum data.

[0075] It should be noted that the receiving device can be a WiFi router, network card, etc., and the sending device can be a mobile phone, laptop, etc.

[0076] Since wireless signals propagate through WiFi channels, the state of the channel will change due to objects or human activities in the environment, so the complex channel feature data can map the corresponding actions.

[0077] Specifically, in an M×N transceiver system, the wireless signal is described as y=h×x+noise, where x represents the transmitted signal, y represents the received signal, g represents the gain matrix, and noise represents the noise. The signal between a pair of transceivers in the communication system is divided into static components h along its propagation path. s (f) and the dynamic component h d (f,t), the gain matrix is ​​composed of these two components:

[0078] h(f,t)=h s (f)+h d (f,t)=h s (f)+A d e -2jπfd(t) / c

[0079] Where t represents time, f represents frequency, and the dynamic component is affected by the change in the propagation path length. In this transceiver system, the human body is equivalent to a reflector. The relative motion between the transmitter and the reflector will cause the Doppler effect, which will cause the signal frequency of the observation channel to shift. The essence of the Doppler frequency shift is the change in the propagation path length. Therefore, the above formula can be converted to:

[0080]

[0081] In summary, the original complex channel characteristic data y can be obtained as:

[0082]

[0083] Among them, h s (f) represents the static component, h d (f,t) represents the dynamic component, h s (g)+h d (f, t) = h(f, t), h(f, t) represents the gain matrix, a(f, t) represents the amplitude, represents the Doppler frequency shift corresponding to the dynamic path d, D d represents the dynamic path d, x represents the transmitted signal, and noise represents the noise.

[0084] In order to further process the collected original channel feature data, the embodiment of the present invention performs data dimension reduction and denoising on the original channel feature data through principal component analysis, and then applies short-time Fourier transform to the above formula to obtain corresponding Doppler spectrum data, so as to establish corresponding features with human gestures, thereby realizing the recognition of human gestures.

[0085] Since the short-time Fourier transform is particularly suitable for analyzing non-stationary signals, that is, signals whose frequency components change over time, such as music, speech, WiFi signals, etc., the Fourier transform can only provide the frequency signal of the signal in a global range and cannot describe the frequency changes of the signal at different times. Therefore, the Fourier transform is suitable for analyzing stationary signals, but cannot effectively process non-stationary signals.

[0086] In order to solve this problem, the short-time Fourier transform obtains the characteristics of frequency change over time by performing local Fourier transform on the signal at different times. Specifically, the short-time Fourier transform divides the data into multiple data blocks according to the window. One window corresponds to a time period, and one window corresponds to a data block. Fourier transform is performed in each window to analyze the frequency components of the time period. This method enables the short-time Fourier transform to observe the change of the signal over time and obtain the frequency information of the signal, thereby providing time-frequency representation. The principle is shown in the formula:

[0087]

[0088] Where τ represents the time variable used for integration;

[0089] However, when the short-time Fourier transform is segmented, there will be a window truncation response, which will lead to spectrum pollution. At this time, the spectral lines in the signal spectrum will affect each other, disrupt the measurement results, and cause performance loss. Now we model and analyze this process to explore its actual principle. Let the ideal spectrum data of signal x be s, and the convolution matrix of the window function in the frequency domain be C. Under the actual channel conditions, the measured real Doppler spectrum data The relationship between and the ideal spectrum data can be expressed as:

[0090]

[0091] Where N represents noise.

[0092] It is precisely because the convolution matrix C of the window function in the frequency domain acts on the ideal spectrum s that the spectrum pollution phenomenon occurs in the WiFi perception process.

[0093] Specifically, step 2 includes:

[0094] Input the real Doppler spectrum data into the constructed spectrum cleaning network;

[0095] In the spectrum cleaning network, a uniformly distributed carrier signal is generated within the human activity frequency range according to the sampling rate of the real Doppler spectrum data, and the frequency, amplitude and phase of the carrier signal are determined by statistical analysis of the real Doppler spectrum data;

[0096] Generate a window function according to the window type and window size selected in the short-time Fourier transform process, and perform convolution and padding processing on the window function and the carrier signal to obtain a windowed signal matching the sampling rate;

[0097] Perform fast Fourier transform on the windowed signal to obtain the convolution matrix of the window function in the frequency domain, and truncate the convolution matrix to obtain the ideal spectrum data sample;

[0098] After convolving the ideal spectrum data sample with the convolution matrix, additive Gaussian white noise is added to generate polluted spectrum data samples;

[0099] The spectrum cleaning network is trained using the polluted spectrum data samples to obtain a trained spectrum cleaning network.

[0100] The basic idea of ​​the spectrum cleaning network constructed in the embodiment of the present invention is to restore ideal spectrum data from contaminated spectrum data through training. This method requires a large amount of ideal spectrum data for effective training, but it is extremely challenging to obtain ideal spectrum data in a real environment. Therefore, ideal spectrum data needs to be generated before training.

[0101] In the embodiment of the present invention, generating ideal spectrum data includes two parts: generating a convolution matrix of a window function and generating ideal spectrum data, as follows:

[0102] 1. Generate a uniform carrier signal: According to the sampling rate fs of the real Doppler spectrum data, at the human activity frequency f bin Generate a uniformly distributed carrier signal in the range [-60,60] Hz;

[0103] According to the window function Statistical analysis of the real Doppler spectrum data is performed to determine its amplitude a i The range is [0,1], and the phase φ i The range is [0,2π];

[0104] 2. Windowing: According to the window type w selected in the short-time Fourier transform process t and the window size w s , generate a window function, and perform convolution and padding processing on the window function and the carrier signal so that the dimension matches the sampling rate, and obtain a windowed signal that matches the sampling rate;

[0105] 3. Perform fast Fourier transform (FFT) on the windowed signal to obtain the convolution matrix C of the window function in the frequency domain:

[0106] C = FFT(carrier(t)·W(t))

[0107] 4. The convolution matrix C is truncated to obtain the ideal spectrum data sample, and the amplitude of the ideal spectrum data sample is normalized to [0,1].

[0108] Specifically, the ideal spectrum data sample is convolved with the convolution matrix and then the additive Gaussian white noise is added to generate the polluted spectrum data sample, including:

[0109] For each ideal spectrum data sample S i , randomly select a certain number of frequencies in the range of [-60,60] Hz, and generate subcarrier signals at these frequencies, and the amplitude and phase of each subcarrier signal are randomly assigned;

[0110] Based on the generated ideal spectrum data sample, the ideal spectrum data sample is convolved with the convolution matrix to obtain the polluted spectrum data sample S pHowever, in the real scene of WiFi sensing process, the channel conditions are relatively complex and usually contain a lot of noise. In order to simulate the spectrum pollution under the real channel conditions, the embodiment of the present invention randomly introduces additive white Gaussian noise (AWGN) into the polluted spectrum data sample. This noise is often used to simulate the noise environment in the communication system. It is directly added to the signal and will not change the structure of the signal, but only affect its amplitude. Moreover, its power spectrum density at all frequencies is uniform, that is, the energy of each frequency component is equal. Therefore, the obtained polluted spectrum data sample S p For S p =C×S i +N.

[0111] Most preferably, Figure 2 As shown, the extraction module is used to extract time-frequency features, and extracts higher-level features by increasing the number of convolution channels layer by layer, including a first two-dimensional convolution layer, a second two-dimensional convolution layer, and a third two-dimensional convolution layer connected in sequence;

[0112] The transformation module is used for nonlinear combination and feature transformation, so as to output a cleaned spectrum, including a first complex fully connected layer, a second complex fully connected layer, a third complex fully connected layer, and a fourth complex fully connected layer connected in sequence;

[0113] The input end of the first two-dimensional convolutional layer is the input end of the spectrum cleaning network, which is used to receive the real Doppler spectrum data;

[0114] The output end of the third two-dimensional convolutional layer is connected to the input end of the first complex fully connected layer;

[0115] The output end of the fourth complex fully connected layer is the output end of the spectrum cleaning network, which is used to output the optimal model parameters in the training stage and to output the enhanced spectrum data in the application stage.

[0116] In the embodiment of the present invention, the spectrum cleaning network as a whole uses the tanh activation function, and its formula is as follows:

[0117]

[0118] This formula is more suitable for processing negative and continuous feature data, especially spectral data.

[0119] In the embodiment of the present invention, the spectrum cleaning network aims to restore the polluted spectrum data to the ideal spectrum data, which can be attributed to the regression problem. Therefore, in the loss function design of the network, the mean square error is used as the measurement indicator. In regression analysis, the mean square error is often used to measure the fitting effect of the model. Its square penalty makes it have higher requirements on the accuracy of the prediction. Therefore, the overall loss is defined as the sum of the mean square errors of the real part and the imaginary part, as shown below:

[0120] L=L r +L i

[0121] in represents the mean square error between the real part of the output spectrum and the ideal spectrum, It represents the mean square error between the imaginary part of the output spectrum and the ideal spectrum.

[0122] In the embodiment of the present invention, the spectrum cleaning network is trained according to the overall loss, the number of training rounds is set to 3000, the Adam optimizer is used, the initial learning rate is set to 0.001, and when the module loss does not decrease after 10 training rounds, the learning rate is halved. During the training process, the dropout regularization technology is used to prevent overfitting of the training process and improve the generalization ability of the model.

[0123] Compared with convolutional neural networks (CNN), deep complex networks (DCN) have richer representation capabilities and are easier to optimize. They can not only extract the amplitude characteristics of complex data, but also effectively retain and utilize phase information, thereby providing more accurate and comprehensive feature expressions for applications in fields such as wireless sensing and signal processing. Therefore, an embodiment of the present invention adopts a deep complex network to recognize gesture actions.

[0124] Most preferably, Figure 2 As shown, the feature extraction module in the deep complex network includes:

[0125] A first complex convolution layer and a second complex convolution layer for performing complex convolution operations on input data to extract features, a first maximum pooling layer and a second maximum pooling layer for downsampling and compressing the extracted features;

[0126] The input end of the first complex convolution layer is the input end of the deep complex network and is used to be connected to the output end of the fourth complex fully connected layer. The output end of the first complex convolution layer is connected to the input end of the first maximum pooling layer, the output end of the second maximum pooling layer is connected to the input end of the second complex convolution layer, the output end of the second complex convolution layer is connected to the input end of the second maximum pooling layer, and the output end of the second maximum pooling layer is connected to the input end of the feature compression module for extracting gesture-related features.

[0127] Unlike traditional real convolution, complex convolution can capture patterns in complex spectral data more accurately and retain the phase information in the data. This feature enables DCN to significantly improve the feature extraction effect when processing complex data containing spectral information such as electromagnetic wave signals and acoustic signals.

[0128] After each complex convolution operation, the network downsamples and compresses the extracted features through two layers of maximum pooling layers.

[0129] The embodiment of the present invention defines the expression of complex convolution operation as:

[0130] κ*z=(κ1*a-κ2*b)+i(κ2*a+κ1*b)

[0131] Among them, κ represents the complex convolution kernel, κ=κ1+iκ2, z represents the complex eigenvector, z=a+ib, a represents the real part of the complex number, b represents the imaginary part of the complex number, and i represents the imaginary unit.

[0132] If we use matrix notation to represent the real and imaginary parts of the convolution operation, we have:

[0133]

[0134] In an embodiment of the present invention, the maximum pooling layer downsamples the features with the largest amplitude, thereby reducing the dimension of the feature map while retaining the most important complex-valued features. In this way, the network reduces the computational complexity while retaining information that is critical to subsequent classification tasks.

[0135] Most preferably, Figure 2 As shown, the feature compression module in the deep complex network includes the following connected in sequence:

[0136] A fifth complex fully connected layer and a sixth complex fully connected layer for integrating the extracted gesture-related features, a first real fully connected layer and a second real fully connected layer for mapping the integrated features to the real domain, and a Softmax activation function layer for generating a probability distribution;

[0137] The input end of the fifth complex fully connected layer is connected to the output end of the second maximum pooling layer, and the output end of the Softmax activation function layer is the output end of the deep complex network.

[0138] However, in order to achieve the final classification task, the model needs to map these abstract complex features to the real domain. This step is achieved through two layers of real fully connected layers with Relu activation function. In this process, the model calculates the L2 norm of the complex features, that is, Here, z is an eigenvector, which is usually used to obtain the magnitude value of a complex number.

[0139] In the embodiment of the present invention, the Softmax activation function layer converts an unnormalized vector into a probability distribution, the value of each element of which is between [0, 1], and the sum of all elements is 1; it is essentially a generalization of the Logistic regression model to the multi-classification problem, that is, multivariate logistic regression, and its formula is as follows:

[0140]

[0141] In an embodiment of the present invention, the loss function of the deep complex network adopts the cross-entropy loss function (Cross-Entropy Loss). Cross-Entropy Loss is a commonly used loss function for classification problems. It can measure the difference between the probability distribution predicted by the model and the actual label distribution. If the predicted probability is significantly different from the true label, the loss value will increase significantly, thereby prompting the model to learn better features. It not only uses binary classification tasks, but is also particularly suitable for multi-class classification tasks.

[0142] Specifically, for the multi-classification task of gesture recognition, the cross entropy loss function is calculated as:

[0143]

[0144] Where K is the number of categories, y ij Indicates that if the true category of sample i is equal to j, it takes 1, otherwise it takes 0. It represents the predicted probability that sample i belongs to j.

[0145] Specifically, step 5 includes:

[0146] The complex channel feature data related to the target human gesture is input into the trained spectrum cleaning network, and the time-frequency features are extracted by the extraction module in the spectrum cleaning network. Then, the data is input into the transformation module for nonlinear feature combination and transformation processing, and regression is performed to clean the spectrum pollution and obtain enhanced high-resolution spectrum data.

[0147] The enhanced high-resolution spectrum data is input into the gesture recognition network, and the feature extraction module is used to extract features to obtain gesture-related features. The data is then input into the feature compression module for feature integration and then into the Softmax activation function layer to generate the probability distribution of each category. The gesture recognition results are obtained based on the probability distribution of each category. The gesture recognition results include but are not limited to ok, bye, wave, praise, and stop.

[0148] The embodiment of the present invention collects information about the impact of human gestures on the WiFi channel state based on wireless perception, and obtains real Doppler spectrum data; inputs the real Doppler spectrum data into the constructed spectrum cleaning network for processing to generate contaminated spectrum data samples, and uses the contaminated spectrum data samples to train the spectrum cleaning network to obtain a trained spectrum cleaning network; inputs the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data; inputs the enhanced spectrum data into the constructed deep complex network, trains the deep complex network, and obtains a gesture recognition network; inputs complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhanced recognition to obtain a gesture recognition result; compared with the prior art, the embodiment of the present invention restores the contaminated spectrum data to high-resolution spectrum data through the spectrum cleaning network, minimizes spectrum contamination, thereby improving the accuracy of wireless perception, and then recognizes the high-resolution spectrum data through the deep complex network, retains the structural characteristics of the data, thereby improving the recognition accuracy.

[0149] The present invention also provides a gesture recognition device based on signal spectrum enhancement, such as Figure 3 As shown, the gesture recognition device 100 includes:

[0150] An acquisition module 101 is used to collect information about the impact of human gestures on the WiFi channel state based on wireless sensing, and obtain real Doppler spectrum data;

[0151] The processing module 102 is used to input the real Doppler spectrum data into the constructed spectrum cleaning network for processing, generate contaminated spectrum data samples, and train the spectrum cleaning network using the contaminated spectrum data samples to obtain a trained spectrum cleaning network;

[0152] An enhancement module 103 is used to input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data;

[0153] A training module 104, configured to input the enhanced spectrum data into the constructed deep complex network, train the deep complex network, and obtain a gesture recognition network;

[0154] The recognition module 105 is used to sequentially input the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network for enhanced recognition to obtain a gesture recognition result;

[0155] The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process, the extraction module is used to extract features from the data, and the transformation module is used to transform the features output from the previous level.

[0156] The deep complex network includes a feature extraction module, a feature compression module, and a classification module which are connected in sequence. The input end of the feature extraction module is connected to the output end of the transformation module. The feature compression module is used to calculate the probability of various gestures, and the classification module outputs the gesture recognition result.

[0157] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0158] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0159] The present invention also provides a terminal device, such as Figure 4 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, the processor D100 implements the steps of the above-mentioned method for constructing the uncoupled blasting load model when executing the computer program D102. Alternatively, the processor D100 implements the functions of each module / unit in the above-mentioned device embodiments when executing the computer program D102.

[0160] The terminal device D10 may be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will appreciate that Figure 4This is only an example of the terminal device D10 and does not constitute a limitation on the terminal device D10. The terminal device D10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0161] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0162] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0163] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0164] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0165] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a gesture recognition method based on signal spectrum enhancement is implemented.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0167] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A gesture recognition method based on signal spectrum enhancement, characterized in that: include: Step 1: Based on wireless sensing, collect information about the impact of human gestures on the WiFi channel state and obtain real Doppler spectrum data; Step 2: input the real Doppler spectrum data into the constructed spectrum cleaning network for processing to generate contaminated spectrum data samples, and use the contaminated spectrum data samples to train the spectrum cleaning network to obtain a trained spectrum cleaning network; Step 3, inputting the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data; Step 4, inputting the enhanced spectrum data into the constructed deep complex network, training the deep complex network, and obtaining a gesture recognition network; Step 5, inputting the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhanced recognition to obtain a gesture recognition result; The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process, the extraction module is used to extract features from the data, and the transformation module is used to transform the features output from the previous level; The deep complex network includes a feature extraction module, a feature compression module, and a classification module which are connected in sequence. The input end of the feature extraction module is connected to the output end of the transformation module. The feature compression module is used to calculate the probability of each type of gesture. The classification module outputs the gesture recognition result.

2. The gesture recognition method based on signal spectrum enhancement according to claim 1, characterized in that: The step 1 comprises: In wireless sensing, a receiving device obtains original complex channel feature data by receiving a wireless signal sent by a sending device, and the original complex channel feature data is used to characterize a human gesture action; Performing data dimension reduction and denoising on the original complex channel characteristic data through principal component analysis to obtain complex channel characteristic data; Segmenting the complex channel characteristic data by window by short-time Fourier transform to obtain a plurality of complex channel characteristic data blocks, wherein one window corresponds to one complex channel characteristic data block; Fourier transform is performed on the complex channel characteristic data blocks in each window to obtain real Doppler spectrum data.

3. The gesture recognition method based on signal spectrum enhancement according to claim 2, characterized in that: The original complex channel characteristic data y is: Among them, h s (f) represents the static component, h d (f,t) represents the dynamic component, h s (f)+h d (f, t) = h(f, t), h(f, t) represents the gain matrix, a(f, t) represents the amplitude, represents the Doppler shift of dynamic path d, D d represents the dynamic path d, x represents the transmitted signal, and noise represents the noise.

4. The gesture recognition method based on signal spectrum enhancement according to claim 3 is characterized in that: The step 2 comprises: Inputting the real Doppler spectrum data into the constructed spectrum cleaning network; In the spectrum cleaning network, a uniformly distributed carrier signal is generated within the human activity frequency range according to the sampling rate of the real Doppler spectrum data, and the frequency, amplitude and phase of the carrier signal are determined by statistical analysis of the real Doppler spectrum data; Generate a window function according to the window type and window size selected in the short-time Fourier transform process, and perform convolution and padding processing on the window function and the carrier signal to obtain a windowed signal matching the sampling rate; Performing a fast Fourier transform on the windowed signal to obtain a convolution matrix of the window function in the frequency domain, and truncating the convolution matrix to obtain an ideal spectrum data sample; Convolving the ideal spectrum data sample with the convolution matrix, and then adding additive Gaussian white noise to generate a polluted spectrum data sample; The spectrum cleaning network is trained using the contaminated spectrum data samples to obtain a trained spectrum cleaning network.

5. The gesture recognition method based on signal spectrum enhancement according to claim 4, characterized in that: The extraction module includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a third two-dimensional convolutional layer connected in sequence; The transformation module includes a first complex fully connected layer, a second complex fully connected layer, a third complex fully connected layer, and a fourth complex fully connected layer connected in sequence; The input end of the first two-dimensional convolutional layer is the input end of the spectrum cleaning network, which is used to receive the real Doppler spectrum data; An output end of the third two-dimensional convolutional layer is connected to an input end of the first complex fully connected layer; The output end of the fourth complex fully connected layer is the output end of the spectrum cleaning network, which is used to output the optimal model parameters in the training stage and to output the enhanced spectrum data in the application stage.

6. The gesture recognition method based on signal spectrum enhancement according to claim 5, characterized in that: The feature extraction module comprises: A first complex convolution layer and a second complex convolution layer for performing complex convolution operations on input data to extract features, a first maximum pooling layer and a second maximum pooling layer for downsampling and compressing the extracted features; The input end of the first complex convolution layer is the input end of the deep complex network and is used to be connected to the output end of the fourth complex fully connected layer, the output end of the first complex convolution layer is connected to the input end of the first maximum pooling layer, the output end of the second maximum pooling layer is connected to the input end of the second complex convolution layer, the output end of the second complex convolution layer is connected to the input end of the second maximum pooling layer, and the output end of the second maximum pooling layer is connected to the input end of the feature compression module for extracting gesture-related features.

7. The gesture recognition method based on signal spectrum enhancement according to claim 6, characterized in that: The feature compression module includes: A fifth complex fully connected layer and a sixth complex fully connected layer for integrating the extracted gesture-related features, a first real fully connected layer and a second real fully connected layer for mapping the integrated features to the real domain, and a Softmax activation function layer for generating a probability distribution; The input end of the fifth complex fully connected layer is connected to the output end of the second maximum pooling layer, and the output end of the Softmax activation function layer is the output end of the deep complex network.

8. A gesture recognition device based on signal spectrum enhancement, characterized in that: include: An acquisition module is used to collect information about the impact of human gestures on the WiFi channel state based on wireless sensing, and obtain real Doppler spectrum data; A processing module, used for inputting the real Doppler spectrum data into the constructed spectrum cleaning network for processing to generate contaminated spectrum data samples, and using the contaminated spectrum data samples to train the spectrum cleaning network to obtain a trained spectrum cleaning network; An enhancement module, used for inputting the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data; A training module, used for inputting the enhanced spectrum data into a constructed deep complex network, training the deep complex network, and obtaining a gesture recognition network; A recognition module, used for sequentially inputting the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network for enhanced recognition to obtain a gesture recognition result; The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process, the extraction module is used to extract features from the data, and the transformation module is used to transform the features output from the previous level; The deep complex network includes a feature extraction module, a feature compression module, and a classification module which are connected in sequence. The input end of the feature extraction module is connected to the output end of the transformation module. The feature compression module is used to calculate the probability of each type of gesture. The classification module outputs the gesture recognition result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the gesture recognition method based on signal spectrum enhancement according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the gesture recognition method based on signal spectrum enhancement as claimed in any one of claims 1 to 7 is implemented.

Images