A gesture recognition method based on signal spectrum enhancement and related device

Abstract

The application provides a gesture recognition method based on signal spectrum enhancement and related equipment, information caused by human gesture action on WiFi channel state is collected based on wireless sensing, real Doppler spectrum data is obtained to input a spectrum cleaning network for processing, contaminated spectrum data samples are generated to train the spectrum cleaning network, and the trained spectrum cleaning network is obtained; the real Doppler spectrum data is input into the trained spectrum cleaning network for enhancement, and enhanced spectrum data is obtained; the enhanced spectrum data is input into a deep complex network, the deep complex network is trained, and a gesture recognition network is obtained; complex channel feature data related to a target human gesture is input into the trained spectrum cleaning network and the gesture recognition network in sequence for enhancement and recognition, and a gesture recognition result is obtained, so that the recognition accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and Internet of Things, and particularly relates to a gesture recognition method based on signal spectrum enhancement and related equipment. BACKGROUND

[0002] With the wide deployment of WiFi infrastructure, WiFi sensing technology is gradually attracting extensive attention from both industry and academia. Compared with traditional human activity sensing systems based on dedicated sensors, wearable devices or cameras, WiFi sensing has the advantages of seamless integration with existing communication devices, no need for additional hardware deployment, and non-contact and privacy-friendly characteristics. Benefiting from the rich complex channel feature data in WiFi sensing, various Internet of Things sensing applications based on WiFi are emerging, including vital sign monitoring and indoor positioning. Although these applications show broad prospects, WiFi sensing technology still faces a major challenge in in-depth research: the limited bandwidth of current WiFi cannot provide sufficient spatial resolution, which limits the performance of WiFi sensing systems in the above applications.

[0003] Specifically, the traditional WiFi sensing system usually adopts a short-time Fourier transform (STFT) to generate a frequency spectrum as a sensing primitive. In this process, signal processing usually needs to perform a window function operation (such as a rectangular window, a Hanning window, etc.). However, the limited length of the window function will cause waveform spreading in the frequency domain, which will cause the signal energy to "scatter" from the actual frequency component to the nearby frequency component, produce other frequency components, and cause spectrum pollution. The traditional method reduces the influence of spectrum pollution by selecting a suitable window function. The window function reduces the edge effect by applying a weight to the time-domain signal, thereby alleviating the pollution problem. Although these window functions can reduce pollution to some extent, they cannot completely eliminate it, especially in high-noise or complex scenarios such as WiFi sensing, the effect is limited. At the same time, due to the limited bandwidth of WiFi, the signal energy is limited in a small frequency band, which makes the pollution problem more prominent, thereby blurring important frequency features and affecting the overall accuracy of the system.

[0004] In addition, the traditional WiFi sensing classification network design usually only focuses on the spectrum amplitude and ignores the phase information in the spectrum, which will cause feature loss and reduce the classification performance of the network. SUMMARY

[0005] The present application provides a gesture recognition method based on signal spectrum enhancement and related equipment, which aims to reduce spectrum pollution and improve recognition accuracy.

[0006] In order to achieve the above object, the application provides a gesture recognition method based on signal spectrum enhancement, comprising:

[0007] Step 1, based on wireless sensing, collecting information about the influence of human gesture action on WiFi channel state, obtaining real Doppler spectrum data;

[0008] Step 2, input the real Doppler spectrum data into the constructed spectrum cleaning network for processing, generate pollution spectrum data samples, and use the pollution spectrum data samples to train the spectrum cleaning network, obtain the trained spectrum cleaning network;

[0009] Step 3, input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement, obtain the enhanced spectrum data;

[0010] Step 4, input the enhanced spectrum data into the constructed deep complex network, train the deep complex network, and obtain the gesture recognition network;

[0011] Step 5, input the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and gesture recognition network in sequence for enhancement and recognition, and obtain the gesture recognition result;

[0012] The spectrum cleaning network comprises a data construction module, an extraction module and a transformation module connected in sequence, the data construction module is only used for generating training data in the training process, the extraction module is used for feature extraction, and the transformation module is used for transforming the features output by the previous stage;

[0013] The deep complex network comprises a feature extraction module, a feature compression module and a classification module connected in sequence, the input end of the feature extraction module is connected with the output end of the transformation module, the feature compression module is used for calculating the probability of each type of gesture, and the classification module outputs the gesture recognition result.

[0014] Further, step 1 comprises:

[0015] In wireless sensing, the receiving device receives the wireless signal sent by the sending device to obtain the original complex channel feature data, and the original complex channel feature data is used to represent human gesture action, including static component and dynamic component;

[0016] The principal component analysis is used to perform data dimension reduction and denoising on the original complex channel feature data to obtain the complex channel feature data;

[0017] The complex channel feature data is segmented by short-time Fourier transform according to windows to obtain a plurality of complex channel feature data blocks, and one window corresponds to one complex channel feature data block;

[0018] The complex channel feature data block is subjected to Fourier transform to obtain real Doppler spectrum data.

[0019] Further, the original complex channel feature data y is:

[0020]

[0021] wherein h s (f) represents a static component, h d (f,t) represents a dynamic component, h s (f) + h d (f,t) = h(f,t), h(f,t) represents a gain matrix, and a(f,t) represents an amplitude, D d represents a dynamic path d, x represents a transmitted signal, and noise represents noise.

[0022] Further, the step 2 comprises:

[0023] The real Doppler spectrum data is input into a constructed spectrum cleaning network.

[0024] In the spectrum cleaning network, according to a sampling rate of the real Doppler spectrum data, carrier signals uniformly distributed in a human activity frequency range are generated, and the frequency, amplitude and phase of the carrier signals are determined through statistical analysis of the real Doppler spectrum data.

[0025] According to the window type and window size selected in the short-time Fourier transform process, a window function is generated, and the window function is convolved and padded with the carrier signal to obtain a windowed signal matched with the sampling rate.

[0026] The windowed signal is subjected to fast Fourier transform to obtain a convolution matrix of the window function in the frequency domain, and the convolution matrix is truncated to obtain ideal spectrum data samples.

[0027] After the ideal spectrum data samples are convolved with the convolution matrix, additive white Gaussian noise is added to generate contaminated spectrum data samples.

[0028] The spectrum cleaning network is trained using the contaminated spectrum data samples to obtain a trained spectrum cleaning network.

[0029] Further, the extraction module comprises a first two-dimensional convolution layer, a second two-dimensional convolution layer and a third two-dimensional convolution layer connected in sequence.

[0030] The transformation module comprises a first complex full connection layer, a second complex full connection layer, a third complex full connection layer and a fourth complex full connection layer connected in sequence.

[0031] The input end of the first two-dimensional convolutional layer is the input end of the spectrum cleaning network, and is used for receiving real Doppler spectrum data;

[0032] The output end of the third two-dimensional convolutional layer is connected with the input end of the first complex full connection layer;

[0033] The output end of the fourth complex full connection layer is the output end of the spectrum cleaning network, and is used for outputting optimal model parameters in the training stage and outputting enhanced spectrum data in the application stage.

[0034] Further, the feature extraction module comprises:

[0035] The first complex convolutional layer, the second complex convolutional layer, the first maximum pooling layer and the second maximum pooling layer are used for performing complex convolution operation on the input data to extract features, and performing down-sampling and compression processing on the extracted features;

[0036] The input end of the first complex convolutional layer is the input end of the deep complex network and is connected with the output end of the fourth complex full connection layer, the output end of the first complex convolutional layer is connected with the input end of the first maximum pooling layer, the output end of the second maximum pooling layer is connected with the input end of the second complex convolutional layer, the output end of the second complex convolutional layer is connected with the input end of the second maximum pooling layer, the output end of the second maximum pooling layer is connected with the input end of the feature compression module, and the feature compression module is used for extracting gesture-related features.

[0037] Further, the feature compression module comprises, which are connected in sequence:

[0038] The fifth complex full connection layer and the sixth complex full connection layer are used for integrating the extracted gesture-related features, the first real full connection layer and the second real full connection layer are used for mapping the integrated features to the real number field, and the Softmax activation function layer is used for generating a probability distribution.

[0039] The input end of the fifth complex full connection layer is connected with the output end of the second maximum pooling layer, and the output end of the Softmax activation function layer is the output end of the deep complex network.

[0040] The application further provides a gesture recognition device based on signal spectrum enhancement, comprising:

[0041] The acquisition module is used for acquiring real Doppler spectrum data based on information that human gesture actions affect WiFi channel states collected through wireless sensing;

[0042] The processing module is used for inputting the real Doppler spectrum data into the constructed spectrum cleaning network for processing, generating a polluted spectrum data sample, and training the spectrum cleaning network by using the polluted spectrum data sample to obtain a trained spectrum cleaning network.

[0043] an enhancement module, configured to input real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data;

[0044] a training module, configured to input the enhanced spectrum data into the constructed deep complex network to train the deep complex network to obtain a gesture recognition network;

[0045] a recognition module, configured to input complex channel feature data related to a target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhancement and recognition to obtain a gesture recognition result;

[0046] The spectrum cleaning network comprises a data construction module, an extraction module and a transformation module connected in sequence, the data construction module is only used for generating training data in a training process, the extraction module is configured to perform feature extraction on data, and the transformation module is configured to perform transformation on features output by a previous stage.

[0047] The deep complex network comprises a feature extraction module, a feature compression module and a classification module connected in sequence, an input end of the feature extraction module is connected with an output end of the transformation module, the feature compression module is configured to calculate probabilities of various gestures, and the classification module outputs a gesture recognition result.

[0048] The application further provides a terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the gesture recognition method based on signal spectrum enhancement when executing the computer program.

[0049] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the gesture recognition method based on signal spectrum enhancement.

[0050] The above-mentioned scheme of the application has the following advantages:

[0051] The application is based on wireless sensing to collect information that human gesture actions affect WiFi channel state, obtain real Doppler spectrum data, input the real Doppler spectrum data into a constructed spectrum cleaning network for processing to generate polluted spectrum data samples, train the spectrum cleaning network using the polluted spectrum data samples to obtain a trained spectrum cleaning network, input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data, input the enhanced spectrum data into a constructed deep complex network, train the deep complex network to obtain a gesture recognition network, input complex channel feature data related to a target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhancement and recognition to obtain a gesture recognition result; compared with the prior art, the application restores polluted spectrum data to high-resolution spectrum data through the spectrum cleaning network, maximally reduces spectrum pollution, thereby improving the accuracy of wireless sensing, then identifies the high-resolution spectrum data through the deep complex network, retains the structural features of the data, thereby improving the recognition accuracy.

[0052] Other benefits of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The flowchart of the embodiment of the present application is shown in the figure;

[0054] Figure 2 The structural diagram of the spectrum cleaning network and the deep complex network in the embodiment of the present application is shown in the figure;

[0055] Figure 3 The structural diagram of the gesture recognition device in the embodiment of the present application is shown in the figure;

[0056] Figure 4 The structural diagram of the terminal device in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] To make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the figures and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0058] In the description of the present application, it should be noted that the terms "first", "second", "third" are only for description purpose, and cannot be understood as indicating or implying relative importance.

[0059] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "linking" should be understood in a broad sense, for example, it can be a locking connection, or a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or a communication inside two elements. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0060] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as there is no conflict.

[0061] The present application aims at the existing problems, and provides a gesture recognition method based on signal spectrum enhancement and related equipment.

[0062] As Figure 1 shown, the embodiment of the present application provides a gesture recognition method based on signal spectrum enhancement, comprising:

[0063] Step 1, based on wireless sensing, collecting information about the influence of human gesture action on WiFi channel state, obtaining real Doppler spectrum data;

[0064] Step 2, inputting the real Doppler spectrum data into the constructed spectrum cleaning network for processing, generating pollution spectrum data samples, and training the spectrum cleaning network using the pollution spectrum data samples, to obtain the trained spectrum cleaning network;

[0065] Step 3, inputting the real Doppler spectrum data into the trained spectrum cleaning network for enhancement, to obtain enhanced spectrum data;

[0066] Step 4, inputting the enhanced spectrum data into the constructed deep complex network, training the deep complex network, to obtain a gesture recognition network;

[0067] Step 5, inputting the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and gesture recognition network in sequence for enhancement and recognition, to obtain a gesture recognition result;

[0068] The spectrum cleaning network comprises a data construction module, an extraction module and a transformation module connected in sequence, the data construction module is only used for generating training data in the training process, the extraction module is used for feature extraction of data, and the transformation module is used for transformation of the features output by the previous stage;

[0069] The deep complex network comprises a feature extraction module, a feature compression module and a classification module connected in sequence, the input end of the feature extraction module is connected with the output end of the transformation module, the feature compression module is used for calculating the probability of each gesture, and the classification module outputs a gesture recognition result.

[0070] Specifically, step 1 comprises:

[0071] In wireless sensing, a receiving device receives a wireless signal sent by a sending device to obtain original complex channel feature data, and the original complex channel feature data is used to represent a human gesture action and comprises a static component and a dynamic component.

[0072] The original complex channel feature data is subjected to data dimension reduction and denoising through principal component analysis to obtain complex channel feature data.

[0073] The complex channel feature data is segmented through short-time Fourier transform by window to obtain a plurality of complex channel feature data blocks, and one window corresponds to one complex channel feature data block.

[0074] The complex channel feature data blocks in each window are subjected to Fourier transform to obtain real Doppler spectrum data.

[0075] It should be noted that the receiving device can be a WiFi router, a network card or the like, and the sending device can be a mobile phone, a notebook computer or the like.

[0076] Since the wireless signal is transmitted through a WiFi channel, the state of the channel will change due to the activities of objects or human bodies in the environment, and therefore the complex channel feature data can map the corresponding action.

[0077] Specifically, in an MxN transceiver system, the wireless signal is described as y=h x+noise, wherein x represents a transmitted signal, y represents a received signal, g represents a gain matrix, and noise represents noise; the signal between a pair of transceiver devices in a communication system is divided into a static component h s (f) and a dynamic component h d (f,t) along its propagation path, and the gain matrix is composed of the two components:

[0078] h(f,t)=h s (f)+h d (f,t)=h s (f)+A d e -2jπfd(t) / c

[0079] Wherein, t represents time, f represents frequency, dynamic component is affected by the change of propagation path length, in the transceiver system, the human body is equivalent to a reflector, the relative movement between the transmitter and the reflector will cause Doppler effect, which will cause the frequency of the observed channel signal to shift, and the essence of Doppler shift is the change of the propagation path length, so the above formula can be converted to:

[0080]

[0081] Therefore, the original complex channel feature data y can be obtained as follows:

[0082]

[0083] Wherein, h s (f) represents a static component, h d (f,t) represents a dynamic component, h s (g)+h d (f,t)=h(f,t), h(f,t) represents a gain matrix, a(f,t) represents an amplitude, D d represents the dynamic path d, x represents the transmitted signal, and noise represents noise.

[0084] In order to further process the collected original channel feature data, the embodiment of the application processes the original channel feature data by principal component analysis for data dimension reduction and denoising, and then applies short-time Fourier transform to the above formula, so as to obtain corresponding Doppler spectrum data, so as to establish corresponding features of human body gestures, so as to realize recognition of human gesture actions.

[0085] Since the short-time Fourier transform is particularly suitable for analyzing non-stationary signals, i.e. signals whose frequency components change with time, such as music, speech, WiFi signals, etc., the Fourier transform can only provide the frequency signal of the signal in the global range, and cannot describe the frequency change of the signal at different times. Therefore, the Fourier transform is suitable for analyzing stationary signals, but cannot effectively process non-stationary signals.

[0086] In order to solve this problem, the short-time Fourier transform obtains the characteristics of the frequency changing with time by performing local Fourier transform on the signal at different times. Specifically, the short-time Fourier transform divides the data into multiple data blocks according to the window, one window corresponds to one time period, one window corresponds to one data block, and Fourier transform is performed in each window to analyze the frequency components of the time period. This way makes the short-time Fourier transform can observe the change of the signal in time and also get the frequency information of the signal, so as to provide time-frequency representation, and its principle is shown in the formula as follows:

[0087]

[0088] wherein τ represents a time variable for integration;

[0089] But the short-time Fourier transform will have a window truncation corresponding when segmentation, resulting in spectrum pollution, at this time the signal spectrum between each spectrum line mutual influence, disturb the measurement results, cause performance loss. Now the process is modeled and analyzed, and the actual principle is discussed. Let the ideal spectrum data of the signal x be s, and the convolution matrix of the window function in the frequency domain be C. Under the actual channel condition, the relationship between the real Doppler spectrum data measured and the ideal spectrum data can be expressed as:

[0090]

[0091] wherein N represents noise.

[0092] It is the convolution matrix C of the window function in the frequency domain that acts on the ideal spectrum s, resulting in the spectrum pollution phenomenon in the WiFi sensing process.

[0093] Specifically, step 2 comprises:

[0094] inputting the real Doppler spectrum data into the constructed spectrum cleaning network;

[0095] In the spectrum cleaning network, according to the sampling rate of the real Doppler spectrum data, uniformly distributed carrier signals are generated in the human activity frequency range, and the frequency, amplitude and phase of the carrier signal are determined by statistical analysis of the real Doppler spectrum data;

[0096] According to the window type and window size selected in the short-time Fourier transform process, a window function is generated, and the window function is convolved and filled with the carrier signal to obtain a windowed signal matched with the sampling rate;

[0097] The windowed signal is subjected to fast Fourier transform to obtain a convolution matrix of the window function in the frequency domain, and the convolution matrix is truncated to obtain ideal spectrum data samples;

[0098] After the ideal spectrum data samples are convolved with the convolution matrix, additive white Gaussian noise is added to generate polluted spectrum data samples;

[0099] The polluted spectrum data samples are used to train the spectrum cleaning network to obtain a trained spectrum cleaning network.

[0100] The basic idea of the spectrum cleaning network constructed by the embodiment of the present application is to recover ideal spectrum data from polluted spectrum data through training, and this method needs a large amount of ideal spectrum data to effectively train, but it is extremely challenging to obtain ideal spectrum data in a real environment, so ideal spectrum data needs to be generated before training.

[0101] In the embodiment of the present application, generating ideal spectrum data includes two parts: generating a convolution matrix of a window function and generating ideal spectrum data, as follows:

[0102] 1. Generating a uniform carrier signal: according to the sampling rate fs of the real Doppler spectrum data, a uniform distribution of carrier signals is generated in the human activity frequency range f bin [-60, 60] Hz;

[0103] According to the window function Statistical analysis of real Doppler spectrum data to determine its amplitude a i range [0, 1], phase φ i range [0, 2π];

[0104] 2. Windowing: according to the window type w t and the window size w s selected in the short-time Fourier transform process, a window function is generated, and the window function is convolved and filled with the carrier signal to match the sampling rate in dimension, to obtain a windowed signal matched with the sampling rate;

[0105] 3. Fast Fourier transform (FFT) is performed on the windowed signal to obtain the convolution matrix C of the window function in the frequency domain:

[0106] C = FFT(carrier(t)·W(t))

[0107] 4. The convolution matrix C is truncated to obtain an ideal spectrum data sample, and the amplitude of the ideal spectrum data sample is normalized to [0, 1].

[0108] Specifically, after the ideal spectrum data sample is convolved with the convolution matrix, an additive white Gaussian noise is added to generate a polluted spectrum data sample, including:

[0109] For each ideal spectrum data sample S i , a certain number of frequencies are randomly selected in the range [-60, 60] Hz, and subcarrier signals are generated at these frequencies, and the amplitude and phase of each subcarrier signal are randomly assigned;

[0110] Based on the generated ideal spectrum data sample, the ideal spectrum data sample is convolved with the convolution matrix to obtain a polluted spectrum data sample S pHowever, in real-world WiFi sensing scenarios, channel conditions are complex and often contain significant noise. To simulate spectrum pollution under real channel conditions, this invention introduces additive white Gaussian noise (AWGN) randomly into the polluted spectrum data samples. This noise is commonly used to simulate noise environments in communication systems. Adding it directly to the signal does not alter its structure, only affecting its amplitude. Furthermore, its power spectral density is uniform across all frequencies, meaning each frequency component has equal energy. Therefore, the resulting polluted spectrum data sample S... p For S p =C×S i +N.

[0111] The optimal choice is, such as Figure 2 As shown, the extraction module is used to extract time-frequency features. It extracts higher-level features by increasing the number of convolution channels layer by layer, including a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a third two-dimensional convolutional layer connected in sequence.

[0112] The transformation module is used for nonlinear combination and feature transformation to output the cleaned spectrum, which includes a first complex fully connected layer, a second complex fully connected layer, a third complex fully connected layer, and a fourth complex fully connected layer connected in sequence.

[0113] The input of the first two-dimensional convolutional layer is the input of the spectrum cleaning network, which is used to receive real Doppler spectrum data;

[0114] The output of the third two-dimensional convolutional layer is connected to the input of the first complex fully connected layer.

[0115] The output of the fourth complex fully connected layer is the output of the spectrum cleaning network. It is used to output the optimal model parameters during the training phase and to output the enhanced spectrum data during the application phase.

[0116] In this embodiment of the invention, the spectrum cleaning network as a whole uses the tanh activation function, the formula of which is as follows:

[0117]

[0118] This formula is well-suited for handling negative and continuous feature data, especially spectral data.

[0119] In this embodiment of the invention, the spectrum cleaning network aims to restore contaminated spectrum data to ideal spectrum data, which can be reduced to a regression problem. Therefore, in the design of the loss function of this network, mean squared error is used as a measurement index. In regression analysis, mean squared error is often used to measure the fitting effect of the model. Its squared penalty makes it require high accuracy in prediction. Therefore, the overall loss is defined as the sum of the mean squared errors of the real part and the imaginary part, as shown below:

[0120] L = L r + L i

[0121] wherein represents the mean square error between the real part of the output spectrum and the ideal spectrum, represents the mean square error between the imaginary part of the output spectrum and the ideal spectrum.

[0122] The embodiment of the application trains the spectrum cleaning network according to the overall loss, sets the training round number to 3000, uses the Adam optimizer, sets the initial learning rate to 0.001, reduces the learning rate by half when the loss of the module is not reduced for 10 training rounds, and applies the dropout regularization technique in the training process to prevent overfitting in the training process and improve the generalization ability of the model.

[0123] Since the deep complex network (DCN) has more abundant representation ability and is easier to optimize than the convolutional neural network (CNN), it can not only extract the amplitude features of complex data, but also effectively retain and utilize the phase information, thereby providing more accurate and comprehensive feature expression for the application in the fields of wireless sensing and signal processing, therefore, the embodiment of the application adopts the deep complex network to identify the gesture action.

[0124] Most preferably, as shown in Figure 2 The feature extraction module in the deep complex network comprises:

[0125] a first complex convolution layer, a second complex convolution layer for performing complex convolution operation on the input data to extract features, a first max-pooling layer, a second max-pooling layer for performing down-sampling and compression processing on the extracted features;

[0126] The input end of the first complex convolution layer is the input end of the deep complex network for connection with the output end of the fourth complex fully connected layer, the output end of the first complex convolution layer is connected with the input end of the first max-pooling layer, the output end of the second max-pooling layer is connected with the input end of the second complex convolution layer, the output end of the second complex convolution layer is connected with the input end of the second max-pooling layer, the output end of the second max-pooling layer is connected with the input end of the feature compression module for extracting gesture-related features.

[0127] Unlike the traditional real number convolution, the complex number convolution can more accurately capture the pattern in the complex spectrum data and retain the phase information in the data, and this characteristic enables the DCN to significantly improve the feature extraction effect when processing complex data containing spectrum information such as electromagnetic wave signals and acoustic signals.

[0128] After each complex convolution operation, the network down-samples and compresses the extracted features through two layers of max-pooling layers.

[0129] The expression of the complex convolution operation defined in the embodiment of the application is:

[0130] κ*z=(κ1*a-κ2*b)+i(κ2*a+κ1*b)

[0131] wherein κ represents a complex convolution kernel, κ=κ1+iκ2, z represents a complex feature vector, z=a+ib, a represents the real part of the complex number, b represents the imaginary part of the complex number, and i represents the imaginary unit.

[0132] If the real part and the imaginary part of the convolution operation are represented by matrix symbols, then:

[0133]

[0134] In the embodiment of the application, the max-pooling layer down-samples the features with the largest amplitude, thereby reducing the dimension of the feature map while retaining the most important complex-valued features. In this way, the network reduces the computational complexity while retaining information that is crucial for subsequent classification tasks.

[0135] Most preferably, as shown in Figure 2 The feature compression module in the deep complex network comprises, in sequence:

[0136] a fifth complex fully connected layer and a sixth complex fully connected layer for integrating the extracted gesture-related features, a first real fully connected layer and a second real fully connected layer for mapping the integrated features to the real number domain, and a Softmax activation function layer for generating a probability distribution.

[0137] The input end of the fifth complex fully connected layer is connected with the output end of the second max-pooling layer, and the output end of the Softmax activation function layer is the output end of the deep complex network.

[0138] However, in order to implement the final classification task, the model needs to map these abstract complex features to the real number domain, and this step is implemented by two layers of real fully connected layers with Relu activation functions. In this process, the model calculates the L2 norm of the complex features, i.e. wherein z is a feature vector, which is usually used to obtain the amplitude value of the complex number.

[0139] In the embodiment of the application, the Softmax activation function layer converts an unnormalized vector into a probability distribution, each element of which has a value between 0 and 1, and the sum of all elements is 1; it is essentially an extension of the Logistic logistic regression model to multi-classification problems, namely multivariate logistic regression, whose formula is as follows:

[0140]

[0141] In the embodiment of the application, the loss function of the deep complex network adopts a cross-entropy loss function (Cross-Entropy Loss), which is a commonly used loss function for classification problems and can measure the difference between the predicted probability distribution and the actual label distribution. If the predicted probability and the true label differ greatly, the loss value will increase significantly, thereby prompting the model to learn better features. It is not only used for binary classification tasks, but is particularly suitable for multi-class classification tasks.

[0142] Specifically, for gesture recognition, a multi-classification task, the cross-entropy loss function is calculated as follows:

[0143]

[0144] where K is the number of classes, y ij represents 1 if the true class of sample i is equal to j, and 0 otherwise, represents the predicted probability that sample i belongs to j.

[0145] Specifically, step 5 includes:

[0146] The complex channel feature data related to the target human gesture is input into the trained spectral cleaning network, the time-frequency features are extracted through the extraction module in the spectral cleaning network, then the feature nonlinear combination and transformation processing are performed through the transformation module, and regression is fitted, the spectral pollution is cleaned, and enhanced high-resolution spectral data is obtained;

[0147] The enhanced high-resolution spectral data is input into the gesture recognition network, the feature extraction is performed through the feature extraction module therein, the gesture-related features are obtained, then the feature integration is performed through the feature compression module, and the Softmax activation function layer is input to generate the probability distribution of each class, and the gesture recognition result is obtained based on the probability distribution of each class, the gesture recognition result includes but is not limited to ok, bye, wave, praise and stop.

[0148] The embodiment of the present application is based on wireless sensing to collect information about the influence of human gesture action on the WiFi channel state, to obtain real Doppler spectrum data; the real Doppler spectrum data is input into a constructed spectrum cleaning network for processing, to generate polluted spectrum data samples, and the polluted spectrum data samples are used to train the spectrum cleaning network, to obtain the trained spectrum cleaning network; the real Doppler spectrum data is input into the trained spectrum cleaning network for enhancement, to obtain enhanced spectrum data; the enhanced spectrum data is input into a constructed deep complex network, to train the deep complex network, to obtain a gesture recognition network; complex channel feature data related to a target human gesture is input into the trained spectrum cleaning network and the gesture recognition network in sequence for enhancement and subsequent recognition, to obtain a gesture recognition result; compared with the prior art, the embodiment of the present application restores the polluted spectrum data to high-resolution spectrum data through the spectrum cleaning network, maximally reduces the pollution of the spectrum, thereby improving the accuracy of wireless sensing, and then identifies the high-resolution spectrum data through the deep complex network, retains the structural features of the data, thereby improving the recognition accuracy.

[0149] The present application also provides a gesture recognition device based on signal spectrum enhancement, as shown in Figure 3 The gesture recognition device 100 comprises:

[0150] The acquisition module 101 is configured to collect information about the influence of human gesture action on the WiFi channel state based on wireless sensing, to obtain real Doppler spectrum data.

[0151] The processing module 102 is configured to input the real Doppler spectrum data into a constructed spectrum cleaning network for processing, to generate polluted spectrum data samples, and use the polluted spectrum data samples to train the spectrum cleaning network, to obtain the trained spectrum cleaning network.

[0152] The enhancement module 103 is configured to input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement, to obtain enhanced spectrum data.

[0153] The training module 104 is configured to input the enhanced spectrum data into a constructed deep complex network, to train the deep complex network, to obtain a gesture recognition network.

[0154] The recognition module 105 is configured to input complex channel feature data related to a target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhancement and subsequent recognition, to obtain a gesture recognition result.

[0155] The spectrum cleaning network comprises a data construction module, an extraction module and a transformation module connected in sequence, the data construction module is only used to generate training data in the training process, the extraction module is configured to extract features from data, and the transformation module is configured to transform the features output by the previous stage.

[0156] The deep complex network consists of a feature extraction module, a feature compression module, and a classification module connected in sequence. The input of the feature extraction module is connected to the output of the transformation module. The feature compression module is used to calculate the probability of various gestures. The classification module outputs the gesture recognition result.

[0157] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0158] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0159] The present invention also provides a terminal device, such as Figure 4 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, it implements the steps of the above-described method for constructing the decoupled blasting load model. Alternatively, when the processor D100 executes the computer program D102, it implements the functions of each module / unit in the above-described device embodiments.

[0160] The terminal device D10 can be a desktop computer, laptop, handheld computer, server, server cluster, or cloud server, etc. This terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 4The terminal device D10 is merely an example and does not limit the terminal device D10, which can include more or fewer components than shown, or have components in different configurations and / or have different components, such as input / output devices, network access devices, and the like.

[0161] The processor D100 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0162] The memory D101 can be an internal storage unit of the terminal device D10 in some embodiments, such as a hard disk or a memory of the terminal device D10. The memory D101 can also be an external storage device of the terminal device D10 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory D101 can include both an internal storage unit and an external storage device of the terminal device D10. The memory D101 is used to store an operating system, application programs, a boot loader, data, and other programs, such as program codes of the computer program, etc. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0163] It should be noted that the information interaction, execution process, etc. between the above apparatuses / units, since based on the same concept as the method embodiments, the specific functions and the brought technical effects can be referred to the method embodiments part, and will not be described here.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0165] The application further provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the gesture recognition method based on signal spectrum enhancement.

[0166] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer-readable medium can at least include any entity or device capable of carrying the computer program code to the built-in device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0167] The above-mentioned is the preferred embodiment of the application. It should be noted that for those skilled in the art, without departing from the principles of the application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the application.

Claims

1. A gesture recognition method based on signal spectrum enhancement, characterized in that, include: Step 1: Based on wireless sensing, collect information on the impact of human hand gestures on the WiFi channel state to obtain real Doppler spectrum data; Step 2: Input the real Doppler spectrum data into the constructed spectrum cleaning network for processing to generate contaminated spectrum data samples, and use the contaminated spectrum data samples to train the spectrum cleaning network to obtain the trained spectrum cleaning network. Step 3: Input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement to obtain enhanced spectrum data; Step 4: Input the enhanced spectrum data into the constructed deep complex network, train the deep complex network to obtain the gesture recognition network; Step 5: Input the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network in sequence for enhanced recognition to obtain the gesture recognition result; The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process. The extraction module is used to extract features from the data. The transformation module is used to transform the features output by the previous stage. The deep complex network includes a feature extraction module, a feature compression module, and a classification module connected in sequence. The input of the feature extraction module is connected to the output of the transformation module. The feature compression module is used to calculate the probability of various gestures. The classification module outputs the gesture recognition result. Step 1 includes: In wireless sensing, the receiving device obtains raw complex channel feature data by receiving wireless signals sent by the transmitting device. The raw complex channel feature data is used to characterize human hand gestures. Principal component analysis is used to perform dimensionality reduction and noise reduction on the original complex channel feature data to obtain complex channel feature data. The complex channel feature data is segmented by window using short-time Fourier transform to obtain multiple complex channel feature data blocks, with one window corresponding to one complex channel feature data block; For each complex channel feature data block within a window, a Fourier transform is performed on the complex channel feature data block to obtain the real Doppler spectrum data; The original complex channel feature data y is: Among them, h s (f) represents the static component, h d (f,t) represents the dynamic component, h s (f)+h d (f,t) = h(f,t), where h(f,t) represents the gain matrix and a(f,t) represents the magnitude. D represents the Doppler frequency shift of the dynamic path d. d Let d represent the dynamic path, x represent the transmitted signal, and noise represent noise. Step 2 includes: The real Doppler spectrum data is input into the constructed spectrum cleaning network; In the spectrum cleaning network, a uniformly distributed carrier signal is generated within the human activity frequency range based on the sampling rate of the real Doppler spectrum data, and the frequency, amplitude, and phase of the carrier signal are determined by statistical analysis of the real Doppler spectrum data. Based on the window type and window size selected during the short-time Fourier transform, a window function is generated, and the window function is convolved and padded with the carrier signal to obtain a windowed signal that matches the sampling rate. The windowed signal is subjected to a fast Fourier transform to obtain the convolution matrix of the window function in the frequency domain, and the convolution matrix is ​​truncated to obtain an ideal spectrum data sample. The ideal spectrum data sample is convolved with the convolution matrix, and then additive white Gaussian noise is added to generate contaminated spectrum data samples. The spectrum cleaning network is trained using the polluted spectrum data samples to obtain the trained spectrum cleaning network.

2. The gesture recognition method based on signal spectrum enhancement according to claim 1, characterized in that, The extraction module includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a third two-dimensional convolutional layer connected in sequence; The transformation module includes a first complex fully connected layer, a second complex fully connected layer, a third complex fully connected layer, and a fourth complex fully connected layer connected in sequence; The input end of the first two-dimensional convolutional layer is the input end of the spectrum cleaning network, used to receive the real Doppler spectrum data; The output of the third two-dimensional convolutional layer is connected to the input of the first complex fully connected layer. The output of the fourth complex fully connected layer is the output of the spectrum cleaning network, and is used to output the enhanced spectrum data during the application phase.

3. The gesture recognition method based on signal spectrum enhancement according to claim 2, characterized in that, The feature extraction module includes: A first complex convolutional layer and a second complex convolutional layer are used to perform complex convolution operations on the input data to extract features; a first max pooling layer and a second max pooling layer are used to perform downsampling and compression processing on the extracted features. The input of the first complex convolutional layer is the input of the deep complex network and is connected to the output of the fourth complex fully connected layer. The output of the first complex convolutional layer is connected to the input of the first max pooling layer. The output of the first max pooling layer is connected to the input of the second complex convolutional layer. The output of the second complex convolutional layer is connected to the input of the second max pooling layer. The output of the second max pooling layer is connected to the input of the feature compression module for extracting gesture-related features.

4. The gesture recognition method based on signal spectrum enhancement according to claim 3, characterized in that, The feature compression module comprises, in sequence: The fifth and sixth complex fully connected layers are used to integrate the extracted gesture-related features; the first and second real fully connected layers are used to map the integrated features to the real domain; and the Softmax activation function layer is used to generate the probability distribution. The input of the fifth complex fully connected layer is connected to the output of the second max pooling layer, and the output of the Softmax activation function layer is the output of the deep complex network.

5. A gesture recognition device based on signal spectrum enhancement for implementing the method of any one of claims 1-4, characterized in that, include: The acquisition module is used to collect information on the impact of human hand gestures on the WiFi channel state based on wireless sensing, and to acquire real Doppler spectrum data. The processing module is used to input the real Doppler spectrum data into the constructed spectrum cleaning network for processing, generate contaminated spectrum data samples, and use the contaminated spectrum data samples to train the spectrum cleaning network to obtain the trained spectrum cleaning network. The enhancement module is used to input the real Doppler spectrum data into the trained spectrum cleaning network for enhancement, so as to obtain enhanced spectrum data. The training module is used to input the enhanced spectrum data into the constructed deep complex network, train the deep complex network, and obtain the gesture recognition network. The recognition module is used to sequentially input the complex channel feature data related to the target human gesture into the trained spectrum cleaning network and the gesture recognition network for enhanced recognition, so as to obtain the gesture recognition result; The spectrum cleaning network includes a data construction module, an extraction module, and a transformation module connected in sequence. The data construction module is only used to generate training data during the training process. The extraction module is used to extract features from the data. The transformation module is used to transform the features output by the previous stage. The deep complex network includes a feature extraction module, a feature compression module, and a classification module connected in sequence. The input of the feature extraction module is connected to the output of the transformation module. The feature compression module is used to calculate the probability of various gestures. The classification module outputs the gesture recognition result.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the gesture recognition method based on signal spectrum enhancement as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the gesture recognition method based on signal spectrum enhancement as described in any one of claims 1 to 4.

Images