Gesture recognition method using CSI ratio signal based on attention mechanism
By introducing a gesture recognition method based on attention mechanism in Wi-Fi gesture recognition, using CSI ratio signals and deep learning models, the problem of insufficient recognition accuracy and robustness in the prior art is solved, and higher recognition accuracy is achieved and adaptability.
Patent Information
- Application Number
- CN202411675544.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-11
AI Technical Summary
Existing Wi-Fi gesture recognition methods have challenges in recognition accuracy and robustness, especially in the case of fluctuations in signal quality.
Using a gesture recognition method based on attention mechanism, CSI signals are collected through Wi-Fi transceiver devices, CSI ratio signals are calculated, and significant subcarriers are extracted in combination with principal component analysis and short-time Fourier transform. Then, the DFS image is input to the ResNet residual network model, and the feature learning weights are dynamically assigned in combination with the adaptive attention mechanism.
The accuracy and robustness of gesture recognition are significantly improved in both intra-domain and cross-domain situations, and are better than existing technical solutions.
Smart Images
Figure CN119414966A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of gesture recognition, and in particular relates to a method for performing gesture recognition using an attention-based mechanism by using a CSI ratio signal in a Wi-Fi system. Background Art
[0002] Gesture is an important way of human-computer interaction. Fast and accurate gesture recognition can make human-computer interaction more efficient and natural. According to the deployment method of the device, the existing gesture recognition technology is usually divided into two categories: contact and non-contact.
[0003] Contact gesture recognition methods require users to hold or wear dedicated sensing devices. Although such methods can achieve high recognition accuracy, the high reliance on dedicated devices limits their large-scale deployment in daily life. At the same time, users wearing devices will reduce the convenience and naturalness of interaction. In addition, gesture recognition methods based on vision or sound waves also rely on dedicated sensing devices and may involve user privacy issues, which greatly limits the actual feasibility and scope of promotion.
[0004] Compared with contact gesture recognition, contactless methods do not require users to wear any sensor devices, and their non-invasive characteristics enable gesture recognition to be performed in a more natural state. Depending on the type of signal, contactless gesture recognition methods are usually based on wireless signals (such as Wi-Fi, millimeter-wave radar, or ultra-wideband technology), and recognition is completed by analyzing the changes in wireless signals caused by gestures. Although gesture recognition methods based on millimeter-wave radar or ultra-wideband technology can achieve high accuracy, they rely on dedicated hardware devices, have limited usage scenarios, and have high deployment costs. Gesture recognition based on Wi-Fi devices benefits from its wide deployment in indoor and outdoor environments, and has advantages such as low cost and universality. Natural gesture recognition can be achieved without additional hardware configuration. Current Wi-Fi gesture recognition methods mostly process CSI data collected within the monitoring range, extract amplitude or phase features, and combine deep learning to achieve recognition and classification of gestures.
[0005] In summary, among the current gesture recognition technologies, the contactless Wi-Fi gesture recognition method is at the forefront of technology. This method does not require users to wear any special sensors and can provide a more natural and flexible interactive experience. After combining with deep learning, this method has shown certain application prospects, but its recognition effect is affected by fluctuations in signal quality, and recognition accuracy and robustness still face challenges. Summary of the invention
[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a gesture recognition method based on an attention mechanism, which can use CSI ratio signals as features in a intra-domain or cross-domain environment to achieve high-precision and high-robustness gesture action recognition.
[0007] The present invention discloses a gesture recognition method using a CSI ratio signal based on an attention mechanism, which specifically includes the following steps:
[0008] Step 1: Deploy Wi-Fi transceiver equipment in the monitoring area.
[0009] Step 2: Collect CSI signals corresponding to human gestures within the monitoring range.
[0010] Step 3: Calculate the CSI ratio signal using the CSI signals received by different antennas at the receiving end, and use the CSI ratio signal as the basic signal for gesture recognition.
[0011] Step 4: Extract the correlation of the CSI ratio signal based on principal component analysis, and then extract the subcarriers with significant gesture features from the processed signal.
[0012] Step 5: Use short-time Fourier transform to convert the salient subcarriers associated with the gesture into a DFS image.
[0013] Step 6: Input the DFS image into the ResNet residual network model and dynamically assign feature learning weights in combination with the adaptive attention mechanism.
[0014] Step 7: Through experiments, we compared the existing advanced gesture recognition methods and found that it has higher accuracy in both intra-domain and cross-domain situations.
[0015] Specifically, the specific process of step three is as follows:
[0016] The CSI signal received by each antenna at the receiving end of a Wi-Fi device can usually be divided into a static part and a dynamic part. The former consists of line-of-sight propagation and other reflection paths of static objects in the environment, and the latter is the path reflected by human motion. Therefore, CSI can be expressed as follows:
[0017]
[0018] Where A(f, t) represents the amplitude impulse noise, θ0(f, t) is the random phase offset caused by timing alignment offset, sampling frequency offset and carrier frequency offset, and H s (f,t) and H d (f, t) represent the sum of static signal and dynamic signal respectively, a i (f,t) and d i(t) represent the complex attenuation and propagation length of the i-th multipath component, and P d and λ are the set of dynamic signals and the signal wavelength respectively.
[0019] For commercial Wi-Fi devices, since the transmitter and receiver clocks are not synchronized, there is amplitude pulse noise A(f, t) and random phase offset θ0(f, t). Fortunately, this random offset is consistent between different antennas on the same Wi-Fi network interface because they share the same RF oscillator. Therefore, this pulse amplitude noise and random phase offset can be effectively eliminated by the CSI ratio model:
[0020]
[0021] In the formula, H1(f, t) and H2(f, t) respectively represent the CSI readings of the two receiving antennas corresponding to the same receiving end. Since the power ratios of different antennas on the same receiving end are the same and they share the same clock, that is, the different antennas on the same receiver A(f, t)e -jθ0(f,t) By dividing, the pulse amplitude noise and random phase offset can be eliminated.
[0022] Further, the specific process of step 4 is as follows:
[0023] Assume that the CSI ratio signal is H r (f,t)∈R m✕n , where m represents the number of subcarriers and n represents the length of the time series. According to principal component analysis, the signal matrix is first centered and its covariance matrix X is calculated:
[0024]
[0025] Then, the eigenvalue decomposition is used to obtain the eigenvector matrix H and the corresponding eigenvalue λ of the covariance matrix, and the first k eigenvectors whose cumulative eigenvalue contribution rate exceeds the threshold of 95% are selected to construct the dimension reduction matrix P∈R1 n✕k , use this matrix to reduce the dimension of the original CSI ratio signal:
[0026]
[0027] In the formula, the reduced-dimensional matrix Y retains the main features of the CSI ratio signal, removes redundant information and noise, and then extracts the significant subcarrier signal x(t) related to the gesture action from the processed signal.
[0028] Furthermore, the specific process of step 5 is as follows:
[0029] The selected significant subcarrier signal x(t) related to the gesture action is passed through the time-localized Gaussian window function ω, and a short-time Fourier transform is performed on x(t):
[0030]
[0031] Where t is the time center, f is the frequency variable, and ω(ht) is used for the local segment selected at each time t, by calculating |X(f,t)| 2 The time-frequency power spectrum is obtained and used as the pixel value of the DFS image.
[0032] Finally, the specific process of step six is as follows:
[0033] Since the input DFS images have different sizes, they cannot be used directly for deep learning. Therefore, before inputting the DFS images into the deep learning model based on the ResNet residual network and the attention mechanism, the DFS images are first scaled to obtain a 3-channel RGB image x with a size of 224×224.
[0034] Then the scaled feature x is input into the gesture attention module, and learning weights are dynamically assigned to feature x to obtain the gesture attention weight matrix M(x):
[0035]
[0036] In the formula, Sigmoid represents the activation function, Conv represents the convolution operation, AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively.
[0037] After calculating M(x), multiply it by x to perform adaptive feature optimization to obtain the gesture attention feature x', which is calculated as:
[0038]
[0039] Then use the ResNet network to further extract deep feature information x'' from the gesture attention feature x':
[0040]
[0041] Finally, the output result is mapped to the dimension of the number of gesture categories through the fully connected layer to obtain the output result z. The Softmax function is used to map the output of each neuron to a result variable whose element size is in the interval (0,1) and the sum of all elements is 1, and the judgment probability of each gesture category is obtained. The probability calculation formula of the Softmax calculation output result z belonging to the i-th type of gesture is:
[0042]
[0043] Where K is the total number of gesture categories, z i and z c Represent the original scores of the network models of the i-th and c-th categories respectively.
[0044] During the model training process, the cross entropy loss function is used to calculate the difference between the model predicted label category and the actual label category to obtain the model loss value. The cross entropy loss function calculation formula is as follows:
[0045]
[0046] In the formula, L is the loss value, N represents the total number of samples, c represents the model prediction category, and y ic represents the true label of the i-th sample. If the i-th sample category is c, then y ic =1, otherwise 0; p ic It represents the probability that the i-th sample is predicted to be of category c.
[0047] Through the above design, the model can effectively extract the salient features of the DFS image, combine the self-attention mechanism to realize dynamic weight allocation, enhance the model's ability to recognize different gesture features, and thus improve the accuracy and robustness of gesture recognition.
[0048] The beneficial effects of the present invention are as follows: the present invention proposes a gesture recognition method based on an attention mechanism using a CSI ratio signal. The method first deploys a Wi-Fi transceiver in the monitoring area, and then collects CSI signals corresponding to human gestures within the monitoring range; uses CSI signals received by different antennas at the receiving end to calculate the CSI ratio signal, and uses the CSI ratio signal as the basic signal for gesture recognition; then extracts the correlation of the CSI ratio signal according to PCA, and then extracts subcarriers with significant gesture features from the processed signal; then uses STFT to convert the significant subcarriers related to the gesture into a DFS image; finally, the DFS image is input into a ResNet residual network model, and the feature learning weights are dynamically allocated in combination with an adaptive attention mechanism. The present invention compares the existing advanced gesture recognition methods through experiments on public data sets, and finds that it has higher accuracy in both intra-domain and cross-domain situations, and is superior to the current technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 This is a flow chart of a gesture recognition method using CSI ratio signals based on an attention mechanism according to the present invention.
[0050] Figure 2 A schematic diagram of obtaining CSI in the experimental deployment of the present invention.
[0051] Figure 3This is a schematic diagram of the structure of the attention mechanism combined with the ResNet residual network model of the present invention.
[0052] Figure 4 Schematic diagram of the confusion matrix of gesture recognition accuracy of the present invention, where (a) is under the intra-domain condition, (b) is under the cross-position condition, (c) is under the cross-direction condition, and (d) is under the cross-environment condition.
[0053] Figure 5 This is the overall performance of gesture recognition using cross-experiments under different settings of the present invention.
[0054] Figure 6 The present invention is compared with advanced gesture recognition methods under different conditions. DETAILED DESCRIPTION
[0055] The present invention discloses a gesture recognition method using CSI ratio signals based on an attention mechanism. In order to more conveniently compare the existing advanced gesture recognition methods, the present invention conducts a test comparison on a public data set (https: / / tns.thss.tsinghua.edu.cn / widar3.0 / ). The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings.
[0056] like Figure 1 As shown, a gesture recognition method using CSI ratio signals based on an attention mechanism includes the following steps:
[0057] Step 1: If Figure 2 As shown, Wi-Fi devices are deployed in three environments: classroom, hall, and office. In each environment, a signal transmitting device Tx is placed at a fixed position, and a signal receiving device Rx is placed at 6 different positions.
[0058] Step 2: Collect CSI signals corresponding to human gestures (pushing, pulling, swiping, clapping, sliding, drawing O, drawing Z, drawing N, drawing triangle, and drawing rectangle) within the monitoring range, including 10,125 sample data for the classroom environment (9 users × 9 gestures × 5 positions × 5 directions × 5 repetitions), 2,250 sample data for the hall environment (3 users × 6 gestures × 5 positions × 5 directions × 5 repetitions), and 3,000 sample data for the office environment (4 users × 6 gestures × 5 positions × 5 directions × 5 repetitions).
[0059] Step 3: Calculate the CSI ratio signal using the CSI signals received by different antennas at the receiving end, and use the CSI ratio signal as the basic signal for gesture recognition. The process is as follows:
[0060] The CSI signal received by each antenna of the Rx receiving device can be divided into a static part and a dynamic part. The former is composed of line-of-sight propagation and other reflection paths of static objects in the environment, and the latter is the path reflected by gesture actions. Therefore, CSI can be expressed as follows:
[0061]
[0062] Where A(f, t) represents the amplitude impulse noise, θ0(f, t) is the random phase offset caused by timing alignment offset, sampling frequency offset and carrier frequency offset, and H s (f,t) and H d (f, t) represent the sum of static signal and dynamic signal respectively, a i (f,t) and d i (t) represent the complex attenuation and propagation length of the i-th multipath component, and P d and λ are the set of dynamic signals and the signal wavelength respectively.
[0063] For commercial Wi-Fi devices, since the transmitter and receiver clocks are not synchronized, there is amplitude impulse noise A(f, t) and random phase offset θ0(f, t). Fortunately, this random offset is consistent between different antennas on the same Wi-Fi network interface because they share the same RF oscillator.
[0064] Therefore, this pulse amplitude noise and random phase offset can be effectively eliminated by the CSI ratio model:
[0065]
[0066] Where H1(f, t) and H2(f, t) represent the CSI readings of the two receiving antennas corresponding to the same receiving end. Since the power ratios of different antennas on the same receiving end are the same and they share the same clock, i.e., the different antennas on the same receiver have the same A(f, t)e -jθ0(f,t) By dividing, the pulse amplitude noise and random phase offset can be eliminated.
[0067] Step 4: Extract the correlation of the CSI ratio signal based on PCA, and then extract the subcarriers with significant gesture features from the processed signal. The process is as follows:
[0068] Assume that the CSI ratio signal is H r (f,t)∈R m✕n , where m represents the number of subcarriers and n represents the length of the time series. According to principal component analysis, the signal matrix is first centered and its covariance matrix X is calculated:
[0069]
[0070] Then, the eigenvalue decomposition is used to obtain the eigenvector matrix H and the corresponding eigenvalue λ of the covariance matrix, and the first k eigenvectors whose cumulative eigenvalue contribution rate exceeds the threshold of 95% are selected to construct the dimension reduction matrix P∈R1 n✕k , use this matrix to reduce the dimension of the original CSI ratio signal:
[0071]
[0072] In the formula, the reduced-dimensional matrix Y retains the main features of the CSI ratio signal, removes redundant information and noise, and then extracts the significant subcarrier signal x(t) related to the gesture action from the processed signal.
[0073] Step 5: Use STFT to convert the salient subcarrier signal related to the gesture action into a DFS image. The process is as follows:
[0074] The extracted significant subcarrier signal x(t) related to the gesture action is passed through the time-localized Gaussian window function ω, and a short-time Fourier transform is performed on x(t):
[0075]
[0076] Where t is the time center, f is the frequency variable, and ω(ht) is used for the local segment selected at each time t. By calculating |X(f,t)| 2 The time-frequency power spectrum is obtained and used as the pixel value of the DFS image.
[0077] Step 6: If Figure 3 As shown, the DFS image is input into the ResNet residual network model, and the feature learning weights are dynamically allocated in combination with the adaptive attention mechanism. The specific process is as follows:
[0078] Due to the difference in the size of the input DFS image, it cannot be directly used for deep learning. Therefore, before inputting the DFS image into the deep learning model based on the ResNet residual network and attention mechanism, the DFS image is first scaled to obtain a 3-channel RGB image x with a size of 224×224;
[0079] Then the scaled feature x is input into the gesture attention module, and learning weights are dynamically assigned to feature x to obtain the gesture attention weight matrix M(x):
[0080]
[0081] In the formula, Sigmoid represents the activation function, Conv represents the convolution operation, AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively.
[0082] After calculating M(x), multiply it by x to perform adaptive feature optimization to obtain the gesture attention feature x', which is calculated as:
[0083]
[0084] Then use the ResNet network to further extract deep feature information x'' from the gesture attention feature x':
[0085]
[0086] Finally, the output result is mapped to the dimension of the number of gesture categories through the fully connected layer to obtain the output result z; then the Softmax function is used to map the output of each neuron to a result variable whose element size is in the interval (0,1) and the sum of all elements is 1, and the judgment probability of each gesture category is obtained. The probability calculation formula of the Softmax calculation output result z belonging to the i-th gesture is:
[0087]
[0088] Where K is the total number of gesture categories, z i and z c Represent the original scores of the network models of the i-th and c-th categories respectively.
[0089] During the model training process, the cross entropy loss function is used to calculate the difference between the model predicted label category and the true label category to obtain the model loss value. The cross entropy loss function calculation formula is as follows:
[0090]
[0091] In the formula, L is the loss value, N represents the total number of samples, c represents the model prediction category, and y ic represents the true label of the i-th sample. If the i-th sample category is c, then y ic =1, otherwise 0; p ic It represents the probability that the i-th sample is predicted to be of category c.
[0092] To optimize the model, the Adam optimizer was used in the training process, the training epoch was set to 20, the initial learning rate was set to 0.0001, and an adaptive learning rate decay strategy was adopted. The learning rate was reduced to 0.5 times the original value every 3 epochs of training, and the batch size was set to 20.
[0093] Step 7: Through experimental evaluation, a 5-fold crossover experiment is used for intra-domain, cross-location, and cross-direction (80% of the data is used as a training set and 20% of the data is used as a test set), and a 3-fold crossover experiment is used for cross-environment (two environment data are used for training and the other environment data is used for testing). Figure 4 As shown, the confusion matrices within the domain, across locations, across directions, and across environments are shown; Figure 5 As shown in , the accuracy of cross-experiments under different settings is shown, where the average accuracy within the domain is 99.71%, the average accuracy across positions is 98.45%, the average accuracy across directions is 95.63%, and the average accuracy across environments is 95.38%; Figure 6 As shown, compared with existing advanced gesture recognition methods (Widar3.0, WiHF, WiGRUNT), it is found that it has higher accuracy in both intra-domain and cross-domain situations.
[0094] It should be noted that the above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto, and any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.
Claims
1. A gesture recognition method using CSI ratio signals based on an attention mechanism, comprising the following steps: Step 1: Deploy Wi-Fi transceiver equipment in the monitoring area. Step 2: Collect CSI signals corresponding to human gestures within the monitoring range. Step 3: Calculate the CSI ratio signal using the CSI signals received by different antennas at the receiving end, and use the CSI ratio signal as the basic signal for gesture recognition. Step 4: Extract the correlation of the CSI ratio signal based on principal component analysis, and then extract the subcarriers with significant gesture features from the processed signal. Step 5: Use short-time Fourier transform to convert the salient subcarriers associated with the gesture into a DFS image. Step 6: Input the DFS image into the ResNet residual network model and dynamically assign feature learning weights in combination with the adaptive attention mechanism. Step 7: Through experiments, we compared the existing advanced gesture recognition methods and found that it has higher accuracy in both intra-domain and cross-domain situations.
2. A gesture recognition method based on an attention mechanism using a CSI ratio signal as claimed in claim 1, characterized in that The process of step three is as follows: The CSI signal received by each antenna at the receiving end of a Wi-Fi device can usually be divided into a static part and a dynamic part. The former consists of line-of-sight propagation and other reflection paths of static objects in the environment, and the latter is the path reflected by human motion. Therefore, CSI can be expressed as follows: Where A(f, t) represents the amplitude impulse noise, θ0(f, t) is the random phase offset caused by timing alignment offset, sampling frequency offset and carrier frequency offset, and H s (f,t) and H d (f, t) represent the sum of static signal and dynamic signal respectively, a i (f,t) and d i (t) represent the complex attenuation and propagation length of the i-th multipath component, and P d and λ are the set of dynamic signals and the signal wavelength respectively. For commercial Wi-Fi devices, since the transmitter and receiver clocks are not synchronized, there is amplitude impulse noise A(f, t) and random phase offset θ0(f, t). Fortunately, this random offset is consistent between different antennas on the same Wi-Fi network interface because they share the same RF oscillator. Therefore, this pulse amplitude noise and random phase offset can be effectively eliminated by the CSI ratio model: Where H1(f, t) and H2(f, t) represent the CSI readings of the two receiving antennas corresponding to the same receiving end. Since the power ratios of different antennas on the same receiving end are the same and they share the same clock, i.e., the different antennas on the same receiver have the same A(f, t)e -jθ0(f,t) By dividing, the pulse amplitude noise and random phase offset can be eliminated.
3. A gesture recognition method based on an attention mechanism using a CSI ratio signal as claimed in claim 1, characterized in that The process of step 4 is as follows: Assume that the CSI ratio signal is H r (f,t)∈R m✕n , where m represents the number of subcarriers and n represents the length of the time series. According to principal component analysis, the signal matrix is first centered and its covariance matrix X is calculated: Then, the eigenvalue decomposition is used to obtain the eigenvector matrix H and the corresponding eigenvalue λ of the covariance matrix, and the first k eigenvectors whose cumulative eigenvalue contribution rate exceeds the threshold of 95% are selected to construct the dimension reduction matrix P∈R1 n✕k , use this matrix to reduce the dimension of the original CSI ratio signal: In the formula, the reduced-dimensional matrix Y retains the main features of the CSI ratio signal, removes redundant information and noise, and then extracts the significant subcarrier signal x(t) related to the gesture action from the processed signal.
4. A gesture recognition method based on an attention mechanism using a CSI ratio signal as claimed in claim 1, characterized in that The process of step five is as follows: The extracted significant subcarrier signal x(t) related to the gesture action is passed through the time-localized Gaussian window function ω, and a short-time Fourier transform is performed on x(t): Where t is the time center, f is the frequency variable, and ω(ht) is used for the local segment selected at each time t. By calculating |X(f,t)| 2 The time-frequency power spectrum is obtained and used as the pixel value of the DFS image.
5. A gesture recognition method based on an attention mechanism using a CSI ratio signal as claimed in claim 1, characterized in that The process of step six is as follows: Since the input DFS images have different sizes, they cannot be directly used for deep learning. Therefore, before inputting the DFS images into the deep learning model based on the ResNet residual network and the attention mechanism, the DFS images are first scaled to obtain a 3-channel RGB image x with a size of 224×224. Then the scaled feature x is input into the gesture attention module, and learning weights are dynamically assigned to feature x to obtain the gesture attention weight matrix M(x): In the formula, Sigmoid represents the activation function, Conv represents the convolution operation, AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively. After calculating M(x), multiply it by x to perform adaptive feature optimization to obtain the gesture attention feature x', which is calculated as: Then use the ResNet network to further extract deep feature information x'' from the gesture attention feature x': Finally, the output result is mapped to the dimension of the number of gesture categories through the fully connected layer to obtain the output result z; then the Softmax function is used to map the output of each neuron to a result variable whose element size is in the interval (0,1) and the sum of all elements is 1, and the judgment probability of each gesture category is obtained. The probability calculation formula of the Softmax calculation output result z belonging to the i-th gesture is: Where K is the total number of gesture categories, z i and z c Represent the original scores of the network models of the i-th and c-th categories respectively. During the model training process, the cross entropy loss function is used to calculate the difference between the model predicted label category and the true label category to obtain the model loss value. The cross entropy loss function calculation formula is as follows: In the formula, L is the loss value, N represents the total number of samples, c represents the model prediction category, and y ic represents the true label of the i-th sample. If the i-th sample category is c, then y ic =1, otherwise 0; p ic It represents the probability that the i-th sample is predicted to be of category c.