A gesture recognition and sensing method and system based on Wi-Fi signals

Abstract

This invention discloses a gesture recognition and perception method and system based on Wi-Fi signals. The method includes: acquiring Wi-Fi signals in the environment and outputting a two-dimensional data matrix containing amplitude and phase information; transforming the data matrix into a standard input format adapted to a sequence encoder, and pre-training the encoder and decoder through a mask self-supervised reconstruction mechanism; retaining the structure and parameters of the pre-trained encoder, replacing the decoder with a gesture recognition network composed of a lightweight classification network, and performing end-to-end training of the model using supervised optimization; using the end-to-end trained model, inputting the actually acquired raw signal data containing amplitude and phase information, and outputting the corresponding gesture category. This invention directly utilizes raw Wi-Fi CSI signals to achieve low-cost, high-precision, and highly generalizable non-contact gesture recognition, suitable for deployment on mobile terminals and edge devices in real-world environments.

Description

Technical Field

[0001] This invention belongs to the field of wireless sensing and artificial intelligence technology, and in particular relates to a gesture recognition and sensing method and system based on Wi-Fi signals, which is applicable to various human-computer interaction scenarios. Background Technology

[0002] Wireless sensing technology has been widely applied in recent years in fields such as smart homes, human body sensing, and human-computer interaction due to its advantages of being contactless, privacy-preserving, and capable of operating 24 / 7. Among these, Wi-Fi signals, with their extensive deployment and good penetration capabilities, have become an important means of sensing indoor and outdoor environments. By analyzing the Channel State Information (CSI) in Wi-Fi signals, researchers have achieved various sensing tasks, such as human presence detection, posture estimation, and gesture recognition.

[0003] However, most existing Wi-Fi sensing technologies rely on complex signal processing algorithms or manually designed feature extraction processes. These methods typically require specialized domain knowledge, have long algorithm development cycles, and are difficult to adapt to complex environmental changes in different scenarios. Furthermore, to improve system sensing accuracy, some solutions depend on high-cost, complex hardware deployments, limiting their practical application. Summary of the Invention

[0004] The purpose of this invention is to provide a gesture recognition and sensing method and system based on Wi-Fi signals, which realizes low-cost, high-precision, and highly generalizable non-contact gesture recognition, and is suitable for deployment in mobile terminals and edge devices in real-world environments.

[0005] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows:

[0006] A gesture recognition and sensing method based on Wi-Fi signals includes the following steps:

[0007] Based on the raw Wi-Fi signals collected from the environment, a two-dimensional data matrix containing amplitude and phase information of the wireless channel is output.

[0008] The data matrix is ​​uniformly transformed into a standard input format suitable for the sequence encoder. Parts are masked through random masking. Two sequence encoders with identical structures are used to encode the masked amplitude and phase information respectively, resulting in amplitude and phase feature sequences. The two sequences are fused by a joint encoding module to obtain a joint feature sequence. The decoder is used to reconstruct the amplitude and phase at the masked positions in the joint feature sequence, obtaining a reconstruction result with the same dimensions as the original input. The difference between the reconstructed result and the corresponding amplitude and phase in the original signal is calculated as the loss function, and the encoder and decoder are pre-trained.

[0009] The structure and parameters of the pre-trained sequence encoder and feature encoding module are retained, and the decoder is replaced with a gesture recognition network composed of a lightweight classification network. Supervised optimization is used to minimize the difference between the predicted class distribution and the true label, and the model is trained end-to-end.

[0010] Using a model trained end-to-end, the system takes raw signal data containing amplitude and phase information as input, processes it through a sequence encoder, a joint encoding module, and a gesture classification network, and outputs the corresponding gesture category to achieve non-contact gesture recognition.

[0011] Preferably, based on the collected raw Wi-Fi signal from the environment, a two-dimensional data matrix containing amplitude and phase information of the wireless channel is output, including:

[0012] Based on channel state information data covering several subcarriers collected by several receivers, channel response data with time and spatial dimensions is constructed. The channel response data is a three-dimensional data tensor with the structure (B, T, D), where B is the batch size, T is the time length, and D is the spatial dimension. The value of the spatial dimension is obtained by multiplying the number of subcarriers × the number of antennas × the number of receivers.

[0013] Preferably, partially occluding a segment through random occlusion operation includes:

[0014] The amplitude branch and phase branch of the input channel state information are linearly transformed through a linear layer to convert the spatial dimension into the feature dimension, and the output tensor shape is (B, T, C), where C is the feature dimension.

[0015] Add a sinusoidal position code to each branch;

[0016] T - L time steps are randomly selected from the total time length T of the sequence as the masking positions. At each selected time step, all spatial channel dimensions corresponding to that time step are set as masks, while the other time steps remain intact. The final amplitude and phase output after masking are (B, L, C).

[0017] Preferably, two identical sequence encoders are used to encode the masked amplitude and phase information respectively, obtaining amplitude feature and phase feature sequences. The two sequences are then fused into a joint feature sequence via a joint encoding module, including:

[0018] The amplitude and phase features after masking are tensors of shape (B, L, C), where B is the batch size, L is the masking time length, and C is the feature dimension. The two signals are fed into two Mamba encoders with the same structure but independent parameters, respectively, to perform time modeling on the amplitude sequence and phase sequence. The encoder outputs amplitude and phase feature sequences of (B, L, C) respectively.

[0019] The two sequences output by the encoder are concatenated along the feature dimension to obtain a fused feature sequence of shape (B, L, 2C). This (B, L 2C) fused feature sequence is input into the joint encoding module and, after a linear transformation, is mapped to the output features of (B, L, C).

[0020] Preferably, the decoder reconstructs the amplitude and phase of the occluded positions in the joint feature sequence to obtain a reconstruction result with the same dimensions as the original input, including:

[0021] Generate a set of learnable temporal step location representations, a trainable embedding matrix of shape (TL, C), and concatenate it with the joint feature sequence at the occlusion location along the temporal dimension to form a new input representation (B, T, C);

[0022] The combined input sequence decoding module employs a multi-layer Mamba decoder similar in structure to the encoder. It processes the fused features along the time dimension and outputs two decoding branches, corresponding to the reconstruction of amplitude and phase, respectively. Each decoding branch uses a separate projection head to restore the features to the same dimension as the original input, i.e., (B, T, D), and reconstructs the amplitude and phase of the masked positions in the original signal.

[0023] Preferably, the gesture recognition network, which consists of a lightweight classification network, includes:

[0024] The joint feature sequence of (B, T, C) is compressed along the time dimension to obtain the global feature representation (B, C), which is then input into a classifier to output the final category prediction result (B, N), where N is the number of gesture categories.

[0025] Preferably, using the end-to-end trained model, the input is the raw signal data containing amplitude and phase information, which is actually collected. After processing by a sequence encoder, joint encoder, and gesture classification network, the corresponding gesture category is output, including:

[0026] The input data is a complete sequence of amplitude and phase information of the original channel state, i.e., amplitude and phase tensors of shape (B, T, D). No masking operation is performed. The complete input signal is first processed by two independent linear layers and a Mamba encoder to extract the signal features of amplitude and phase, respectively, and outputs a temporal representation of shape (B, T, C). The two features are then concatenated along the channel dimension to form a fused representation of (B, T, 2C), which is then mapped to a joint feature sequence of (B, T, C) by a linear transformation in the joint coding module. The gesture recognition network compresses the joint feature sequence of (B, T, C) along the time dimension to obtain a global feature representation of (B, C), which is then input into a classifier to output the final category prediction result (B, N).

[0027] A gesture recognition and sensing system based on Wi-Fi signals, comprising:

[0028] The data acquisition module is used to output a two-dimensional data matrix containing amplitude and phase information of the wireless channel based on the raw Wi-Fi signal collected from the environment.

[0029] The self-supervised masking reconstruction pre-training module transforms the data matrix into a standard input format suitable for sequence encoders. It masks certain segments through random masking operations, and uses two identical sequence encoders to encode the masked amplitude and phase information, respectively, to obtain amplitude and phase feature sequences. These two sequences are then fused by a joint encoding module to obtain a joint feature sequence. A decoder reconstructs the amplitude and phase at the masked positions in the joint feature sequence, yielding a reconstruction result with the same dimensions as the original input. The difference between the reconstructed result and the corresponding amplitude and phase in the original signal is used as a loss function for pre-training the encoder and decoder.

[0030] The model fine-tuning module is used to retain the structure and parameters of the pre-trained sequence encoder and feature encoding module, replace the decoder with a gesture recognition network composed of a lightweight classification network, and use supervised optimization to minimize the difference between the predicted class distribution and the true label for end-to-end model training.

[0031] The gesture classification and determination module is used to take the raw signal data containing amplitude and phase information that is actually collected as input, which is trained by the end-to-end model, and processed by the sequence encoder, joint encoding module and gesture classification network to output the corresponding gesture category, thereby realizing non-contact gesture recognition function.

[0032] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of a gesture recognition and perception method based on Wi-Fi signals as described above.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a gesture recognition and perception method based on Wi-Fi signals as described above.

[0034] Beneficial Effects: This invention proposes a Wi-Fi signal gesture recognition and sensing method based on self-supervised learning. By designing a lightweight joint coding structure and mask reconstruction strategy, it achieves efficient extraction of key features from amplitude and phase information without signal processing. Compared with existing solutions, this invention significantly reduces data dependency and annotation costs, has a simple system structure, low computational resource consumption, and is suitable for deployment on mobile terminals or edge devices. It solves the problems of traditional Wi-Fi sensing systems relying on complex signal processing, cumbersome feature design, and difficult model deployment. Actual tests show that in typical Wi-Fi gesture classification tasks, this invention achieves a classification accuracy of 90.52% directly based on raw data without any signal preprocessing, balancing high precision, low overhead, and good system practicality, and has broad practical application prospects. Attached Figure Description

[0035] Figure 1 This is a flowchart of the overall process of the present invention;

[0036] Figure 2 This is a diagram of the pre-trained model for self-supervised mask reconstruction in this invention;

[0037] Figure 3 This is a structural diagram of the Mamba encoder / decoder of the present invention;

[0038] Figure 4 This is a diagram of the gesture classification task framework after fine-tuning in this invention. Detailed Implementation

[0039] In recent years, some studies have attempted to introduce data-driven end-to-end modeling methods, using deep learning to automatically extract features from raw wireless signals, aiming to eliminate the need for cumbersome manual signal processing. However, most of these methods rely on large amounts of labeled data for supervised training, making it difficult to obtain sufficient high-quality labeled data in practical deployments, severely limiting the system's scalability and generalization ability. Furthermore, Wi-Fi CSI signals inherently possess dual-modal information of amplitude and phase, which are complexly coupled. Traditional independent processing methods often overlook their inherent correlation, affecting the final sensing effect. To address these challenges, there is an urgent need to develop a wireless sensing system that requires no complex signal processing, possesses self-supervised learning capabilities, and can jointly model the amplitude and phase information of Wi-Fi signals, achieving low-cost, high-precision, and highly generalized sensing capabilities suitable for various practical application scenarios.

[0040] Based on this need, this invention designs a Wi-Fi signal gesture recognition and perception method and system based on self-supervised learning. Through innovative signal encoding design and self-supervised training methods, this invention effectively utilizes the rich information in the original Wi-Fi signal, eliminating the dependence of traditional methods on complex signal processing and large amounts of labeled data, and improving the actual performance and deployment flexibility of the system in tasks such as gesture recognition.

[0041] Reference Figure 1 The present invention proposes a gesture recognition and sensing method based on Wi-Fi signals, comprising the following steps:

[0042] Step S1, signal acquisition.

[0043] This invention utilizes multiple Wi-Fi receiving devices to acquire wireless signals from the environment, enabling contactless gesture sensing. The signal acquisition process includes the following steps:

[0044] Step S1-1: Deployment of Multiple Receivers. Several receivers are deployed in the target sensing area. In this example, six receivers (six Wi-Fi receivers) are used. Each receiver is equipped with several antennas; in this example, three antennas are used, meaning each receiver has three receiving antennas. The receiving devices maintain a stable communication connection with the fixed transmitter and use a network card (Intel 5300 device) that supports CSI message acquisition to obtain the physical layer channel response during communication. In this example, 30 subcarriers are used, and the sampling frequency is set in the range of 100–1000 Hz to ensure that the channel status can fully reflect the minor disturbances during the gesture activity.

[0045] Step S1-2: Raw Data Acquisition. Each sampled CSI data point is a complex tensor with shape (6, 3, 30), where each element represents the channel response at a specific receiver, antenna, and subcarrier. This signal is then split into amplitude and phase components, stored as real-valued matrices respectively. After continuous time-step sampling, the data is stacked to form a four-dimensional structure with shape (T, 6, 3, 30), where T is the time step number.

[0046] Steps S1-3: Data Reconstruction. Data is cached and timestamped at the acquisition end using a driver program, saving it as a time-ordered signal sequence. All raw CSI samples are directly input into the subsequent modeling process without any manual signal processing, preserving the original physical layer feature structure. To standardize the input format, each sequence is cropped according to a fixed time window (e.g., 1450 frames per sample), and the spatial dimension is flattened, ultimately stored as a standardized two-dimensional tensor sample (T, D), where T=1450 is the time dimension and D=540 is the spatial dimension after flattening the receiver, antenna, and subcarriers.

[0047] Step S2: Pre-training of self-supervised mask reconstruction based on MAE (Masked Autoencoder).

[0048] To address the issues of traditional methods relying on large amounts of labeled data and incurring high training costs, this invention designs a self-supervised mask reconstruction strategy based on MAE, used for feature modeling of the original CSI signal under self-supervised conditions. This stage is also known as the pre-training stage. Figure 2 The diagram shows the MAE pre-trained model of this invention: The pre-training structure takes the amplitude and phase of the original Wi-Fi CSI as input, performs dimensionality transformation through two linear layers, then performs temporal masking, and finally feeds it into an independent Mamba encoder for temporal modeling. The encoder uses Mamba blocks to capture temporal dependencies in the sequence. The temporal features of the two branches are fused after encoding and then input into two Mamba decoders that reconstruct the amplitude and phase, respectively, to achieve self-supervised learning. No manual annotation is required throughout the process; by reconstructing the time segments of the mask, the model can automatically learn effective spatiotemporal features. To enable the encoder to perform more functions during the pre-training stage, the number of encoder layers should be greater than the number of decoder layers.

[0049] The specific process is as follows:

[0050] Step S2-1: Signal Mapping. For each batch of CSI samples, the amplitude and phase signals are fed into two independent linear projection modules. Both modules have identical structures, consisting of a single learnable fully connected layer to map the input spatial dimension D to a unified feature dimension C. This process outputs two feature sequences of shape (B, T, C), representing the amplitude and phase features, respectively. This independent mapping operation preserves the representational differences between different signal types. Then, sinusoidal position encoding is added to each branch to prepare for subsequent random masking and joint processing.

[0051] Step S2-2: Random Masking. The masking strategy is designed based on the time dimension. During training, T - L time steps (corresponding to proportions that can be set to 25%–90%) are randomly selected from the total sequence time length T as the masked positions. At each selected time step, all spatial channel dimensions corresponding to that time step (i.e., all receivers, antennas, and subcarriers before flattening) are masked to simulate signal loss at the entire time point. Masked time steps do not participate in the encoder input, while unmasked time steps will continue to be fed into the encoder for feature encoding processing.

[0052] Step S2-3, sequence encoding / decoding training.

[0053] To enhance the system's ability to comprehensively process complex sequence signal information, this invention designs a sequence encoding / decoding mechanism. Figure 3 The following is a structural diagram of the Mamba encoder / decoder in this invention: Each encoder / decoder block contains a state space transition path to process the original timing information, resulting in encoded features containing timing information. These encoded features are then activated for subsequent processing.

[0054] Specifically, during the pre-training phase, the encoding and decoding processes proceed as follows:

[0055] Step S2-3-1: Independent Sequence Encoding of Amplitude and Phase. During the pre-training phase, the input consists of two feature sequences of amplitude and phase with shapes (B, L, C), where B is the batch size, L is the remaining time steps after occlusion, and C is the feature dimension. The amplitude and phase are fed into two structurally identical but parameter-independent Mamba encoders for temporal modeling. Each Mamba encoder consists of multiple cascaded Mamba blocks, utilizing trainable one-dimensional convolutional kernels and state transition layers to perform efficient temporal convolution operations on the input sequence, thereby capturing long-range temporal dependencies. Specifically, each layer in the encoder performs convolution calculations on the input sequence x using a one-dimensional convolutional kernel K, followed by further processing of the sequence using a state transition layer. The final output is added to the input after activation function processing, and a linear transformation is performed to stabilize the training. The final output sequence shape remains (B, L, C). After temporal modeling by their respective encoders, the amplitude and phase obtain corresponding encoded feature sequences for use in the subsequent joint encoding stage.

[0056] Step S2-3-2: Joint Encoding and Feature Fusion. After completing the independent sequence encoding of the amplitude and phase signals, two temporal feature representations with shapes (B, L, C) are obtained, corresponding to the amplitude channel and the phase channel, respectively. To capture the potential complementary relationship between the two, this invention designs a joint encoding and feature fusion mechanism. Specifically, the two feature sequences are first concatenated along the channel dimension to obtain a fused feature with shape (B, L, 2C). This operation preserves the original expressive power of the two types of signals and provides a foundation for subsequent information interaction. Subsequently, the fused feature is input into a linear transformation module, which is essentially a fully connected layer with learnable parameters, to map the concatenated channel dimension from 2C back to C, achieving preliminary information integration. After the linear transformation, the output feature still maintains the shape (B, L, C) and carries the fused expression of the two types of signals.

[0057] This fusion process not only preserves the independence of amplitude and phase, but also establishes a coupling channel between the two, providing a unified and structured timing representation for the entire system.

[0058] The output joint feature sequence (B, L, C) is used by the decoder to reconstruct the masked amplitude and phase signals. The specific process of the decoder is as follows:

[0059] Step S2-3-3: To recover the temporal structure of the original sequence, a set of learnable temporal position embeddings is constructed, where each row represents the time-step embedding of a masked position. This temporal position embedding is used as part of the input during decoding, progressively generating and recovering the temporal data of the original masked positions. Subsequently, according to the order of the masked positions, this temporal position embedding is concatenated with the joint feature sequence in the original temporal order, ultimately forming a decoding input representation of shape (B, T, C). This input is fed into two decoding branches: an amplitude decoder and a phase decoder. Each branch consists of several layers of lightweight Mamba blocks, possessing effective temporal modeling capabilities and able to capture the contextual dependencies required for signal reconstruction step-by-step. The sequence decoding process is consistent with the sequence encoding process.

[0060] Step S2-3-4: Signal Recovery. The two decoding branches output amplitude and phase feature recovery results of shape (B, T, C), representing the reconstruction estimate of the occluded time step in the feature space. To restore the original signal space, a linear reconstruction layer is connected to the end of each branch, mapping the feature vector back to the same amplitude / phase space dimension as the input signal. It is worth noting that, to enable the encoder to perform more functions during the self-supervised pre-training stage, the number of encoder layers will be greater than the number of decoder layers.

[0061] With supervision only applied at the occluded time step, the reconstructed outputs are aligned with the true amplitude and phase at the occluded location in the original signal. The encoder / decoder model is trained by calculating the mean square error (MSE) element-wise as the reconstruction loss. The loss term calculates the difference between the reconstructed result and the corresponding amplitude and phase in the original CSI signal, accumulating only at the occluded time step.

[0062] Step S3, model fine-tuning stage.

[0063] After completing self-supervised pre-training, this invention enters the fine-tuning stage. During this stage, a supervised recognition network is constructed and trained end-to-end while retaining the learned representation capabilities of the pre-trained model. Specifically, the entire structure and parameters of the amplitude encoder, phase encoder, and joint encoding module trained in the self-supervised stage are retained for feature extraction and fusion processing of the CSI signal. The input data is the complete original CSI amplitude and phase sequence, without any masking operations. The input complete CSI signal first extracts the amplitude and phase signal features through two independent linear layers and a Mamba encoder, outputting a time-series representation of shape (B, T, C). The remaining processing steps are consistent with those in the pre-training stage; that is, the two features are then concatenated along the channel dimension to form a fused representation of (B, T, 2C), which is then mapped to a joint feature sequence of (B, T, C) through a linear transformation in the joint encoding module.

[0064] During the fine-tuning phase, the original decoder module was replaced with a gesture recognition network to classify motion patterns in the entire CSI sequence. This gesture recognition network consists of a lightweight classification network, such as a multilayer perceptron. First, the feature sequences along the time dimension are aggregated using global average pooling, compressing the three-dimensional features (B, T, C) into a two-dimensional vector representation of (B, C). This process generates a global feature summary for each input sequence. The pooled feature vector serves as the input to the classifier, transformed into an output of (B, N) through a linear layer, where N represents the total number of predefined gesture categories. The output is then converted into a probability distribution using a softmax function for classification. The training process employs cross-entropy loss as the loss function to guide supervised optimization, aiming to minimize the difference between the predicted category distribution and the true label.

[0065] Step S4, gesture classification and determination.

[0066] In gesture classification, a finely tuned network model is used, namely... Figure 4The end-to-end network shown first extracts the amplitude and phase signal features of the input complete CSI signal through two independent linear layers and a Mamba encoder, respectively, and outputs a temporal representation with shape (B, T, C). The two features are then concatenated along the channel dimension to form a fused representation of (B, T, 2C), which is then mapped to a joint feature sequence of (B, T, C) by a linear transformation in the joint encoding module. This sequence then enters a lightweight gesture classification network (such as a multilayer perceptron), where the joint feature sequence of (B, T, C) is first compressed along the time dimension to obtain a global feature representation of (B, C). Finally, the classifier outputs the final category prediction result (B, N), where N is the number of gesture categories.

[0067] Through the above design, this invention comprehensively integrates features while ensuring the system's lightweight nature and high efficiency. Table 1 shows the gesture classification accuracy of this invention: Compared with supervised methods that require extensive manual feature design and signal processing, this method models Wi-Fi CSI signals in an end-to-end manner. Based on label-free pre-training, it achieves an accuracy of 90.52% in gesture classification tasks with only fine-tuning, demonstrating excellent performance and generalization ability. This indicates that the design of this invention is naturally suitable for processing sequence data of signal types, enabling excellent end-to-end performance. It meets the high-performance and convenient deployment requirements of wireless sensing systems in practical applications.

[0068] Table 1 Comparison of Gesture Classification Accuracy of Several Methods

[0069]

[0070] Another embodiment of the present invention provides a gesture recognition and sensing system based on Wi-Fi signals, comprising:

[0071] The data acquisition module is used to output a two-dimensional data matrix containing amplitude and phase information of the wireless channel based on the raw Wi-Fi signal collected from the environment.

[0072] The self-supervised masking reconstruction pre-training module transforms the data matrix into a standard input format suitable for sequence encoders. It masks certain segments through random masking operations, and uses two identical sequence encoders to encode the masked amplitude and phase information, respectively, to obtain amplitude and phase feature sequences. These two sequences are then fused by a joint encoding module to obtain a joint feature sequence. A decoder reconstructs the amplitude and phase at the masked positions in the joint feature sequence, yielding a reconstruction result with the same dimensions as the original input. The difference between the reconstructed result and the corresponding amplitude and phase in the original signal is used as a loss function for pre-training the encoder and decoder.

[0073] The model fine-tuning module is used to retain the structure and parameters of the pre-trained sequence encoder and feature encoding module, replace the decoder with a gesture recognition network composed of a lightweight classification network, and use supervised optimization to minimize the difference between the predicted class distribution and the true label for end-to-end model training.

[0074] The gesture classification and determination module is used to take the raw signal data containing amplitude and phase information that is actually collected as input, which is trained by the end-to-end model, and processed by the sequence encoder, joint encoding module and gesture classification network to output the corresponding gesture category, thereby realizing non-contact gesture recognition function.

[0075] It should be understood that the gesture recognition and sensing system based on Wi-Fi signals in the embodiments of the present invention can implement all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above embodiments, which will not be repeated here.

[0076] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the gesture recognition and perception method based on Wi-Fi signals as described above.

[0077] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the gesture recognition and perception method based on Wi-Fi signals as described above.

[0078] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems (apparatus), electronic devices, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations thereof can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1A device for a function specified in one or more processes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A gesture recognition and sensing method based on Wi-Fi signals, characterized in that, Includes the following steps: Based on the raw Wi-Fi signals collected from the environment, a two-dimensional data matrix containing amplitude and phase information of the wireless channel is output. The data matrix is ​​uniformly transformed into a standard input format suitable for the sequence encoder. Parts are masked through random masking. Two sequence encoders with identical structures encode the masked amplitude and phase information respectively, obtaining amplitude and phase feature sequences. The two sequences are fused by a joint encoding module to obtain a joint feature sequence. The decoder reconstructs the amplitude and phase at the masked positions in the joint feature sequence, obtaining a reconstruction result with the same dimensions as the original input. The difference between the reconstructed result and the corresponding amplitude and phase in the original signal is calculated as the loss function, and the sequence encoder and decoder are pre-trained. The structure and parameters of the pre-trained sequence encoder and joint encoder module are retained, and the decoder is replaced with a gesture recognition network composed of a lightweight classification network. Supervised optimization is used to minimize the difference between the predicted class distribution and the true label, and the model is trained end-to-end. Using the end-to-end trained model, the raw signal data containing amplitude and phase information is actually collected and processed by a sequence encoder, a joint coding module and a gesture recognition network to output the corresponding gesture category, thus realizing non-contact gesture recognition function. Among them, partial segments are masked through random masking operations, including: The amplitude branch and phase branch of the input channel state information are linearly transformed through a linear layer to convert the spatial dimension into the feature dimension. The output tensor shape is (B, T, C), where B is the batch number, T is the time length, and C is the feature dimension. Add a sinusoidal position code to each branch; T - L time steps are randomly selected from the total time length T of the sequence as the masking positions. At each selected time step, all spatial channel dimensions corresponding to that time step are set as masks, while the other time steps remain intact. The final amplitude and phase output after masking are (B, L, C).

2. The method according to claim 1, characterized in that, Based on the raw Wi-Fi signals collected from the environment, a two-dimensional data matrix containing amplitude and phase information of the wireless channel is output, including: Based on channel state information data covering several subcarriers collected by several receivers, channel response data with time and spatial dimensions is constructed. The channel response data is a three-dimensional data tensor with the structure (B, T, D), where B is the batch size, T is the time length, and D is the spatial dimension. The value of the spatial dimension is obtained by multiplying the number of subcarriers × the number of antennas × the number of receivers.

3. The method according to claim 1, characterized in that, Two identical sequence encoders are used to encode the masked amplitude and phase information respectively, resulting in amplitude and phase feature sequences. These two sequences are then fused into a joint feature sequence via a joint encoding module, including: The amplitude and phase features after masking are tensors of shape (B, L, C), where B is the batch size, L is the masking time length, and C is the feature dimension. The two signals are fed into two Mamba encoders with the same structure but independent parameters, respectively, to perform time modeling on the amplitude sequence and phase sequence. The outputs of the Mamba encoders are amplitude and phase feature sequences of shape (B, L, C). The two sequences output by the Mamba encoder are concatenated along the feature dimension to obtain a fused feature sequence of shape (B, L, 2C). This fused feature sequence of shape (B, L, 2C) is input into the joint encoding module and, after a linear transformation, is mapped to the output features of shape (B, L, C).

4. The method according to claim 3, characterized in that, The decoder reconstructs the amplitude and phase of the occluded locations in the joint feature sequence, obtaining a reconstruction result with the same dimensions as the original input, including: Generate a set of learnable temporal step location representations, a trainable embedding matrix of shape (TL, C), and concatenate it with the joint feature sequence at the occlusion location along the temporal dimension to form a new input representation (B, T, C); The combined input sequence decoding module employs a multi-layer Mamba decoder similar in structure to the Mamba encoder. It processes the fused features along the time dimension and outputs two decoding branches, corresponding to the reconstruction of amplitude and phase, respectively. Each decoding branch uses a separate projection head to restore the features to the same dimension as the original input, i.e., (B, T, D), and reconstructs the amplitude and phase of the masked positions in the original signal.

5. The method according to claim 1, characterized in that, A gesture recognition network consisting of a lightweight classification network includes: The joint feature sequence of (B, T, C) is compressed along the time dimension to obtain the global feature representation (B, C), which is then input into a classifier to output the final category prediction result (B, N), where N is the number of gesture categories.

6. The method according to claim 1, characterized in that, Using the end-to-end trained model, the input is the raw signal data containing amplitude and phase information, which is actually collected. After processing by a sequence encoder, a joint coding module, and a gesture recognition network, the corresponding gesture category is output, including: The input data is a complete sequence of amplitude and phase information of the original channel state, i.e., amplitude and phase tensors of shape (B, T, D). No masking operation is performed. The complete input signal is first processed by two independent linear layers and a Mamba encoder to extract the signal features of amplitude and phase, respectively, and outputs a temporal representation of shape (B, T, C). The two features are then concatenated along the channel dimension to form a fused representation of (B, T, 2C), which is then mapped to a joint feature sequence of (B, T, C) by a linear transformation in the joint coding module. The gesture recognition network compresses the joint feature sequence of (B, T, C) along the time dimension to obtain a global feature representation of (B, C), which is then input into a classifier to output the final category prediction result (B, N).

7. A gesture recognition and sensing system based on Wi-Fi signals, characterized in that, include: The data acquisition module is used to output a two-dimensional data matrix containing amplitude and phase information of the wireless channel based on the raw Wi-Fi signal collected from the environment. The self-supervised masking reconstruction pre-training module transforms the data matrix into a standard input format suitable for the sequence encoder. It masks certain segments through random masking operations, and uses two structurally identical sequence encoders to encode the masked amplitude and phase information, respectively, obtaining amplitude and phase feature sequences. These two sequences are then fused by a joint encoding module to obtain a joint feature sequence. A decoder reconstructs the amplitude and phase at the masked positions in the joint feature sequence, obtaining a reconstruction result with the same dimensions as the original input. The difference between the reconstructed result and the corresponding amplitude and phase in the original signal is used as a loss function for pre-training the sequence encoder and decoder. The random masking operation masks certain segments, including: The amplitude branch and phase branch of the input channel state information are linearly transformed through a linear layer to convert the spatial dimension into the feature dimension. The output tensor shape is (B, T, C), where B is the batch number, T is the time length, and C is the feature dimension. Add a sinusoidal position code to each branch; T - L time steps are randomly selected from the total time length T of the sequence as the masked positions. At each selected time step, all spatial channel dimensions corresponding to that time step are set as masks, while the other time steps remain intact. The final amplitude and phase output after masking are (B, L, C). The model fine-tuning module is used to retain the structure and parameters of the pre-trained sequence encoder and joint encoder module, replace the decoder with a gesture recognition network composed of a lightweight classification network, and use supervised optimization to minimize the difference between the predicted class distribution and the true label for end-to-end model training. The gesture classification and determination module is used to take the raw signal data containing amplitude and phase information that is actually collected as input, which is trained by the end-to-end model, and processed by the sequence encoder, joint encoding module and gesture recognition network to output the corresponding gesture category, thereby realizing non-contact gesture recognition function.

8. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the gesture recognition and perception method based on Wi-Fi signals as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the gesture recognition and perception method based on Wi-Fi signals as described in any one of claims 1-6.

Images